Channels ▼


Change Code Without Fear

Source Code Accompanies This Article. Download It Now.

Understanding the Key Phases

There are two key phases involved in creating and using Behavioral Regression Test Suites:

  • Phase 1. Automated one-time test generation. A one-time process that involves generating a regression test suite that can be run on a regular basis and will not report any errors unless the captured behavior changes (Figure 1).
  • Phase 2. Automated test generation for new code + Automated daily execution + extend tests. A daily process that involves automated generation of regression tests for new code, as well as automated execution of the existing test suite. After the execution completes, you review and respond to any functional changes that the test suite detected. The response process involves injecting human intelligence into the test suite.

[Click image to view at full size]

Figure 1: Phase 1.

Once per project, an automated unit test generation tool scans the project code base (accessed from the source control system), then automatically generates and executes unit test cases with assertions that capture the project code's current functionality for a wide range of automatically generated test scenarios. Test execution results should be stored on a central server, where they can be shared team-wide, and the generated tests should be added to source control. At this point, you might want to fine-tune the test suite to minimize noise (by ignoring any assertions that fail on a second run against the exact same code base, ignoring time-sensitive and date-sensitive assertions, and so forth). However, such fine-tuning is completely optional at this phase.

Because your goal is to establish a behavioral baseline rather than verify the application's current functionality, there is no need to actually review—or even glance at—the generated tests or the assertions/observations/outcomes attained. Assuming that your application is currently functioning as expected, just blindly verify all the reported assertions. Or, even better, configure the tool to automatically set the actual outcomes as the expected ones.

No test failures should be reported for the initial run of this test suite. To minimize the amount of noise presented, failures should be reported only when code behavior changes as a result of code-base modifications (Figure 2).

[Click image to view at full size]

Figure 2: Phase 2.

This daily process involves running the evolving code base against the behavioral regression test suite on a regular basis. Running in batch mode, the unit-testing tool scans the modified project code base (accessed from the source control system), automatically executes the existing regression test suite, and also automatically generates regression tests for new code. Test failures are reported only if the current code behavior does not match the behavior captured in the baseline test suite.

This alerts you whenever the previous day's changes caused the tests to fail. If this occurs, you need to review and address the reported failures. At minimum, you need to respond to failures daily to keep the test suite in sync with the app. As resources permit, you may also extend the test suite as needed to cover any critical application functionality not yet represented in the test suite.

Ideally, the results from this automated test execution not only reveal which test case assertions failed as a result of the previous day's code modifications, but also indicate exactly which developer modified the code that broke each assertion. For instant feedback on whether their code changes broke the existing functionality, developers can import information into their IDE about the regression failures caused by their modifications. Because the regression failures are directed to the developers responsible for them, the overall process of fixing them is much more streamlined than it would be if all developers were looking at the same list of regression failures. And since the results are available in their IDEs, resolution is faster and easier.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.