Channels ▼


Scaling Test-Driven Development

Validation at Scale

The second aspect of TDD is validation, and it too could do with some scaling help. TDD is in effect an approach to confirmatory testing where you validate the system to the level of your understanding of the requirements. This is the equivalent of "smoke testing" or testing against the specification. This is clearly an important thing to do, but it isn't the whole validation picture. The fundamental challenge with confirmatory testing, and hence TDD, is that it assumes that stakeholders actually know and are able to describe their requirements. Yes, iterative approaches clearly increase the chance of this, but there are no guarantees. TDD further compounds the problem by assuming that developers have the skills to actually write and run the tests, skills that can clearly be gained over time but still must be gained.

So, to scale the validation aspects of TDD, you need to add investigative testing practices into your software process. The goal of investigative testing is to explore issues that your stakeholders may not have thought of, such as usability issues, system integration issues, production performance issues, security issues, and a multitude of others.

In my January 2007 column (Agile Testing Strategies,, I described how Agile teams approach testing, often having a small, independent investigative test team working in parallel to the development team. Your testing environment should simulate your production environment as much as possible. Figure 2 overviews this strategy, showing how the development team deploys the current working build into your testing environment. This should occur at least once an iteration because you want to reduce the feedback lifecycle as much as possible.

[Click image to view at full size]

Figure 2: Continuous investigative testing.

The independent testers don't need a lot of details; their goal is to break the system in ways that the development team hasn't thought about—the implication is that they don't need a detailed requirements specification from you. The only documentation that they might need is a list of changes since the last deployment so that they know what to focus on first, because most likely new defects would have been introduced in the implementation of the changes. The investigative testers will use complex, and often expensive, tools to do their jobs and will usually be very highly skilled people.

When the investigative testers find a potential problem, it might be what they believe is missing functionality or it might be something that doesn't appear to work properly, which they write up as a "change story." Change stories are basically the agile form of a defect report or enhancement request, and many teams will often use defect-tracking tools such as ClearQuest or Bugzilla to manage them. The development team treats change stories like requirements—they estimate the effort to address the requirement and ask their project stakeholder(s) to prioritize it accordingly. Then, when the change story makes it to the top of their prioritized work-item list, they address it at that point in time. In effect, any potential defect found via investigative testing becomes a known issue that is then specified and validated via TDD. Because TDD is performed in an automated manner to support regression testing, the implication is that the investigative testers do not need to be as concerned about automating their own efforts, but instead can focus on the high-value activity of defect detection.

TDD at Scale

Test-driven development (TDD) is an important agile development technique that enables you to specify and validate functionality at a finely detailed level. But it does not enable you to achieve these same goals at a higher level, so you need to adopt other, complementary techniques. The principles of Agile Modeling can be applied to address the high-level specification issues that TDD does not, and an independent test team working with your development team can address the testing concerns that TDD does not.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.