Validation at Scale
The second aspect of TDD is validation, and it too could do with some scaling help. TDD is in effect an approach to confirmatory testing where you validate the system to the level of your understanding of the requirements. This is the equivalent of "smoke testing" or testing against the specification. This is clearly an important thing to do, but it isn't the whole validation picture. The fundamental challenge with confirmatory testing, and hence TDD, is that it assumes that stakeholders actually know and are able to describe their requirements. Yes, iterative approaches clearly increase the chance of this, but there are no guarantees. TDD further compounds the problem by assuming that developers have the skills to actually write and run the tests, skills that can clearly be gained over time but still must be gained.
So, to scale the validation aspects of TDD, you need to add investigative testing practices into your software process. The goal of investigative testing is to explore issues that your stakeholders may not have thought of, such as usability issues, system integration issues, production performance issues, security issues, and a multitude of others.
In my January 2007 column (Agile Testing Strategies, www.ddj.com/development-tools/196603549), I described how Agile teams approach testing, often having a small, independent investigative test team working in parallel to the development team. Your testing environment should simulate your production environment as much as possible. Figure 2 overviews this strategy, showing how the development team deploys the current working build into your testing environment. This should occur at least once an iteration because you want to reduce the feedback lifecycle as much as possible.
The independent testers don't need a lot of details; their goal is to break the system in ways that the development team hasn't thought aboutthe implication is that they don't need a detailed requirements specification from you. The only documentation that they might need is a list of changes since the last deployment so that they know what to focus on first, because most likely new defects would have been introduced in the implementation of the changes. The investigative testers will use complex, and often expensive, tools to do their jobs and will usually be very highly skilled people.
When the investigative testers find a potential problem, it might be what they believe is missing functionality or it might be something that doesn't appear to work properly, which they write up as a "change story." Change stories are basically the agile form of a defect report or enhancement request, and many teams will often use defect-tracking tools such as ClearQuest or Bugzilla to manage them. The development team treats change stories like requirementsthey estimate the effort to address the requirement and ask their project stakeholder(s) to prioritize it accordingly. Then, when the change story makes it to the top of their prioritized work-item list, they address it at that point in time. In effect, any potential defect found via investigative testing becomes a known issue that is then specified and validated via TDD. Because TDD is performed in an automated manner to support regression testing, the implication is that the investigative testers do not need to be as concerned about automating their own efforts, but instead can focus on the high-value activity of defect detection.
TDD at Scale
Test-driven development (TDD) is an important agile development technique that enables you to specify and validate functionality at a finely detailed level. But it does not enable you to achieve these same goals at a higher level, so you need to adopt other, complementary techniques. The principles of Agile Modeling can be applied to address the high-level specification issues that TDD does not, and an independent test team working with your development team can address the testing concerns that TDD does not.