Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼


Dr. Dobb's Agile Update 01/09

In This Issue:

  • Test-Driven Development (TDD): Reality Over Rhetoric
  • Book Review: Agile Testing
  • Hot Links

Test-Driven Development (TDD): Reality Over Rhetoric

Test-driven development (TDD) is an agile development practice which combines test-first development (TFD) and refactoring. With TFD you write a single test and then you write just enough software to fulfill that test. You can do TFD at the requirements level by writing a single customer test, the equivalent of a function test or acceptance test in the traditional world, and at the design level with developer tests. Refactoring is a technique where you make a small change to your existing code to improve its design without changing its semantics. Examples of refactorings include moving an operation up the inheritance hierarchy and renaming an operation in application source code; aligning fields and applying a consistent font on user interfaces; and renaming a column or splitting a table in a relational database.

During the second week of October 2008 I ran a survey exploring the adoption rate of Test-Driven Development (TDD) and related techniques within the agile community. The survey was announced on the Extreme Programming (XP) and Test-Driven Development (TDD) mailing lists, lists which I felt best represented the TDD community, and there were 121 respondents in total. I fully recognize the bias resulting from going directly to the TDD community, therefore I wouldn't want to claim that my adoption rate statistics for TDD reflected the actual adoption rate. The goal was to find out what's actually going on with TDD practitioners to get a better understanding of the range of specification and validation techniques being applied in practice by agile teams. Hopefully by shedding some light on what they're doing in practice we can start moving away from the macho TDD rhetoric and instead move towards more realistic and comprehensive strategies for specification and validation on agile teams.

The impetus for this survey was concern within the agile community regarding the results of two previous surveys that I had run during July 2008. The first was a Dr. Dobb's survey which explored how development teams approached modeling and documentation and the second was an Ambysoft survey sent out to several agile mailing lists which explored the adoption rate of agile principles and practices. Both surveys showed that the amount of discussion going on within the agile community about TDD didn't reflect the apparent adoption rate. In fact, there appeared to be greater adoption of agile modeling and documentation techniques than TDD, yet very little discussion occurred about these topics. Many Agilists weren't happy having this pointed out to them, and some said that the surveys didn't explore TDD enough to accurately state what was really happening. So I decided to look into TDD and related subjects further with a new survey.

Within the TDD/XP community there is a significant focus on other testing and validation techniques. Although there is a clear bias towards TDD on these lists, for obvious reasons, TDD is clearly not the only validation technique that has been commonly adopted by agile developers. For example the survey found that 71% were doing developer TDD (design-level) and 40% doing acceptance TDD (requirements-level), but as I indicated earlier I certainly wouldn't take this as a fair assessment of the true adoption rate across the industry. But what was very revealing was that 52% indicated that they were doing reviews or inspections on their agile projects, 45% were doing testing at the end of the lifecycle (something that rarely seems to get discussed on the agile lists), and 36% are working with an independent test team in parallel to development (also something that gets drowned out in the discussion around the virtues of the whole team philosophy).

The survey explored the benefits that respondents were actually experiencing with TDD. When it came to acceptance TDD (requirements-level TDD) the top three benefits were improved specification accuracy (73% response rate), increased ability of software developers to actually change software (66%), and increased quality (61%). For developer TDD (design-level TDD) the top three were increased ability of software developers to actually change software (98%), increased quality (92%), and increased ability to react to stakeholders' changing needs (72%).

When it comes to capturing requirements on their current agile project, 85% of respondents indicated that they used some sort of documentation approach, 53% used some sort of modeling approach, and 45% used acceptance tests. Even with a group of respondents clearly biased towards TDD it was still more common for them to capture requirements using a word processor (52%) than tests (45%). Interestingly defect management tools, such as Bugzilla or IBM Rational ClearQuest, were being used by 39% of agile teams to capture requirements, a technique that I've been writing about for several years now. Wikis were only being used by 26% of respondents to capture requirements, a fairly low figure considering how much time is invested within the agile community talking about wikis, whereas 20% indicated that they were using requirements management tools such as Telelogic DOORS to capture requirements even though virtually nothing is mentioned of this strategy on the agile mailing lists.

The results were similar when it came to capturing design specifications on their current agile project. 80% indicated that they captured design information using some sort of documentation approach, 67% via a modeling approach, and 57% via tests. One difference was that the single most popular strategy capturing design was via developer tests (57%), although whiteboard sketches came in a close second at 49% and both word processors and wikis at 38%. Only 27% indicated that they were using paper tools, a low figure considering the amount of discussion that we hear about the virtues of index cards and sticky notes, and only 21% were taking digital snapshots of whiteboards (this low figure could in part be due to many organizations banning cameras).

I suspect that there are several reasons why developer TDD at 57% is more common than acceptance TDD at 45%. First, people seem to be getting better results from developer TDD. Second, acceptance TDD is harder to automate, particularly for dynamic user interfaces. Third, although there's some really great tooling for both techniques in my opinion the developer TDD tools are in general better.

So, why aren't people doing more TDD? First and foremost, TDD requires discipline and it requires skill. When asked about the difficulty of learning acceptance TDD, 39% said it was either very difficult or difficult and 49% said the same of developer TDD. The survey explored four potential issues which could be hampering the adoption of TDD, and all four seemed to be having a significant impact. The lack of TDD skills had the largest impact followed closely by a lack of regression test suite for existing functionality. The other two factors, lack of training and education (which clearly has an impact on skills) and a waterfall culture within organizations trailed closely behind.

Another serious challenge is that management support for TDD could be better. For example, only 37% indicated that management was willing to fund training and education for acceptance TDD and 52% for developer TDD. An additional 21% indicated that management was interested in adopting acceptance TDD but wasn't funding it and 31% for developer TDD. 6% and 8% of respondents indicated that management was actively against acceptance and developer TDD respectively.

The survey also explored how people were learning TDD. For both types of TDD, respondents indicated that pairing with an experienced person and mentoring were the most effective means of learning the technique. Training came in a distant third followed by other techniques such as pairing with another learner, reading, and online forum discussions. I believe that these results reflect that fact that TDD requires both skill and discipline, two things which take time to master.

I'd like to wrap up with three thoughts. First, although the potential benefits of TDD are great they require an investment on the part of your organization to obtain. Second, I think it's fairly clear that the agile community still struggles to communicate what they actually do in practice, and I suspect that it's hampering the successful adoption of agile strategies in many organizations. Third, you don't need to trust my analysis: The results of this survey, including the original questions as they were asked, the resulting data (without identifying information to preserve people's privacy), and a summary slide deck can be found here.

Book Review

Agile Testing: A Practical Guide For Testers and Agile Teams
Lisa Crispin and Janet Gregory
Addison Wesley, 2009

Testing is an important aspect of software development, regardless of paradigm. Agile developers are arguably "test infected" and have a greater appreciation for the need to test their software as often and early as possible. Agile Testing: A Practical Guide For Testers and Agile Teams, by Lisa Crispin and Janet Gregory, describes many of the aspects of how agile teams approach testing, overviewing the philosophies and strategies followed by agile teams. It provides wonderful advice for how existing testers and quality assurance (QA) professionals can become productive members of agile teams and argues for a tight integration between development and testing professionals. It also includes strategies for test-driven development (TDD) and effective test automation.

There are seven success factors, summarized in the last chapter, which weave throughout this book Five of the success factors are "soft" in nature: Whole team, where testers are "embedded" in the development team and actively participate in all aspects of the project; Providing and obtaining feedback on a daily basis to stay on track and in sync with the rest of the team; Collaborating and working closely with stakeholders; Adopting an agile testing mindset which tells you to focus on building the best possible product and not fall into the trap of being "quality police"; and looking at the big picture which says to look beyond the immediate development needs to consider the implications to the system as a whole. Two success factors are more technical or process-oriented in nature, automating regression testing and adopting core practices such as continuous integration, managing technical debt, and working incrementally.

All in all it's a well-written, easy to understand book. The only weakness of the book is that it gives short shrift to scaling issues. End-of-lifecycle testing, which is a common reality for many agile teams, is barely discussed nor is the need to go beyond the whole team approach to include parallel independent testing. Distributed team issues are discussed briefly in a few places although large-team issues, regulatory compliance, and multiple-organization issues weren't covered. Hopefully the next edition of the book will address these issues.

Hot Links

The results of Ambysoft's 2008 Test Driven Development (TDD) Survey are posted here.

The results of Ambysoft's 2008 Agile Practice and Principles Survey are posted here.

The results of Dr. Dobb's 2008 Modeling and Documentation Practices on IT Projects Survey are posted here.

My November 2008 column "Newsflash: Agilists Write Documentation" which summarizes the results of the modeling and documentation survey are posted here.

The article Introduction to Test-Driven Design (TDD) provides a good overview of TDD.

My Agile Testing Strategies column in the January 2007 issue of Dr. Dobb's described how agile teams approach testing.

The article Database Regression Testing describes in detail how to regression test a relational database.

My February 2008 Dr. Dobb's column, Scaling Test-Driven Development (TDD), explores how TDD fits into the overall specification and validation picture on agile projects.

The article Agile Testing and Quality Strategies overviews how agile development teams approach testing.

The article Agile Model Driven Development (AMDD) describes how TDD fits into the overall specification portion of the development lifeycycle.

The article Requirements Envisioning: An Agile Best Practice describes how to go about initial requirements modeling on an agile project in a streamlined manner.

My [email protected] blog is where I discuss strategies for adopting and applying agile strategies in the complex environments.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.