Dr. Dobb's is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Channels ▼


Test Driven Development

Grant Lammi is a technology evangelist at Seapine Software with over a decade of experience as a developer in the software industry. He can be contacted at www.seapine.com.

One of the biggest challenges of adopting Test Driven Development (TDD) surprisingly is not technical. Instead it is often the psychological challenges that cause TDD to be used poorly or not adopted at all. For developers, writing in a TDD style can be akin to signing their name with the wrong hand. While it is certainly possible to make the signature look the same, it takes more time and concentration because it just feels unnatural. Furthermore, the idea of writing test code in order to exercise production code is already strange enough for most developers. The situation can then seem downright bizarre when coupled with writing test code before the application code.

Frequently, developers are also reluctant to accept a method that requires them to write more code than they did previously. This stems not only from the age-old programming adage that less code is better code, but also from a fear that the new method could jeopardize already tight schedules.

In this article, I examine a project that used both TDD and traditional development methods. Many TDD projects have a large number of variables, which makes it difficult to successfully quantify the benefits. However, the project discussed in this article has a couple of advantages.

  • One developer worked on all the features for this release, which made the programmer's skills a constant and eliminated a large amount of variance.
  • This release was small in scope and was shipped with a larger release of another product. This took the stress of scheduling and the temptation to cut corners away since, even if using TDD took twice as long, the project would still be completed easily before the larger release.

The project was written in C++ and had to support Windows, Mac OS X, Linux, and Solaris. We used the UnitTest++ testing library for unit testing because it is cross-platform nature and easy to use. To make an informed decision about the effectiveness of TDD, the new release features were divided into two categories. The first set used traditional software development methods. Code was written first, then tested by the developer, and then handed over to QA for formal testing. The second set used TDD to design and test code before giving it to QA. To evaluate the methods we compared the number of defects found by QA for each feature and gathered some empirical observations from the developer about how he felt the code came out.

The QA department found 61 total defects across all features. This included defects for the newly added features as well as existing ones. It is possible that some of these defects existed in previous versions of the product.

Table 1: Total defects: All features.

Focusing on only the new features, a total of 33 defects were found. When analyzing the severity, an increase in quality is clear, as evidenced by the lack of any Critical defects and a reduction in Major defects.

Table 2: Total defects: New features.

Looking at the total number of defects broken down by feature, the data shows that most of the reported defects were either UI issues or upgrade issues to new libraries and compilers. (Trolltech's Qt cross-platform UI framework is used for the GUI and was upgraded to a new version in this release. The compilers for Windows, Linux, and Solaris were also upgraded, which caused some critical plug-in binary compatibility issues.)

Figure 1: Defects by all features (63).

Removing the upgrade and existing issues from consideration and comparing only the new features provides a better view of the data. Notice there are no Column Enhancements or Support File Generation defects. The comparison also highlights how areas that are hard to unit test in general, such as the UI, are still problematic.

Figure 2: Defects by new features (33).

When the defects from the new features are broken down into the traditional and TDD development methods, the difference in defects found is very apparent.

A final view of the data compares traditional and TDD methods, when looking at the severity levels of the reported defects. It is worth noting that the single TDD defect only occurred on a single QA test computer. The issue appeared to be an integration bug between a third-party library and an external system that only occurred under specific circumstances. It is entirely possible that it might not have been found if the QA department was not so thorough, which goes to show that no development methodology is a silver bullet to a quality.

Table 3: Traditional development.

Taking away cosmetic issues, which were not eligible for unit testing, there is still an 11 to 1 ratio of defects found when the two methods are compared.

Table 4: Test Driven Development.

Related Reading

More Insights

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.

Tools Recent Articles

Upcoming Events

Most Recent Premium Content