Defect Removal Efficiency In Depth
The U.S. average for defect removal efficiency is only 85%, based on my research on about 13,000 development projects. The primary cause for projects running late or having cost overruns is excessive numbers of defects that aren't discovered or removed until testing starts. Such projects appear to be on schedule and within budget -- until testing begins. Then, hundreds or even thousands of latent defects are discovered causing delays and cost overruns, and the test schedule ends up far exceeding the original plans.
Most software testing averages about 35% to 50% in defect removal efficiency levels. As application size increases, test coverage and test removal efficiency drop. This suggests that additional quality control methods such as inspections are needed.
Formal inspections of requirements, design, and source code have been in use since IBM began looking for better quality control methods in the '70s. With inspections, defect removal efficiency levels spike higher than 85% and testing defect removal efficiency goes up by about 5% per test stage. More recently, static analysis tools used prior to testing were found to contribute to high levels of defect removal efficiency (in the 85% range), although not against dynamic problems such as performance. (See Table 2 for defect removal efficiency levels for various stages.)
While defect potentials and defect removal efficiency are the most effective ways of evaluating software quality controls, actually improving software quality requires two process improvements: defect prevention and defect removal. Defect prevention refers to technologies and methodologies that lower defect potentials or reduce the numbers of bugs that must be eliminated. Examples of defect prevention methods include joint application design, quality function deployment, Six Sigma, structured design, and participation in formal inspections.
For its part, defect removal refers to methods that can either raise the efficiency levels of specific forms of testing or raise the overall cumulative removal efficiency by adding other reviews or tests. The two approaches can be implemented at the same time.
To achieve a cumulative defect removal efficiency of 95%, it's necessary to apply at least nine defect removal activities in sequence:
- Design inspections
- Code inspections
- Automated static analysis
- Unit test
- New function test
- Regression test
- Performance test
- System test
- External beta test
Requirements inspections, test case inspections, and specialized forms of testing (such as human factors, performance, and security testing) add to defect removal efficiency levels.
Who's Using These Metrics?
Since defect potentials and defect removal efficiency metrics are among the easiest to use and most effective tech niques for improving software quality, you have to wonder why everyone isn't using them. My research has shown that companies most likely to use these metrics are ones that make computers and other complex hardware devices, such as telecommunication, aerospace, embedded equipment, and defense contracting companies. Many of them are topping 95% in defect removal efficiency, compared with an industry average of 82%.
Dovel Technologies, a software developer and system integrator that builds IT systems for government and private industry, reported in 2009 a 96% defect removal efficiency, which it credits to the adoption of defect potentials and defect removal efficiency metrics, along with close monitoring throughout the development life cycle using formal and informal reviews, among other approaches.
Companies that have adopted these metrics have cut their development and maintenance costs as well. When you have to rework defective requirements, design, and code, it can consume as much as 50% of the total cost of software and development.
What It All Means
Combining inspections, static analysis, and testing is cheaper than testing by itself and leads to much better defect removal efficiency levels. In concert, these approaches also shorten development schedules by more than 45% because, when testing starts after inspections, almost 85% of the defects already will have been addressed.
To measure defect potentials, it's necessary to keep good records of all defects found during development. When IBM applied formal inspections to a large database project, delivered defects were reduced by more than 50% compared with previous releases, according to my research, and the schedule was shortened by about 15%. Testing was reduced from three shifts over 60 days to one shift over 40 days. Most importantly, customer satisfaction improved to "good" compared with prior releases in which customers rated "very poor" satisfaction levels.
Cumulative defect removal efficiency was raised from about 80% to just over 95% as a result of using formal design and code inspections, and maintenance costs came down by more than 45% for the first year of deployment. Those are the kind of results that speak for themselves.