The Efficiency Factor
Our research shows that most forms of testing are less than 30% efficient in finding errors and bugs. Formal inspections, on the other hand, have a defect-removal efficiency rate that ranges from just less than 50% to more than 85% and an average efficiency level of roughly 65%. Static analysis tools have about the same level of defect-removal efficiency, but they tackle only a narrow band of code-related defects.
Gilb says that some inspection efficiency rates have been as high as 90%. Authors Lew Priven and Roger Stewart report inspection defect-removal rates topping 95%. With the possible exception of high-volume beta testing involving more than 10,000 simultaneous test sites, testing never approaches this level of efficiency.
A combination of formal inspections, static analysis, formal testing by test specialists, and a formal (and active) quality-assurance group are the methods most often associated with projects achieving a cumulative defect-removal efficiency higher than 99%.
How Inspections Are Conducted
Formal inspections are manual activities in which four to six colleagues go over requirement and design specifications page by page using a specific protocol. Code inspections follow the same approach but go over listings or screens line by line. To qualify as an inspection, certain criteria must be met, including:
- Appointing a moderator to keep sessions moving;
- Having a recorder take notes;
- Having participants adequately prepare for each session;
- Keeping records of discovered defects;
- Not using defect data for appraisals or punitive purposes.
The original concept of inspections was based on participants actually getting together. Online communications and collaborative Web tools now mean that inspections can be performed electronically, saving travel costs for teams that are geographically dispersed.
Any software deliverable can be subject to a formal inspection. Some types of inspections have developed enough empirical data to indicate that the inspection process is generally beneficial. They include architecture, requirements, project plan, software cost estimate, software marketing plan, software governance plan, software security plan, design, database design, code, test plan, test case, user documentation, and plaintiff complaints and defendant responses in breachof-contract litigation inspections.
Thanks to the flexibility of the human mind and its ability to handle inductive logic as well as deductive logic, inspections are the most versatile form of defect removal and can be applied to essentially any software artifact. Indeed, inspections have even been applied recursively to themselves to fine-tune the inspection process and eliminate bottlenecks and obstacles. The Development Process One of the best ways of showing the effectiveness of inspections is to produce graphs that connect the point in time when software defects are discovered with the point where the defects originate (see Figure 1). An acute angle in the line connecting defect discovery and origin points indicates a serious software quality-control problem, because the gap between making an error and discovering it can be many months. The goal of defect removal is to have the angle connecting defect origins and discoveries approach 90 degrees, indicating the defect happens and is discovered at the same time. A 90-degree angle is unlikely, but formal inspections can at least bring the angle up from perhaps 30 degrees to more than 60 degrees (see Figure 1).
Software projects that don't use formal inspections enter a "zone of chaos" during the test cycle (see Figure 2). Problems with requirements and specifications that are deep in the process suddenly emerge, and require extensive and expensive repair and rework.
When you use formal inspections, the lines connecting the discovery points of defects with their origins have angles approaching 90 degrees. Defects that originate within a phase tend to be eliminated during that phase and don't pass to downstream phases.
These graphs provide a quick visual gauge of the delays in software defect discovery, and resolution. Wider use of this approach and of manual inspections would result in earlier discovery and correction of defects, leading to higher-quality software that's produced faster and more efficiently. That's a goal we should all support.
Capers Jones is an independent consultant and has written several books on software productivity management and software estimation. Olivier Bonsignour is VP of product development at CAST, which specializes in software testing. This article is adapted from their book, The Economics Of Software Quality, published by Addison-Wesley.