Moshe Cohen and Jon Chard both work in the Rational division at IBM Software.
The challenge to any software development effort is that quality management is a tricky balancing act that must factor in time, cost, and risks. Get it wrong, and you could face issues ranging from unsustainable costs, missed windows of opportunity, and unhappy customers, to a massive recall or the complete failure of a system at a critical moment. Get it right, and you can achieve a positive operational return on investment from efficiencies gained in development activities. With effective quality management, you can also create opportunities to deliver critical (but difficult-to-qualify) benefits, such as improved marketshare, higher customer satisfaction, and increased brand equity.
What’s the problem with quality?
Complexity is now a given in many products and systems. Software, microelectronics, sensors, and mechanical technologies are often combined to create products that can measure conditions, respond to changes, and interact with other products, people, and IT systems in new ways.
Consider the potential overall effect of quality issues: In many industries, the entire success of an organization can hinge on consistently solving complex challenges. Yet success and brand image cultivated over many years can evaporate quickly when something goes wrong. Examples are plentiful: an aerospace company faced a $1 billion loss when a rocket self-destructed because of a bug in a guidance system. In the healthcare industry, software problems and poor quality control in cancer treatment systems designed to deliver precise radiation treatments have led to tragic consequences for a number of patients. In these types of cases, what may seem to be a small defect is actually critical to the mission.
One of the biggest challenges related to quality management is how to invest intelligently to minimize risk, given economic constraints. Figuring out how to relate quality to outcome and what constitutes the right level of quality for individual products, however, is not always clear.
The safest way to address quality problems early in the lifecycle is to use proven approaches and process frameworks, such as the Capability Maturity Model (CMM), Capability Maturity Model Integration (CMMI), agile development, or other general or domain-specific approaches. Focusing on best practices that can deliver a positive return provides a way to separate practices that will deliver tangible business results.
During any software delivery process, “When do we release?” is a key question with no simple answer. You must consider project-specific variables, such as the cost of delays, the opportunity value of early delivery, marketplace quality expectations, and the costs associated with defects. Ultimately, the delivery strategy will be based on the actual or perceived importance of each variable.
Schedule-driven delivery effectively implies “deliver on time, regardless of other factors” and is often used in consumer products to hit marketplace windows or in contractually governed environments, where delay penalties may apply. Quality issues are often magnified, given that software contractors frequently get paid on a time and materials basis regardless of the quality of software they deliver. In many cases, you may even end up paying extra for them to fix their own defects, so the potential costs of defects can add up quickly.
According to the Carnegie Mellon Software Engineering Institute, “Data indicate that 60-80% of the cost of software development is in rework.”
Quality-driven delivery can also be costly but for different reasons. The release timing for this approach is governed by achieving the right quality — but how do you define that? Zero defects is practically impossible to achieve, given that there is no way to determine how many defects still exist in a piece of code or the probability of detecting those defects in use. A target based on defects fixed might be more realistic — but it’s still impossible to know the number of remaining defects in the product.
To cost-effectively balance quality-risk versus time-to-market considerations, a risk-driven approach may be ideal. A risk-driven strategy is a refinement of a quality-driven approach that optimizes risk exposure against development cost and time.
The optimal time to release is when the total risk exposure is minimal, typically around the time where the risk associated with competitive threats starts to outweigh the risk reduction associated with further quality improvements, as illustrated in Figure 1.
The best time to release can vary widely: Move too soon, and customers may be upset with quality issues. Wait too long, and your competition will beat you to the punch and dominate the space. In the end, the optimal time to release lies where your overall risk is minimal. Consider the following examples.
- Mass-market products — In the consumer electronics marketplace, time to market is critical to product success. Here, the opportunity cost of delaying a release to improve quality can be high, thus pushing the optimum release point to an earlier date.
- Safety-critical applications — In applications such as flight control software, quality may be the most critical parameter, given the potential size of loss caused by a defect. In this case, the optimum release point would be a later date.
- Systems with high-availability requirements — Mobile communications in the telecommunications industry, for example, have extremely high availability requirements because of the commercial implications of system outages. As a result, the potential loss resulting from a defect makes quality critical, pushing the optimum release point to a later date.
In practice, the required time-to-market is always ahead of your optimal time to release, where your overall risk exposure is the lowest. The only way to hit your time-to-market window without increasing the risk is by accelerating your risk reduction over time.
Quality management versus testing
If a faster reduction in risk is the goal, how do you achieve it? The answer is not testing, which is focused solely on discovering defects. Rather, an approach that is focused on reducing risk throughout the lifecycle is need. This is what quality management is all about. And by choosing quality management best practices that deliver a quick return on investment, you can focus on reducing risks in your project knowing that your management is behind you and supporting you.
Traditionally, testing has been a late-stage activity in the development lifecycle, conducted between the construction of software and its eventual release. Quality management can be considered as its own lifecycle within the overall software development lifecycle, as shown in Figure 2. Its activities take place across the development process and are synchronized with the development process in key points.
The entire process is closely linked to change and configuration management because discovered defects can be considered as changes that must be managed along with other changes progressing through the change management process.
There are many strategies for reducing risk, so it is essential to distinguish between those that are good ideas and those that also deliver a positive returns. The following are six "best practice" strategies that are proven to deliver a return on investment.
1. Drive testing from requirements. Linking test definition to requirements can help ensure that what you ultimately test reflects requirements. This approach can help reduce risk by ensuring that tests demonstrate compliance with requirements. However, manually building such traceability is time-consuming. A dedicated quality management solution with support for capturing traceability could speed up the process by a factor of ten, typically leading to savings in the order of $150-300K for projects with approximately 5000 requirements and 10,000 test cases.
2. Reduce planning overhead with collaborative quality management. Quality management is a lifecycle activity that affects everyone involved in a project. As such, managing the testing plan is a collaborative activity involving all stakeholders. A central repository where stakeholders can share and access updates and a “single version of the truth” can significantly cut the time spent on collaborative tasks. For a team of 100 testers, this could lead to savings of around $750,000 over 12 months. It also means that a team of 92 testers could do the work of 100, leaving the other 8 to accelerate the schedule.
3. Prioritize testing according to risk. Although you cannot completely remove risk from a development program, you can manage and measure it by taking proportionate mitigation actions. This approach prioritizes the execution of the test cases, where those addressing high risk requirements are held above those with a lower risk. This approach leads to significant reduction in the total number of test case executions, corresponding to large savings.
4. Integrate testing and quality management to improve accuracy and repeatability. Integrating both manual and automated testing within the quality management environment can deliver efficiency and quality benefits, including improved management of tests; likelihood of executing the correct set of tests; and collaboration resulting from better information on testing status and results
5. Automate reporting to improve efficiency, consistency and decision making. Reporting is a key element of the quality management process because it facilitates informed decision making based on information captured across the process. By automating the process, you can produce subsequent reports with minimal effort; therefore you are more likely to produce reports more often, helping to detect potential problems sooner. Automation also helps improve the accuracy and consistency of reporting — factors that can positively influence project management, productivity, and quality.
6. Improve efficiency by applying smart defect management to eliminate duplicates. As project complexity increases, the number of detected defects increases, as does the occurrence of duplicate reporting of defects. Identifying such duplicates is a time-consuming activity that adds no value to the project. Tooling that automates the detection of duplicate defects as testers log them can therefore help reduce overhead costs and shorten delivery timelines.
The overall effect of risk reduction strategies
Each of the best practices discussed here can produce significant results. But collectively, they also contribute to earlier defect detection. And as the cost of fixing defects is lower the earlier it is fixed, applying these best practices leads to significant savings due to earlier defect resolution.
As an example, let’s assume a team that is at a CMMI level 2, and a project with 1000 defects detected by unit and integration testing. Applying the best practices discussed above can lead to the early detection of over 900 additional defects.
To realize these benefits, organizations would do well to invest in software that provides a collaborative, customizable, web-based test and quality management hub for the quality management lifecycle. Whether they are down the hall or across the globe, quality professionals and other decision makers can use quality management software to collaborate on test planning and management, risk-based testing, and defect management, including duplicate defect detection.
The momentum of technology, coupled with customer demands, is pushing organizations to deliver ever-smarter products. Given the added complexity in development processes, finding ways to continually optimize time, cost, and quality is critical to successful innovation. A risk-driven development approach can help you balance quality risk with time-to-market imperatives, and it also can do so while delivering a significant financial returns. The cost benefits are achieved through addressing defects earlier in the development process when they are less expensive to fix, as well as through productivity and efficiency gains in development processes.
What’s more, such an approach can also support long-term benefits that cannot be easily quantified but are key to success, including customer satisfaction, reduced in-service costs, and a strong reputation and brand.