Email	Print Reprint
	add to:
	Del.icio.us Digg Google Furl	Slashdot Y! MyWeb Blink

Recently, I was hired as project manager of a software product release that had been suspended with 32 percent of the feature development in the balance, but was ready to resume development. I had to prepare a new schedule planning for the remaining feature development work, the restart activities and, most importantly, the testing activities.

I didn’t have a lot of elbow room: Acme (a pseudonym) had outsourced this project to my client under the condition that Acme’s established software development process be followed. This process stipulates a well-defined sequence of testing activities consisting of functional, interoperability, performance, installation, internalization, online help, regression and re-verification testing.

In the development phase, Acme’s process calls for reviewing the requirements and test plans, as well as informal code inspections by the lead developer, a daily build and coding standards.

While preparing the schedule, I found that Acme spends 55 percent of its total effort in the development phase; the remaining 45 percent is eaten up in testing and verification. This focus on testing gave me pause, as it not only increases the cost of development, but also delays the product release.

I was skeptical about the cost-effectiveness of Acme’s prescribed process, remembering the adage, Do more with less. Motivated to save money for my client, I went into action.

Seeking Solutions from Research
Before investigating the cost of Acme’s software process, I wanted to analyze the quality of its software products. According to Software Productivity Research chairman Capers Jones’ paper, “Software Defect-Removal Efficiency” (IEEE Computer, April 1996), software products can be considered “high quality” if the defect removal efficiency reaches 95 percent. The defect removal efficiency is a quality metric calculated by dividing the development defects (defects reported prior to product release) by the total number of defects (the sum of the development defects plus the defects reported during the first year of product use). In other words, for best-in-class companies, the investment in defect removal yields 95 percent of the defects prior to product release—and according to Jones, the industry standard of 85 percent just isn’t good enough.

In the same article, Jones remarks that a high level of software quality requires a multifaceted approach, comprising eight to 12 of the following practices:

• Requirements review
• Design review
• Code inspection
• Test plan review
• User manual review
• Unit testing
• New function testing
• Regression testing
• Integration testing
• Performance testing
• System testing
• External beta testing

Acme follows eight of the suggested quality practices—but do these practices remove 95 percent of the defects? Acme’s defect-tracking system classifies flaws as either “pre-general availability” or “failure,” but further investigation revealed inconsistent data collection and unreliable information. As a result, Acme doesn’t really know what the return on its investment in quality practices is.

Investigating the Process
Initially, I believed that a combination of different quality activities might lower Acme’s software cost. But the cost is related not only to what Acme does, but also when and for how long. The project schedule and the phases of Acme’s development process reveal that it follows a modified waterfall model—meaning that most quality practices occur after all features have been implemented. This causes late identification of defects and subsequent rework through redesign and retesting.

I wondered if a more iterative development process might give Acme the same software quality at a lower cost. In their unpublished report, “Flexible Models of Software Development: Must We Trade Off Efficiency and Quality?” (www.softwarecenter.cmu.edu/publications.html), researchers Alan MacCormack, Chris Kemerer, Michael Cusumano and Bill Crandall suggest that an iterative development process, including a combination of early prototype releases, design reviews and regression testing, followed by product builds, result in higher-quality software.

Armed with this information, Acme may want to consider an iterative development process and incorporate some of these quality practices.

Cutting Costs, Not Quality
Next question: Where can Acme find facts and figures to either support or reject alternative quality practices and process changes? Unfortunately, the company can’t cull from its historical data, nor is there time to conduct a pilot project. Research results offer the next best option.

In their study, “Evaluating the Cost of Software Quality” (Communications of the ACM, Aug. 1998), Carnegie Mellon and University of Michigan computer science professors Sandra A. Slaughter, Donald E. Harter and Mayuram S. Krishnan propose a common-sense strategy for reducing software quality costs. The primary objective? Avoid rework—including design, coding and testing efforts required to fix defects—both during the product development cycle and while the product is in use.

Slaughter and company distinguish between two types of quality activities and their costs: prevention and appraisal. As their names imply, prevention activities attempt to avoid the introduction of defects, whereas appraisal activities aim to detect them. Prevention activities—like design reviews and training—occur prior to programming; appraisal activities—such as code inspection and regression testing—follow coding.

This strategy suggests that companies invest in prevention activities, especially those that can be applied early in the development cycle, and then adjust appraisal activities as quality improves. Specifically, the authors recommend conducting a root/cause analysis of defects found during initial development and use the 80/20 rule to identify the defects for which prevention and appraisal practices should be implemented. In Acme’s development process, appraisal activities (functional testing, interoperability testing, performance testing, installation testing, internationalization testing, regression testing, fixed re-verification period and code inspection) dominate prevention activities (requirements review and test-plan review). These researchers would probably advise Acme to adopt prevention activities and re-evaluate and adjust its appraisal activities.

With that in mind, I needed to determine which quality activities, especially prevention activities, are most cost-effective. Studies on open-source software development projects and commercial projects at Internet companies analyze quality practices (prevention and appraisal) and process characteristics. Specifically, in “How Internet Companies Build Software” (MIT Sloan Management Review, Winter 2001), a study of 29 projects from 17 companies, Harvard Business School professor of technology and operations management Alan MacCormack discovered that three practices significantly contribute to software quality improvements and the ultimate success of software projects. In particular, he found that one practice—the early release of a low-functionality product to the customer—explains more than one third of the variation in product quality across all projects. The successful implementation of this practice correlates with a careful selection of and working relationship with beta customers.

A second practice responsible for significant quality improvements is a daily build in conjunction with regression tests providing rapid feedback on design changes. MacCormack’s study reveals that regression tests were necessary, but not sufficient in themselves to guarantee software quality. Finally, a third practice—high-level investment in architecture and design—also shows a strong association with high-quality products. The author argues that an investment in a modular and scalable architecture favors the early-stage release of a low-functionality version of the product. On the other hand, the early-stage release of the product also provides feedback on the investment in the architecture and design. Thus, the combination of a prevention activity—investment in architecture and design—and two appraisal activities—early release of a low-functionality product, with daily builds in conjunction with regression tests—can significantly improve software quality.

Open Source and Quality
Similarly, in “High Quality and Open Source Software Practice,” a position paper delivered at the May 2002 Open Source Workshop at the 24th International Conference on Software Engineering (http://opensource.ucc.ie/icse2002/HalloranScherlis.pdf), Carnegie Mellon computer science professors T.J. Halloran and William L. Scherlis analyze the software quality practices of 11 open-source software projects. The contrast of high visibility through unrestricted access to the source code, engineering practices that limit code commit privileges and required code reviews, distinguishes open-source software development from commercial software development. Engineering practice policies define how these assets, residing on a server, are controlled through people and tools—predominantly appraisal practices, such as code reviews, daily builds with regression testing and portability testing. In contrast, the practices on the developer side are mainly preventive in nature, including coding standards, well-defined roles and decision-making rules. Following these findings, Acme may want to continue with their coding standards and add documentation of particular release procedures for each project, making them open to all stakeholders.

What to Do?
Given the results from these studies, what specific recommendations will I give Acme, and what role can my client play in implementing them? Not allowed to change Acme’s software development process, my client is forced to operate under several constraints. However, it can choose to introduce prevention practices, such as design reviews, during the development phase. It can also document project architecture and publicize project release procedures. These prevention activities may reduce the effort currently scheduled for functional testing and regression testing after code completion. Later in the process, my client may be able to convince Acme to add regression tests following each product build. In the long term, it may be possible to break the development phase into cycles, hiring a domain expert to test for suitability of the functions provided and a QA specialist to test performance after the completion of each development cycle.

If my client decides it’s advantageous to add prevention and appraisal practices to the development phase, it needs to record the time and effort spent following Acme’s traditional quality assurance phase, to determine if the actual time and effort spent is less than the time and effort estimated based on Acme’s past schedules for the QA phase. My client should also record data to calculate defect removal efficiency to evaluate the efficacy of the level of software quality achieved by the changes in the software process and quality activities. In turn, my client must convince Acme to do the same.

Update: What We’ve Accomplished
Even in the absence of historical data and time for a pilot project, studies on software cost and quality can help you make educated choices for change in your development process. At Acme, my client’s presentation of results from these studies convinced management to introduce prevention practices during the development phase, adjust the company’s appraisal practices by adding regression tests to the daily build and adopt an iterative development process in the long term. Paying attention to research created the opportunity for significant improvement in both software quality and cost-effectiveness.

Ulla Merz is a consultant, managing software projects for software product companies, such as Rational Corp. and providing training in Microsoft Project. She holds a Ph.D. in computer science and is PMP certified. Contact her at ullamerz@aol.com or visit www.ullamerz.com

RELATED ARTICLES JDBC Fast Connection Failover with Oracle RAC Micro Focus Rolls Out Visual Studio 2010 Support Looking For The Lost Packets: Part 2 Coarse-Grained Unit Testing Gurock Software Releases TestRail 1.0		TOP 5 ARTICLES No Top Articles.