Making Quality Gates Effective

Despite these limitations, test suites are one of the most common and effective quality gates. The more you understand the benefits and blind spots of your test suites, the more you can compensate for their weaknesses. Many leaders in quality use a run-time error detection system while exercising their test suites. Run-time error detection systems are like the EKG the cardiologist uses. They monitor the application as it runs and detect defects that would otherwise go unnoticed during testing.

Test suites can consist of paper checklists that guide a human operator through the exercise of an application. However, it is more effective to automate test suites. Old-style applications without a GUI can be exercised in UNIX by piping various input data files to the application and using diff on the output. Generating input for GUI programs is far more difficult. One solution is to write an extension language for manipulating the application the way the GUI does, so that the test suite can drive the application using this language. Another strategy is to purchase a specialized record/playback/verify package that will record a user exercising the GUI, and later play the input sequence back while verifying that the same output is produced. Several such packages are on the market.

Quality gates are used traditionally at the end of development to verify correctness. Far more effective is to use quality gates throughout development. Quality rapidly improving at the end of development rarely happens. The problem is that if you change lots of code quickly (to fix many problems), you introduce more problems. Thus, the rate of quality improvement has an upper bound. In reality, you have only two choices: continuously apply the pressure of quality gates throughout development and meet your high quality goals, or wait until the end of development to get a modest surge in otherwise unacceptable quality.

The key to quality is erecting quality gates through which your software must pass. By teaching your organization to view software defects as quality-gate defects rather than code defects, you focus your organization on building strong gates. One of the best types of gates are automatic test suites. The difficulties in making test suites effective (lack of visibility, delay between cause and effect, working by accident) can be overcome, so do not let these limitations deter you. If you commit to building excellent tests, your software will reflect this commitment.

Reed Hastings is the CEO of NetFlix. At the time he wrote this article, he was the CEO of Pure Software, a company that specialized in developer tools. This article first appeared in 1993 in UNIX Review.