Channels ▼


Improving Tests for Reliability

Making Quality Gates Effective

Despite these limitations, test suites are one of the most common and effective quality gates. The more you understand the benefits and blind spots of your test suites, the more you can compensate for their weaknesses. Many leaders in quality use a run-time error detection system while exercising their test suites. Run-time error detection systems are like the EKG the cardiologist uses. They monitor the application as it runs and detect defects that would otherwise go unnoticed during testing.

More Insights

White Papers

More >>


More >>


More >>

Test suites can consist of paper checklists that guide a human operator through the exercise of an application. However, it is more effective to automate test suites. Old-style applications without a GUI can be exercised in UNIX by piping various input data files to the application and using diff on the output. Generating input for GUI programs is far more difficult. One solution is to write an extension language for manipulating the application the way the GUI does, so that the test suite can drive the application using this language. Another strategy is to purchase a specialized record/playback/verify package that will record a user exercising the GUI, and later play the input sequence back while verifying that the same output is produced. Several such packages are on the market.

Quality gates are used traditionally at the end of development to verify correctness. Far more effective is to use quality gates throughout development. Quality rapidly improving at the end of development rarely happens. The problem is that if you change lots of code quickly (to fix many problems), you introduce more problems. Thus, the rate of quality improvement has an upper bound. In reality, you have only two choices: continuously apply the pressure of quality gates throughout development and meet your high quality goals, or wait until the end of development to get a modest surge in otherwise unacceptable quality.

The key to quality is erecting quality gates through which your software must pass. By teaching your organization to view software defects as quality-gate defects rather than code defects, you focus your organization on building strong gates. One of the best types of gates are automatic test suites. The difficulties in making test suites effective (lack of visibility, delay between cause and effect, working by accident) can be overcome, so do not let these limitations deter you. If you commit to building excellent tests, your software will reflect this commitment.

Reed Hastings is the CEO of NetFlix. At the time he wrote this article, he was the CEO of Pure Software, a company that specialized in developer tools. This article first appeared in 1993 in UNIX Review.

Related Reading

Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.