Channels ▼
RSS

Testing

Improving Tests for Reliability


If you have seven years to develop and test a software product with a fixed specification, then quality is readily achievable. This article, however, concerns practical software quality for software professionals who operate under less ideal conditions. I outline a philosophy that centers on quality gates and analyze how to make quality gates as effective as possible.

More Insights

White Papers

More >>

Reports

More >>

Webcasts

More >>

The key to measuring and achieving software quality is developing tests, known as quality gates, through which your software must pass. The tighter the gates, the higher the software quality is guaranteed to be. Quality gates are built up over time and become as valuable as the product itself.

Quality gates take various forms. They can be processes (two peers must review and sign off on every change before check-in), results from beta trials (90% of beta sites will rate the software as defect-free), or actual tests run either manually or automatically. To be a gate, success or failure has to be measurable before release. Retrospective measurements, such as the number of defects per 1,000 lines of code, are not quality gates.

Most software professionals think that defects exist in software because of bugs in the code. A more productive perspective is that defects exist because of bugs in the quality gates — that is, quality gates were not designed or applied sufficiently well to trap all errors. The central idea in this view is that bugs do not exist in code; rather, they exist only in quality gates.

In other words, code is never correct or incorrect. It is only compliant or noncompliant with the quality gates. To fix a defect, the quality gates are improved to detect the defect. Eventually, the code is made compliant with the tightened quality gates, but fixing the code is secondary to fixing the quality gates.

Some of the best quality gates are test suites that verify that, given appropriate input, the expected outputs are produced. Many organizations put enormous amounts of work into their test suites, but they are not always as effective as they could be. Three reasons account for this ineffectiveness.

1. Lack of Visibility. A software tester has very little visibility into the internal state of a system. In fact, today’s software testers are like cardiologists without an EKG. They exercise the patient as hard as possible, and if the patient keels over (core dumps), the tester concludes there must have been a problem. What the software tester lacks is an EKG-equivalent that can detect subtle conditions indicating future trouble. In software, examples of such conditions include a hash table with duplicate entries or a doubly linked list that is not correctly linked. In general, a software tester needs to be able to run a test exercising part of an application, followed by extensive application-specific internal consistency checks. You can combat the lack of visibility into software by requiring applications writers to provide external hooks (environment variables, functions) that perform extensive internal consistency checks that the software tester can use after each test.

2. Delay Between Cause And Effect. Many software defects do not cause immediate visible symptoms, but if the program continues to be tested, the symptom may appear.

In Figure 1, each branch represents a different code path. For example, the initial three-way branch might be the New, Open, and Close menu entry code. In this example, a bug is triggered by the open code, but no symptom appears. Only later, perhaps when the file is accidentally re-opened by the user, will a symptom (crash) appear. Certainly this delay between cause and effect makes tracking causes difficult, but more importantly, it means test suites must be sophisticated enough to trigger an error, and they must go on to trigger a symptom. The combinatorics of exercising code are bad enough, but the extra work needed to trigger symptoms in addition to errors makes the job nearly impossible.


Figure 1: The delay between cause and effect.

3. Working By Accident. Many software programs work only by good fortune and luck. Much of the time, that luck holds and users remain content. A classic example of working by accident can be seen in the following example:

char* x = malloc(100);
fill_and_sort(x, 100);
if (x[100] == ERRVAL) abort();

On the third line of this code fragment, the memory location x[100] is being read and compared against the constant ERRVAL. The problem, of course, is that x[99] is the last valid memory location of x, and that x[100] has uninitialized random contents. Most of the time (good luck), the memory x[100] is not equal to ERRVAL, and the call to abort() is not performed. Sometimes (bad luck), the memory x[100] happens to be equal to ERRVAL, and the program core dumps. Note that no memory is being corrupted in this example, and the good-luck case has no lingering effects of the defect. Thus, a program that is working by accident produces correct results, and the application is entirely internally consistent afterwards, yet this program will fail periodically in the field.


Related Reading






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video