Reigning in C++ Test Harnesses

If you're a science fiction reader, you've probably seen plenty of robot stories. Usually the robots wind up doing scut work for the humans, and many authors-Asimov and Heinlein in particular-explore the moral consequences of building metal slaves. If you root for the robots in these stories, beware C++Test. When your silicon slave gets stuck with building test harnesses for classes, creating test cases, assembling them into an executable and endlessly iterating through them, you may feel a twinge of an overlord's guilt.

Or not. Hey, they're not sentient yet!

Parasoft's C++Test is a unit-testing tool that promises to analyze your C and C++ code, automatically build test harnesses for functions and class methods, run the tests and help you analyze the results. It currently works with Microsoft Visual Studio 6.0 on Windows; Parasoft says that Linux and Solaris versions are in the works.

"Unit testing" is about pieces-individual classes, functions and methods. The goal of unit testing is to show that each piece performs as intended and exhibits no pathological behavior when presented with unexpected input. "Integration testing," on the other hand, is designed to ensure that the pieces fit together as designed and that the system as a whole behaves correctly under all reasonable conditions. It's what most of us think of when we think of software testing: QA engineers in a room, wringing out the finished program by trying all its features and supplying novel inputs.

Unit testing is important-if the bricks are shoddy, you won't have much of a house, regardless of how well they're laid-but it tends to be slighted in the real world, for a variety of reasons:

First, unit testing isn't sexy. When you write a test harness for a class, you're not adding features or killing bugs. It's just infrastructure.

Second, unit test harnesses and stubs are a pain to maintain. They're as bad as documentation: When a class method changes, so must your harnesses and stubs. Mind this every time, or they quickly become worthless.

Third, programmers are blind. OK, maybe that's a bit harsh, but we all tend to overlook the potential problems in our beautiful code.

Fourth, programmers write bugs into everything. Think about it-if you could guarantee bug-free code, you wouldn't need to test in the first place. How do you know that your testing code is itself free of bugs?

Parasoft says they're going to help us with all that. How does their product measure up?

Using C++Test

Installation is completely straightforward. You'll want to do it during Parasoft's business hours, though. When you run the program for the first time, a dialog box appears displaying a "machine ID" string and prompting for a password, which Parasoft will generate for you when you tell them the machine ID.

If you happen to be building a development workstation from scratch, install Visual Studio first, so that C++Test can find it and install itself into the Microsoft Visual C++ GUI.

The documentation contains a tutorial that walks you through most of the program's concepts and features; I recommend starting that way. Once through the half-hour tutorial, I was able to start using the product right away, though exploring all the features took much longer. There's no new language to learn, and not much jargon, other than Parasoft's specific use of such terms as "white-box testing." When you open the application, you see the usual menu and tool bars, a file-list pane, a symbol-tree pane, an output and edit pane, and the "working" pane, which has tabs for source code, test progress, results, test case editing, stub tables and suppressions.

Figure 1. Parasoft's C++ Test GUI

When you open the application, you see the usual menu and tool bars, a file-list pane, a symbol-tree pane, an output and edit pane, and the "working" pane, which has tabs for source code, test progress, results, test case editing, stub tables and suppressions.

Usually you'll start by opening a source file off the menu or the standard toolbar icon. A somewhat quirky file chooser appears. Note that it's not a standard Windows or Swing dialog (Parasoft says their multiplatform plans led them to write their own.) I grumbled more at this dialog than anything else in the program. You can also build and run tests right out of Microsoft Visual C++ (MSVC), which obviates the problem.

When you click the "Build Test" button, C++Test uses the Microsoft C++ compiler to parse the program and build an executable containing a test harness for it. For each of the methods in the class, it builds a set of test cases, permuting and combining the possible values of the parameters, pre- and postconditions (which refer to the state in which member variables and globals start and are left in) and the function's return value. For the fairly simple method setParameter, the program generated quite the boatload of test cases. Tote that barge! Lift that bale! Hah!

Once the file has been parsed, you can use the symbol-tree pane to limit what you're working with in the working pane. For instance, in the symbol pane I clicked the setParameter method, then clicked "Test case editor" to generate the test case list. You can also generate or run tests for a single class or method with this technique.

When you've built a test executable, you can run it by clicking the "Record" or "Play" buttons. To quote from the user manual: "Record automatically generates test cases for the selected file, class or method, then executes the automatically generated test cases. Play executes all test cases that are currently available in the Test Case Editor; it does not automatically generate any test cases."

As the test runs, it iterates methodically (robotically?) through the classes and methods. A code-coverage window opens, indicating each line as it's executed along with running totals. There's also a set of progress indicators that appear in the output and edit pane. Annoyingly, the program does not signal you when a test is done; the indicators reach 100 percent and the coverage window stops jumping around, but that's it. Since a test suite on a do-nothing Microsoft Foundation Class application took over two minutes on my machine (a 650-MHz Athlon with 256 MB of RAM), I quickly tired of staring at the progress bars. A minor bug: the cursor did not change back from the Windows "Busy" to "Normal" until I moved the mouse. This worsened the confusion over whether the program was done or not.

The results of a test run are available in two formats: a listing in the "Output" tab of the output and edit pane, and a condensed tree format in the working pane. (Unfortunately, I saw no easy way to correlate between panes.) Note the red lines in the tree: They delimit failed tests. The generated test expected a return value of -MININT, but got 0. In this particular case, it might not really be a bug; you can control the program's expectations in this respect (see "Advanced Features" below).

Advanced Features

It's all very well to have tests and results generated for you, but you still have to apply the little gray cells-after all, that's what keeps us ahead of the machines. C++Test offers several advanced features to make things as painless as possible.

Once C++Test has formulated a set of tests for you, you should walk through them one by one, verifying that the conditions for success are indeed correct. For example, the method's return value should be appropriate to the inputs, and any side effects (members or globals changed) should be listed.

It sounds like drudgery, and it is. But it's a lot quicker to do with a tool than by writing code!

Once you have a set of test cases validated, C++Test keeps them around so that you can do regression testing. This ensures that any changes you make to the code have to clear the bar of all the old tests, as well as any new ones you generate. The old tests stay around as long as the method's signature stays the same.

Also, if your code makes calls to external routines, C++Test generates stubs for them. You can edit the stub return values, put in your own code, or even bypass the generated stubs and link in the "real" code from your library.

C++Test allows you to suppress any test, method or class from the test suite. And, as you might imagine, if you're testing a project of any size, the number of test cases quickly undergoes combinatorial explosion.

As I noted above, if you install it after MSVC, C++Test adds three buttons to MSVC's toolbar: Test Project, Test File and C++Test. Respectively, they load the current MSVC project into C++Test, then build and run tests; load the current file into C++Test, then build and run tests; and launch the C++Test application. If you start from the C++Test side, you can load an MSVC project all at once instead of doing it by individual files.

You can also attach the MSVC debugger to single-step through a method as it's being tested. (I wasn't able to get this to work on my system; Parasoft says that this was due to a Microsoft bug which is fixed for NT Service Pack 4.)

Parasoft did a nice job on test generation defaults. When C++Test generates tests for you, you can specify how exhaustive a job it does. Here we see that Boolean arrays should be filled with random values; you can also decide to generate test cases for each of the other values listed.

According to the documentation, if you purchase and install Parasoft's Codewizard product, C++Test can "automatically run every file it builds" through it. Codewizard performs static analysis on your code to catch common C++ errors. Additionally, as each file is tested, C++Test can run it through Insure++, Parasoft's run-time error-checking tool.

C++Test excels at catching certain kinds of errors:

• GIGO (Garbage In, Garbage Out) for functions. This is C++Test's bread and butter. Functions and methods will be exercised with a rich set of parameter combinations. But you'll want to identify "tripwire" values of your own and add test cases. For example, if your method does a conditional on whether an int input is >=4, you'd better add 3, 4 and 5 to the automatically-generated test cases.
• Uninitialized member variables. If you leave a member uninitialized, C++Test will try out a variety of interesting values for you-NULL, random, large and tiny values for pointers, for example.
• Code that doesn't get run. By inspecting the results of the "Coverage" window, you can tell at a glance if there are parts of your code that aren't getting executed. Of course this isn't necessarily an error, but it's a good clue that something isn't working quite as you intended.
• Heap overruns and wild pointers-but only if you're lucky enough to cause an exception with them, or you have Parasoft's Insure++ product installed.
• Newly-introduced errors. By accumulating test cases as it goes and rerunning them, C++Test can indicate when you've unintentionally introduced a new "undocumented feature."

Of course, it's a program, not a magic bullet. Here are some kinds of problems that C++ is not designed to find:

• Errors outside the paradigm. For instance, leaving a relational database in an inconsistent state is outside C++Test's scope. You can simulate some of this with stubs, but you're largely on your own when it comes to these errors.
• Infinite recursion (unless you also purchase Insure++).
• Failure to check return values from system and other functions. At least, not unless you go in and set up the test cases manually. For example, in fileIO.cpp (Listing 1), the failure to check the return value from fopen() is a bomb waiting to go off. The default-generated test case runs happily, assuming that ff is supposed to wind up null. It's easy enough to change the test case, you just have to remember to do it. (Then again, you were supposed to check the return value in the first place, right?)

<font face="monospace">#include <stdio.h>

class IOTest {
private:
	FILE *ff;
	char ioBuf[200];
public:
	IOTest() {
		ff = fopen("testfile","r");
		fscanf(ff,"%s",ioBuf);
		printf("%s\n",ioBuf);
	}
	~IOTest() {
		fclose(ff);
	}
};</font>

Documentation and Support

The manual is in HTML format, and includes the following major sections: a Quick Start guide, Essential Tasks (17 HTML pages), Optional Tasks (10 pages), the tutorial (7 pages), GUI help (16 pages) and a couple of pages on troubleshooting. As with most HTML help, it's slick if you know what you want, but troublesome to search. Their tutorial section got me up and running quickly, and I usually could puzzle out where to look for info on a particular feature or task. As for phone support, when I called Parasoft with a few technical questions, I got one hit and one miss. I had to leave a message once, and didn't get a return call for several hours; the second time, I got a live human being in one minute flat, and he knew exactly how to help me.

What I Liked and What Bugged Me

First off, the idea for the program is a real winner, and I applaud Parasoft for getting it so close to target, first time out of the gate. If unit tests can be more or less automatically generated, instead of code having to be written-and maintained-and debugged-your team has a fighting chance of actually accomplishing unit testing!

The tree views for symbols, test cases and test results are extremely useful. Being able to filter the results to show just failed tests helps a lot when you have hundreds of test cases displayed.

I'm guessing on this one, but I suspect that Parasoft's engineers "eat their own dog food." There are just too many features that make life nicer for the programmer using the tool! For example, I imagine the "Suppressions" tab arose when somebody got tired of waiting for the thing to chew through the entire project, time after time. It's a nice touch-and the program has quite a few of them.

Of course, every ointment has a few flies, and this program is no exception.

That dang file chooser! It often forgot where I was in the directory hierarchy; instead, I had to navigate down from C:\. The Windows Desktop wasn't part of the hierarchy; folders you pull to the desktop for quick access won't be in the chooser. Files and folders aren't alphabetized; they're just displayed in some whimsical order known only to the chooser. It wouldn't let me shift-click or control-click to select multiple files; though I discovered that if you simply click again and again, it did multiselection just fine. You can't double-click to select a file and dismiss the dialog; you have to click the file and then click "OK." And when I tried to choose a directory instead of a file, I almost went insane until I realized you had to navigate the left pane to the parent of the directory you want, then select the target directory in the right pane.

OK. OK. I'm calmer now. (Wipes foam from lips.)

I sure wish that the "Output" listing correlated easily with the tree view in the "Results" tab, or even somehow with the tree in the "Test case editor." Instead, you get a notation of what class and method is being tested-which quickly scrolls up off the screen-and a test number.

The program exhibited somewhat quirky behavior when I threw a couple of fairly simple torture tests at it (Listings 2 and 3). One recurses infinitely, the other allocates whopping amounts of heap space. I never got an error indication from C++Test; instead, the test seemed to go off into la-la land. C++Test would still respond to inputs (they employ timeouts to prevent the program from crashing), but I had to stop the test manually. You can purchase another program from Parasoft, Insure++, that is designed to detect these kinds of errors, but it's not exactly cheap.

<font face="monospace">class RecursiveNightmare {
public:
	RecursiveNightmare() {
	}
	recurse(int aDepth) {
		recurse(aDepth+1);
	}
	~RecursiveNightmare() {
	}
};</font>

<font face="monospace">class HeapBomb {
private:
	float *arr[200000];

public:
	HeapBomb() { };
	void iterate(int times) {
		for (int ii=0; ii<times; 
		ii++) {
			arr[ii] = new float[200000];
		}
	}
	~HeapBomb() {};
};</font>

Sometimes it's not clear how to navigate. For instance, in a tree view, sometimes you have to click the parent to highlight it, then click the "+" symbol to expand the tree. When creating a user-defined stub, I had to edit the code, dismiss the editor, deselect the routine, then select it again to enable the "user-defined stub" radio button.

The test case editor takes a little getting used to. Once I recognized that it builds its UI out of your program code, I could parse it better, but it would be nice to just have the return values and parameters plainly labeled as such. I wish the test cases were viewable-or even editable-in some format other than one-by-one viewing in the editor. I'll steal and modify a suggestion here from Larry O'Brien's review of Parasoft's Java product, JTest (Feb. 2000): it'd be nice if the test run output were in XML to allow easy import into databases, but it'd be heavenly if the test cases were! That way, you'd have them as additional documentation, could comment them and view them in bunches. They'd be easier to pass around, too.

Not a Robot Slave

I liked C++Test, and recommend its use. Their documentation emphasizes the advantages of catching bugs early, and I certainly agree that this tool can help. If you're in an Extreme Programming team, you're going to be writing test harnesses and stubs early on anyway; you might as well have your faithful robot do it for you.

While this program can be a great help, you can't simply point it at your code and go. You have to stay involved, validating test cases, coming up with new ones and analyzing where problems might slip by C++Test. In other words, don't treat it as your robot slave, but as your partner.

As for cost, it's a bit painful to shell out over twice the price of Visual Studio Enterprise. If you want to catch static code problems and run-time errors, you'll be paying for Insure++ and CodeWizard too. There's no question that C++Test will pay for itself in time saved, if you're committed to unit testing. If not ... you'll have to decide whether these preventable bugs cost your team enough time and credibility to make it worthwhile.