|
A C++ utility that implements static testing automation of libraries
January 16, 2008
URL:http://www.drdobbs.com/tools/static-testing-c-code/205801074
Assume you have just written a beautiful function library. You could have used any programming language, but the library appears to other programmers as a sequence of functions callable by a program written in C (or in languages that conform to C-like function calling conventions). But before delivering it to users (who are application programmers themselves), you want to be confident about the library's correctness. Consequently, you write a program that uses the library. However, that program presumably won't invoke every function in every possible way. So to guarantee reasonable quality, you need to write a testing program that invokes the library functions, passing a variety of possible parameter values and checks if the outcome matches expected results.
The testing program can be written in a different programming language, provided it interfaces with the library's functions. Because a buggy function could crash a program, you could divide the testing program into several small programs and launch them sequentially from a script. If a program then crashes, the script can proceed to the next program.
Now assume that your library isn't a simple C function library, but rather a library that exports functions or classes usable only by C++ code. In addition, exception-handling issues must be considered. Actually, the C++ functions to test could raise exceptions, both expected and unexpected. The specification of every function can (and should) declare which exceptions may be raised while executing such functions.
For example, if I call a function passing as a parameter the name of a nonexistent file, I could expect that an exception of a specific type is raised by that function. And if I pass the name of an existing but unreadable file, an exception of another type is expected. Finally, if passing the name of a readable file, no exception is expected. Here is the code:
try {
process_file("inexistent_file");
} catch (file_not_found_exception &e) {
success();
} catch (...) {
failure("unexpected exception");
}
try {
process_file("read_protected_file");
} catch (file_unreadable_exception &e) {
success();
} catch (...) {
failure("unexpected exception");
}
try {
process_file("readable_file");
} catch (...) {
failure("unexpected exception");
}
To properly test the library's exception-raising feature, every function call in the testing program must be put in a try block. After the calls for which a specific exception is expected (file_not_found_exception or file_unreadable_exception), a catch block for the expected exception type must be put. And for every call, a catch-all block must be put after the possible specific catch block to ensure that every unexpected exception is handled.
There are several open-source frameworks—Boost.Test, CppUnit, CppUnitLite, NanoCppUnit, Unit++, and CxxTest, among others—that facilitate the development of automatic testing programs of C++ modules.
A testing issue rarely considered by textbooks and testing frameworks is this: "Are you sure that in using your library, the application program will be correctly compiled and linked when it should?"
When you use libraries written in C, this isn't a big issue—library header files clearly define which functions are exported by the library, and it is always clear which function is called by a statement of the testing program. Therefore, in the testing program every library function is called at least once. If you can smoothly compile and link the testing program, then you can be confident that build problems should not arise for library users. This is a reasonabe assumption of developers who create libraries for C, Pascal, Fortran, or other languages according to the interfacing standards.
In fact, even in those languages, there can be problems regarding name collisions. With C++, these problems are avoided by using namespaces. Nonetheless, some complications arise in C++, mainly due to templates and overloaded functions.
In C++, there can be several functions with the same name—the "overloaded" functions. For example, the Standard Library contains the functions double pow(double, double) and float pow(float, float), in addition to other overloaded versions. The first function performs the exponentiation of a double number by a double exponent; the second version does the same operation, but with a float base and a float exponent.
The expression pow(3., 4.) invokes the function double pow(double, double), while the expression pow(3.f, 4.f) invokes the function float pow(float, float).
Yet, the expression pow(3., 4.f) is illegal, as is the expression pow(3.f , 4.), because such expressions are considered ambiguous by the compiler. The ambiguity comes from the fact that there is no overloaded version exactly matching, and the attempt to implicitly convert the parameters to match an existing overloaded version leads to several existing overloaded functions, and the compiler can't decide among the candidate versions.
The rules to bind a function call to the corresponding overloaded function are rather complex, and the situation is often complicated further by implicit conversions. Actually, a constructor that gets only one parameter and that is not preceded by the implicit keyword lets the compiler perform an implicit conversion from the parameter type to the type of the class containing that constructor. A type-conversion operator lets the compiler perform an implicit conversion from the class containing that operator to the type declared as operator.
With such grammar, it's sometimes difficult for programmers to forecast which function will be chosen by the compiler when compiling a function call expression, or if the compiler will surrender at an ambiguous expression.
Although the ambiguity lies in the application code, not in the library code, a good library should be designed so that it is unlikely that typical application code will be ambiguous.
Therefore, there is a need to test the usability of a library, checking that unreasonable usage is forbidden and reasonable usage is allowed. This can be done by preparing a series of typical uses of the library, some reasonable and others unreasonable, and to verify that every reasonable use of the library can be compiled and every unreasonable usage is spotted by the compiler as an error.
In addition to being able to create function libraries and class libraries with C++, you can create libraries containing function templates and class templates, leaving the task of instantiating such templates to the application program.
Often, the actual template instantiation parameters are not foreseeable by the library developers. They can only specify in the documentation that those parameters must satisfy certain requisites. If the templates are instantiated with parameters that do not respect those requisites, the compiler emits error messages.
For example, the Standard Library contains the function templates min and max, and the class templates vector and list.
The function template min can be instantiated with a numeric type or with the string type, but not with the complex<T> type, for any T.
The class template vector can be instantiated with any type if the only goal is to create an empty collection. However, if you want to call the resize member function of that collection, the element type must have a default public constructor or no constructor at all.
When designing a library, you should consider not only the features you want to provide for application programmers, but also the programming errors that you want to prevent.
Actually, sometimes the purpose of a specific feature of a library is not so much to provide a functionality to users as it is to prevent programming errors by forbidding error-prone operations. For example, a time instant and a time span are different kinds of entities. You can add two time spans and you can multiply a time span by a number.
However, you cannot add two time instants nor multiply a time instant by a number. You can subtract a time instant from another time instant, obtaining the time span between them, and you can sum a time span to a time instant, obtaining another time instant, following the former by that time span.
If a library defines a class for time instants and another class for time spans, it should allow only the operations that make sense for every class. The application code that would contain disallowed operations shouldn't be accepted by a compiler.
The most advanced and sophisticated C++ libraries define recursive templates in a programming style called "template metaprogramming." In this context, a logical programming error (almost) always generates a compilation error. In general, the wrong usage of forbidden expressions should, if possible, result in compile-time errors. To this purpose, some libraries (Boost and Loki, for instance) have defined macros to declare compile-time assertions.
If a program using the Loki library contains the statement STATIC_CHECK(3 == 4) when it is compiled, an error is generated. The same happens if a program using the Boost libraries and containing the statement BOOST_STATIC_ASSERT(3 == 4) is compiled.
Such statements, similar to the assert macro from the C Standard Library, allow the compiler to analyze a constant expression, and if the expression comes out as false, they generate a compile-time error.
The assert macro is useless in production code. In fact, in addition to being valuable for internal documentation, its main purpose is to be evaluated in a testing run. In the same logic, Loki and Boost static assertions are most useful while evaluated in a testing compilation.
But what is a testing compilation? It is a test consisting in compiling a test program written for the sole purpose of determining if the compilation is successful or not.
Let's call such a program, whose syntactic correctness is to be checked, a "static testing program." Furthermore, let's call a traditional program, surely syntactically correct, whose runtime correctness is to be checked, a "dynamic testing program."
A dynamic testing program contains a series of dynamic tests, each one of them declaring the result expected from the evaluation of an expression while running the program. Analogously, a static testing program contains a series of tests, each one of them declaring the result expected from the compilation of an expression; that is, whether the expression is syntactically correct (legal) or incorrect (illegal).
The static testing consists in trying to compile each test and observe if any compilation error is output.
A test is considered failed, meaning that an error in the library (or in the test) has been detected, if a code snippet declared as legal generates some compilation errors, or if a code snippet declared as illegal doesn't generate any compilation errors.
Conversely, a test is considered successful, meaning that the behavior has been the expected one, if a code snippet declared as legal doesn't generate any compilation errors, or if a code snippet declared as illegal does generate some compilation errors.
For exception-specification testing, it is possible to specify that a certain operation should cause the raising of a specific exception. Instead, it is not feasible to specify that the compilation of a certain statement should cause a specific compilation error, as compilation errors are not standardized, and vary even with the versions of the same compiler. Therefore, we will be satisfied in distinguishing between legal code and illegal code.
Assume you want to test the legality of 10 statements. If you put them all in the same program, you face the following difficulty: If the program can be compiled with no errors, then you infer that all 10 statements are syntactically correct. But if the compilation generates errors, it is difficult to ascertain which of the statements are syntactically incorrect. It could even be that all the statements are correct when considered one at a time, but their coexistence in the same program is incorrect.
A better method would be to:
This approach is effective, but laborious for users. Therefore, it is convenient to automate it.
For such purposes, you can insert the 10 statements in the same program, and submit the source code to a utility that generates the 10 versions of the source code, one for every statement to test. Of course, to let the utility know which statements to test, those statements should be adequately marked. In fact, they generally aren't simple statements but code snippets containing one or more statements.
In addition, such marking must specify whether the pointed-out code snippets are declared as legal or as illegal. Listing One is an example of a static-check program containing four code snippets. Three markers are used (nesting is not allowed):
LEGAL marks the beginning of a code snippet declared as legal.ILLEGAL marks the beginning of a code snippet declared as illegal.END marks the end of the current code snippet.
The example program includes statements common to every test. After these statements, there is a block of two statements declared as legal, the declaration of a vector of complex numbers, and a call to the min function with two integer numbers as parameters. These statements are actually legal.
Listing One
#include <complex>
#include <vector>
#include <algorithm>
#include <iostream>
int main() {
@LEGAL
std::vector<std::complex<double> > v;
std::min(2, 3);
@END
@ILLEGAL
std::vector<std::complex<double> > v;
sort(v.begin(), v.end());
@END
@ILLEGAL
std::min(2, 3.);
@END
@LEGAL
int i;
std::cout << i;
std::vector<int> vi;
vi.push_back(3);
vi.push_back(-2);
vi.push_back(0);
sort(vi.begin(), vi.end());
@END
}
Afterwards, there is a block of two statements declared as illegal. In fact, only the second statement is illegal, as it tries to sort complex numbers. The v variable is declared twice. This is not an error, as when the first code snippet is active, the second code snippet is inactive, and vice versa; therefore, the two declarations do not coexist.
Next, there is a code snippet containing a single statement that is declared as illegal. It is really illegal because it tries to compute the lesser between an int value and a double value.
Finally, there is a code snippet of seven statements, declared as legal. Actually, this code snippet does not contain syntax errors, but the second line prints the value of a variable that has never received a value. In such a situation, many compilers emit warnings if the needed compiling options are set.
This raises another issue: If a code snippet declared as legal causes the compiler to emit a warning, is the test to be considered successful or failed? And if a code snippet declared as illegal causes the compiler to emit a warning but no errors, is the test to be considered successful or failed?
In a way, in both cases the test is partially successful. Consequently, there is a third possible outcome of a test: In addition to failure and success, there can be a warning.
I present here a utility that implements static testing automation. The utility is a standard C++ program that retrieves from the command line the paths of the files containing the static testing programs, and accesses some configuration environment variables. (The source code is available here.)
The program reads the files to process, and for each one (and for every marked snippet contained in the file), it generates a new source file whose syntactic correctness is then checked.
To check the syntactic validity of a statement, the most efficient technique would be to use a library providing a compilation service or a static-analysis service. Such libraries do not exist for C++ (or they have little popularity). Alternatively, a compiler can be launched. If a compilation is successful, the process usually sets the return code to zero; if it fails, it sets the return code to another value. So, it is enough to execute the compiler, passing to it the desired options and the source code to check, wait for the completion of the compilation, and check the return code of the process. This is what this utility does.
Some compilers allow an option that lets you perform only a syntax check on the code without generating the executable code, thereby speeding up the compilation. A further speedup can be obtained by disabling code optimization. Yet, some warnings are not output if a simple syntax check is performed or if code optimizations are disabled. Therefore, it is advisable to actually generate the optimized executable program, even if it is never executed. Additionally, there isn't a standard technique to launch a compiler—every compiler has its incompatible options. Therefore, I put the dependencies from the compiler and the platform into environment variables.
Included with the utility's download package are two scripts—one for the Windows command interpreter (CMD.EXE) that tests the program with the Visual C++ and GCC compilers, and the other for UNIX/Linux shells that tests the same program with the GCC compiler. Figure 1 is the execution output of the script for Windows.
==== Visual C++ test ==== Processing test file 'example.cpp' Running "cl /nologo /GX /Za /Ox /W4 /WX _test1.cpp 2>NUL >NUL" OK: Compilation with no errors nor warnings, as expected. Running "cl /nologo /GX /Za /Ox _test2.cpp 2>NUL >NUL" OK: Compilation with errors, as expected. Running "cl /nologo /GX /Za /Ox _test3.cpp 2>NUL >NUL" OK: Compilation with errors, as expected. Running "cl /nologo /GX /Za /Ox /W4 /WX _test4.cpp 2>NUL >NUL" Running "cl /nologo /GX /Za /Ox _test4.cpp 2>NUL >NUL" *** WARNING: Compilation with no errors but some warnings, while a compilation *** with no errors nor warnings was expected. Failures: 0, warnings: 1, successes: 3 One test file processed. .......... Total failures: 0, total warnings: 1, total successes: 3.
Figure 1: Execution output of the script for Windows.
Carlo is a CAD/CAM software developer in Bergamo, Italy. You can contact him at [email protected].
Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.