Unit Testing as the Last Resort for Code Verification
November 07, 2009
I love code which is stable and rigorous, I just don't feel that unit testing is the most effective way to achieve it.
I love code which is stable and rigorous, I just don't feel that unit testing is the most effective way to achieve it.
I stumbled upon the following question posted on StackOverflow, "What's your most controversial programming opinion? ", and it made me think about my views on unit testing.
I have mixed feelings on unit testing. I do use it often, but I find that it does not give me sufficient peace of mind, compared to other code verification techniques. The following is my list of code verification techniques in order of importance:
- System testing
- Use the compiler's type checker
- Use assertions
- Code reviews
- Unit testing
First, any code verification technique is pointless without system testing. While it may seem almost obvious to include, it does happen where people get caught up in the minutiae of code verification and overlook the big picture of whether or not a system is behaving as intended.
My preferred code verification technique is to leverage the language's type system to express assumptions, requirements, and invariants. It is nice to know that if something compiles, then it is correct. Some examples of how this can be done are:
- using unsigned types instead of signed types so you don't have to worry about non-negativity. Consider for example the example of using an unsigned type instead of a signed type in a square root function.
- using id types instead of integers to identify objects. I recently ran into problems in my code, because I violated this rule. I confused the index of a vertex, with the index of an index array into the vertex array. The fix was easy: use a new VertexID type.
- using smart pointer classes with the appropriate semantics. E.g. null-checking pointers, deletion prevention pointers, pointers with ownership semantics, etc.
In this bucket I also include any static contract verification tools like those found in Spec# and Eiffel. This is because the techniques of type checking and contract verification are closely related, and both based on theorem proving techniques.
When the type system can't be used easily, I find that the next most powerful code verification technique is to use assertions to check assumptions, requirements and invariants at run-time. Assertions turn regular system testing into a much more powerful tool for uncovering design errors and potential defects. In addition assertions occur directly in the code, as a kind of documentation that is useful during code reviews.
Code reviews are another technique which I find is more powerful than unit testing, if it is done properly. In order to really leverage the power of a code review, code has to be written so that verification is as simple as possible. I will often sacrifice performance and ignore any popular idiom of the week if it means making my code's meaning obvious and unambiguous.
When I find unit testing to really become valuable as a tool, is when code can't be proven correct by the compiler or fully verified through assertions, and that a code review can't give a high confidence that the code is correct because of inherent complexity.
