Results and Limitations of This Approach
There are a couple of issues that limit the use of numerical problem sniffers relying on the runtime instrumentation approach:
- False warnings: Suspicious numerical events are not always errors. For instance, hash functions or random number generators may be responsible for overflows that are in fact part of the algorithms. Many floating-point issues mentioned earlier can be considered anomalies or perfectly sound code depending on the context. Because of this, Cojac enables the programmer to discard Cojac in particular classes/methods with the help of Java annotations.
- Compile-time evaluation: Some constant expressions are evaluated early by the compiler and directly replaced by the result. This step leaves no chance for the tool to watch for anomalies during the evaluation. It is another consequence of the "semantic distance" problem between source code and compilation result mentioned in the preceding section — the compiler has considerable freedom.
- Debug symbols: If the symbols (function name, line numbers) are missing in the code to be instrumented, the warning messages lose much of their value.
Overhead
Predictably, instrumenting every arithmetic operation has an impact on the performance of the processed application. With Cojac, we designed experiments to estimate the slowdown caused by instrumentation. Our observations on individual operations and classic algorithms are summarized in Figures 3 and 4. Of course, it is hard to predict in general how the sniffer will impact the performance of a whole application, but our results suggest that the slowdown factor typically is around 2x, except for number-crunching applications where it can exceed 15x.
Figure 3: Estimated overhead per instruction for Cojac.
Figure 4: Estimated overhead in some code examples, for Cojac.
With Cojac-grind, the impact on the running time is greater, because the Valgrind framework more radically changes the way the application is executed. With the benchmark program nbench, we measured a slowdown between 6x and 60x (see Figure 5). The overhead is roughly similar to the well-known Valgrind Memcheck tool (from 5x faster to 3x slower than Memcheck).
Figure 5: Estimated overhead for Cojac-grind versus Memcheck, in some code examples.
These results suggest that the performance overhead of the numerical problem sniffing approach is quite acceptable.
Validation As A Diagnostic Tool
For the moment, we oriented our efforts toward the robustness of Cojac, ensuring that it can be used on any Java application. It has been verified on a wide range of different applications. The Cojac-grind prototype is available as is, but it is an ongoing development. So far, the functionality has been tested on only a couple of programs. Even though the tool has limitations and does not yet fully take advantage of the Valgrind machinery, we encourage interested readers to give it a try and share feedback. We plan to conduct a more in-depth evaluation in the future.
Conclusion
Raw number types come with limitations, which may create undesirable behavior. Ensuring that there is no window of input values that can open a numerical hazard, be it with integers or floating-point numbers, is a tedious business. Numerical problems are a plague for developers — numbers are omnipresent in code units and a program can be broken for very subtle reasons. In a sense, this is similar to the difficulty of managing dynamic memory with C pointers, and runtime memory checkers proved to be helpful in that context.
This is why we advocate the use of tools to highlight any suspicious numerical events at runtime, which might reveal bad practice, wrong hypotheses, or bugs. On-the-fly code instrumentation makes it possible to employ such numerical problem sniffers with no further constraints on the development process. Such easy-to-use tools can lend valuable assistance, particularly in the context of programmer education.
We created two open-source prototypes. Cojac is now a mature, freely available, highly customizable tool to systematically make observable every numerical problem symptom in any Java programs. Using Cojac is simple. It requires only a small addition on the command line used to launch the application. The current version of Cojac-grind is an experimentation prototype of a Valgrind tool and we are working on its improvement.
Dr. Frederic Bapst is professor at the University of Applied Sciences of Western Switzerland, Fribourg (Computer Science unit). Baptiste Wicht is currently obtaining his MSc in Computer Science at the same institution. He was previously a regular moderator of a public Java forum.
