Channels ▼
RSS

Tools

Project of the Month: Cojac, A Numerical Problem Sniffer


Results and Limitations of This Approach

There are a couple of issues that limit the use of numerical problem sniffers relying on the runtime instrumentation approach:

  • False warnings: Suspicious numerical events are not always errors. For instance, hash functions or random number generators may be responsible for overflows that are in fact part of the algorithms. Many floating-point issues mentioned earlier can be considered anomalies or perfectly sound code depending on the context. Because of this, Cojac enables the programmer to discard Cojac in particular classes/methods with the help of Java annotations.
  • Compile-time evaluation: Some constant expressions are evaluated early by the compiler and directly replaced by the result. This step leaves no chance for the tool to watch for anomalies during the evaluation. It is another consequence of the "semantic distance" problem between source code and compilation result mentioned in the preceding section — the compiler has considerable freedom.
  • Debug symbols: If the symbols (function name, line numbers) are missing in the code to be instrumented, the warning messages lose much of their value.

Overhead

Predictably, instrumenting every arithmetic operation has an impact on the performance of the processed application. With Cojac, we designed experiments to estimate the slowdown caused by instrumentation. Our observations on individual operations and classic algorithms are summarized in Figures 3 and 4. Of course, it is hard to predict in general how the sniffer will impact the performance of a whole application, but our results suggest that the slowdown factor typically is around 2x, except for number-crunching applications where it can exceed 15x.


Figure 3: Estimated overhead per instruction for Cojac.



Figure 4: Estimated overhead in some code examples, for Cojac.


With Cojac-grind, the impact on the running time is greater, because the Valgrind framework more radically changes the way the application is executed. With the benchmark program nbench, we measured a slowdown between 6x and 60x (see Figure 5). The overhead is roughly similar to the well-known Valgrind Memcheck tool (from 5x faster to 3x slower than Memcheck).


Figure 5: Estimated overhead for Cojac-grind versus Memcheck, in some code examples.


These results suggest that the performance overhead of the numerical problem sniffing approach is quite acceptable.

Validation As A Diagnostic Tool

For the moment, we oriented our efforts toward the robustness of Cojac, ensuring that it can be used on any Java application. It has been verified on a wide range of different applications. The Cojac-grind prototype is available as is, but it is an ongoing development. So far, the functionality has been tested on only a couple of programs. Even though the tool has limitations and does not yet fully take advantage of the Valgrind machinery, we encourage interested readers to give it a try and share feedback. We plan to conduct a more in-depth evaluation in the future.

Conclusion

Raw number types come with limitations, which may create undesirable behavior. Ensuring that there is no window of input values that can open a numerical hazard, be it with integers or floating-point numbers, is a tedious business. Numerical problems are a plague for developers — numbers are omnipresent in code units and a program can be broken for very subtle reasons. In a sense, this is similar to the difficulty of managing dynamic memory with C pointers, and runtime memory checkers proved to be helpful in that context.

This is why we advocate the use of tools to highlight any suspicious numerical events at runtime, which might reveal bad practice, wrong hypotheses, or bugs. On-the-fly code instrumentation makes it possible to employ such numerical problem sniffers with no further constraints on the development process. Such easy-to-use tools can lend valuable assistance, particularly in the context of programmer education.

We created two open-source prototypes. Cojac is now a mature, freely available, highly customizable tool to systematically make observable every numerical problem symptom in any Java programs. Using Cojac is simple. It requires only a small addition on the command line used to launch the application. The current version of Cojac-grind is an experimentation prototype of a Valgrind tool and we are working on its improvement.


Dr. Frederic Bapst is professor at the University of Applied Sciences of Western Switzerland, Fribourg (Computer Science unit). Baptiste Wicht is currently obtaining his MSc in Computer Science at the same institution. He was previously a regular moderator of a public Java forum.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video