Measuring Java Performance

I finished the demonstration, mentally making a note to buy a round for the engineering staff for getting the demo up and running two days after having moved our offices. I could see that the man sitting across from me, a founder of one of Japan's largest electronics concerns, was formulating a question. "Your architecture is Java," He said, "I am concerned about the performance." I should have guessed.

If you are at all involved with software development in Java, you are aware that doubts about Java's runtime performance are by far the most common objection to the language, outweighing such real concerns as deployment issues, incomplete specifications of critical technologies such as Enterprise JavaBeans, and availability of programmers. If you have a bit of experience in Java, you are probably aware that Java's performance shortcomings are largely mythical and that high performance applications in Java are not only feasible but relatively easy to create.

The myth of Java's poor performance originates from these three misconceptions:

• that Java is interpreted and therefore slow,
• that Java's safety features in the area of typing and memory management introduce large amounts of overhead, and
• that system performance is primarily a function of CPU opcode selection.

These criticisms, if valid, would point to inherent performance issues for programs written in Java. However, not only are none of them true, an understanding of modern hardware architecture and a survey of typical systems shows that Java is arguably a superior choice for high-performance applications; for instance, in a 24x7 n-tier application, Java's defragmented and compacted heap is likely to be both more robust and higher performance than a memory management scheme cooked up by the average C++ programmer.

This is not to say that there have not been plenty of Java applications with poor performance. To name three common culprits, the performance of Java's user-interface libraries has been a joke, the over-creation of logically transient objects causes performance bumps in an astonishing number of applications, and naive thread synchronization techniques can cause fundamental lapses in quality. However, there are cures for all of these problems, ranging from simple (use StringBuffers where appropriate) to reworking the system architecture to adhere to object-oriented design heuristics. Extracting the highest possible performance from a Java system requires a shift from viewing optimization as a low-level activity with a limited scope to a task with multiple levels of abstraction.

Figure 1. JProbe Profiler

Similar Capabilities

It was with this perspective in mind that I examined NuMega's DevPartner for Java and KL Group's JProbe Suite. Initially, both tools seem to provide a similar core set of capabilities: performance profiling and thread analysis for performance tuning and code coverage to aid in quality assurance. However, in side-by-side comparison, their radically different interfaces are a testament to fundamentally different philosophies.

The KL Group's JProbe Suite Professional Edition (Windows) 2.0 comes on a single CD-ROM with an option for installing fifteen-day trial versions or a serial-number-unlocked permanent version. While I was a little surprised to see a trial version in a shrink-wrapped box, I'm all for the concept of fully feature-enabled trial versions getting into the hands of potential customers, and if KL Group thinks a CD is a way to do that, more power to it. A full installation, including example files, weighs in at 30MB. Installation of NuMega's DevPartner for Java is similarly straightforward, with all options and example files consuming approximately 25MB of drive space. Similarities between the products pretty much ends there, though.

Profiling

Quantitative measurement of the behavior of a running program is the sine qua non of a performance analysis suite. To put the two programs through their paces, I measured a program that I knew to be particularly CPU-intensive: a voxelspace raycaster.

This renderer was originally written by Bob Troia of OliveMedia (www.olivemedia.com) and is based on C++ code developed by Andre Lamothe. Bob graciously provided me the code and permission to modify it for a series of lectures on optimizing Java ( http://www.altnetinc.com/training/berkeley/index.html). JProbe is a Java application; while that shows an encouraging confidence in the underlying technology, Swing-related display glitches are unfortunately common, sabotaging the very point. JProbe's user interface is complex, and while a wealth of information is available, I believe that it could be simplified without losing functionality.

Ironically, JProbe's greatest strength also lies in its user interface. The call graph is a great way to view not only CPU consumption, but the interrelationships between methods that cause that consumption (in this case, you can see that I would be well advised to expend effort optimizing Basic-Render.renderTerrain, which is called from oliveVoxel.singleFrame.

This is precisely the sort of information that can be used to inform a performance-oriented refactoring of the design, and is precisely the sort of information that can be difficult to tease out of a traditional profiler.

In comparison, the interface of NuMega TrueTime (Figure 2) seems positively spartan. The program runs; the Output window details the interaction between the Java application and the system (an area where TrueTime is particularly strong), and the Session window gives an instant summary of the most time-consuming methods (here, TrueTime makes it even clearer than JProbe that renderTerrain is the place to devote efforts).

Figure 2. NuMega TrueTime

Source Codes Just a Tab Away

Source code, with timings at a line-by-line level, are available just a tab away. TrueTime excels as a traditional profiler, providing traditional low-level information about a piece of software. As you'd expect from NuMega, the company that made its reputation creating tools that revealed the interaction of C++ programs with Windows, TrueTime's strengths emerge when working with mixed-language systems.

TrueTime worked perfectly when I reworked the renderer to use a C-based DLL (although you may be interested to know that on my Dell Dimension XPS T500, the Hotspot virtual machine rendered the pure Java algorithm in approximately 70 milliseconds per frame, while the C-based DLL only improved this integer-intensive operation to 63 milliseconds.)

The UI sophistication is reversed in NuMega's JCheck and KL Group's Threadalyzer. I tested these programs, whose purpose is to dynamically reveal conflicts in multithreaded code, with a program that creates a large number of threads, each controlling a single colored box on the display (some readers may recognize this as the "ColorBoxes" example from Bruce Eckel's Productivity Award-winning book "Thinking in Java," where it is used in a discussion of anomalous behavior in heavily threaded programs).

Worth the Money

In these programs, JCheck provides the more sophisticated UI, while Threadalyzer provided a concrete diagnosis of a race condition. I found these programs, however, to be less compelling in terms of delivering value. From neither did I learn anything fundamental about the behavior of the program that could lead me to improve its efficiency.

The argument for either of these two programs is that, if they deliver the goods even once, which is to say if they diagnose an elusive multithreaded bug, they're worth the money. It's hard to argue with that logic, since thread bugs are even more difficult to track down than are memory bugs in C.

The Bottom Line

For most programmers, DevPartner and the JProbe suite are not going to be day-to-day tools but are going to be broken out for intensive use during particular times of the development cycle. In the end, such a usage profile argues for DevPartner's less innovative, familiar interface over JProbe's more ambitious, but somewhat confusing interface.

However, JProbe is winning the race in terms of innovation and I look forward to the maturing of the KL Group's products. NuMega has long stood all-but-unchallenged in terms of tools that enhance the post-build development process. KL Group is clearly coming up fast in this area; the enhanced competition will do nothing but good for the software community.