Examining Win32 C++ Compilers

Ron is author of the Tarma Simulation Framework and can be contacted at [email protected].

With all the hoopla surrounding the Internet, World Wide Web, and Java, it is easy to forget that C++ remains the dominant language for Windows-based application development. With that in mind, I'll examine here some of the leading C++ compilers for Win32 executables.

At the time of testing (third quarter 1996), these were the current versions of C++ compilers:

• Borland C++ 5.0/5.01, for DOS 16/32-bit, and Windows 16/32-bit applications.
• Microsoft Visual C++ 4.1 (superseded by 4.2), for Windows 32-bit applications.
• Symantec C++ 7.21, for DOS 16/32-bit, and Windows 16/32-bit applications.
• Watcom C++ 10.6, for DOS 16/32-bit, Windows 16/32-bit, OS/2 1.x to 3.x, and Novell NetWare applications and modules.
• IBM VisualAge for C++ for Windows 3.5, for Windows 32-bit applications.

The core functionality of these compilers and environments has changed little since then.

In addition to the compiler proper, all packages include a development environment with editors for text and resources, build tools such as Make and Link, debuggers, run-time libraries, and SDK tools. Furthermore, all the compilers include the Microsoft Foundation Classes (MFC), albeit different versions.

My focus here is on the C++ compilers and related tools, not application frameworks, database designers, and the like. Also, the target platform is Win32 (Intel only), even though some compilers address additional platforms. For testing, I used both Windows NT 3.51 and Windows 95 (Servicepack 4) running on a Pentium 120 with 64 MB of RAM, and a 256-KB level-2 cache. Reported times are for Windows NT.

Borland C++ 5.01

The Borland C++ compiler comes in three editions: Standard, Development Suite, and Development Suite with Design Tools. All versions include the same compiler and C++ development tools, plus Visual Database Tools for database-oriented applications. The Development Suite adds CodeGuard 32/16 (Borland's run-time error detection tool), InstallShield Express, a "lite" edition of the PVCS version manager, and Java tools. Additionally, the Design Tools version is bundled with the Together/C++ CASE tool from Object International. (These extras were not considered here.)

The Borland C++ 5.01 environment must be hosted on Win32 platforms (although some tools, such as TLINK for 16-bit applications, are still 16-bit executables). It can target DOS 16-bit, DPMI-286, and DPMI-386 executables (be warned though, that Borland's DOS PowerPack with its DOS extenders is not supported by the 5.0 compiler), Windows 16-bit, and Windows 32-bit executables. You also get an Intel-developed back end for the 32-bit compiler, which offers some advanced code optimizations. The ObjectWindow Library (OWL) is suitable for both 16- and 32-bit Windows targets, with minor differences between the targets. As of 5.01, source code for MFC 4.1 is included; all 5.0 versions come with tools to patch MFC 3.2 and 4.0 to compile with the Borland C++ compiler. Compilation can be performed both from the IDE and from the command line; in all cases, you'll work mostly with 32-bit tools, regardless of the target.

Borland includes stand-alone debuggers for DOS, Win16, and Win32, and an integrated debugger for Win32. Profilers are available for DOS and Win16 only. The IDE includes the Resource Workshop, which used to be a separate program.

In practice, I found the Borland IDE a reasonably good working environment, if unstable (it crashed a few times a day, less under Windows NT than 95). The environment is flexible, and tools such as TargetExpert and hierarchical displays for options settings make it fairly easy to change compilation options without losing sight of the big picture. The text editor is adequate, as are the resource editors. If you work with OWL, you'll find AppExpert and ClassExpert useful for getting started, although for serious work, you'll probably resort to manual editing. Compilation is quick, and the resulting executables are good performers -- if a bit overweight.

New with the 5.0 release of the Borland C++ compiler is the cScript scripting language for controlling the IDE (or any OLE Automation server, for that matter). With its C++-like syntax, you can access the many objects that make up the IDE and its constituent parts and control them however you like. The IDE and add-on tools, such as the Java tools and PVCS, use this capability, but you can add scripts or modify the standard ones. Apart from making modifications to the layout of the various IDE windows, I did not use cScript. You may find it useful to automate build procedures, though.

What I found wanting in the IDE is best described by a lack of "pointedness." For all its flexibility through Source Pools, Style Sheets, and (as of 5.0) an extensive scripting language for the IDE, I still configure each and every project. The default settings are not good for debugging nor for release, and there is no easy way to switch from one configuration to the other. Style Sheets and Source Pools let you create any configuration you want and switch from one to the other, but I'd prefer an automatic setup for the two most commonly used configurations -- Debug and Release -- and an easy way to migrate Style Sheets from one project to another.

Another area that troubled me was file rebuilding. A change in compilation options requires a manual rebuild, because the IDE does not mark affected files for recompilation. Also, a Build All does not clean up output files first. This means that if an error stops your rebuild after 98 of 100 files, you'll have to re-rebuild the first 97 files before you can rebuild files 98-100 as well. In the end, I resorted to hand-crafted make files for most projects.

There were other quirks too: The Borland C++ compiler looks for #include header.h files in the current directory rather than in the directory containing the including source or header file. While this is legal, it differs from most other C++ compilers and leads to problematic situations when referring to another project's nested header files from a new project. Finally, the code generated by the Intel compiler back end is not always reliable -- quite a few test programs crashed when their code was generated by the Intel, rather than the Borland, back end. Since the gains in performance are normally not very dramatic and compilation time increases are noticeable, using the Intel back end did not make sense for me.

Microsoft Visual C++ 4.1

Microsoft Visual C++ 4.1 is the subscription upgrade to the 4.0 version (I also tested the 4.2 successor to 4.1). This family of compilers targets only Win32 platforms; Win16 is out, as far as Microsoft is concerned (although the most recent 16-bit version of the compiler, 1.52, is still bundled with 4.x). The version I tested is for Intel host/target platforms; there is also a Macintosh cross compiler (hosted on Win32 for Intel), and Win32 compilers for MIPS, AXP, and PowerPC. Both the compiler and the linker are incremental, which means that they will only recompile or relink those parts of a source file or executable that have actually changed since the last build.

Developer Studio, the Microsoft development environment, is shared by the C++ tools, Fortran PowerStation, Visual Test, the Developer Network, and Visual J++. You must obtain the non-C++ tools separately. Visual C++ 4.1 comes with MFC 4.1 (4.2 comes with MFC 4.2). Several tools that used to be separate products -- the OLE Control Development Kit, for example -- are integrated into MFC. Recent upgrades have concentrated on adding Internet support classes to MFC, C++ language enhancements and, in 4.2, a draft-standard-compliant C++ library (to replace STL and the older IOStreams).

The Developer Studio integrates text and resource editors, a debugger, and project management tools. From it, the actual compilation and build process is performed through command-line versions of the compiler, linker, and the like. To support MFC programming, AppWizard and ClassWizard are included. Online help is particularly good and deserves mention -- it uses the same InfoViewer tools as the MSDN CD-ROM, and allows you to access the CD-ROMs from within the Developer Studio. A profiler and other tools are present as command-line programs, but accessible through the Tools menu.

This is a development environment I really like. In terms of usability, this environment is nearly on a par with Microsoft's Office applications, something I wouldn't say of other development environments. Initial project configurations are good, switching between them or adding new ones is easy, commands are available where you expect them, and it looks nice and feels responsive (even though some timings indicate that this is not the fastest compiler around). The online help is superb and I felt comfortable with the debugger integrated into the Developer Studio. Also, support for external tools is good to very good, and the package is as stable as a rock. In fact, this is the only development environment that I use routinely and don't feel tempted to leave for real work.

Still, there are a few things wanting. First of all, macro programming capability for the editor is needed, if only to insert commonly used text (file and function headers, for instance). Also, a multifile search-and-replace function should be added. Further, I got numerous internal compiler error messages. Thanks to online help, I found out most were caused by the compiler running out of steam during optimizations; switching off one of the optimizations (as indicated by the online help) solved the problem in most cases. Still, 4.1 generated just plain bad code in one of my test programs (this problem went away with 4.2). Finally, the build manager (or the incremental compiler or linker) is sometimes overly optimistic and fails to recompile some of the dependents of changed source code. In these cases, you're forced to perform a Build All to get everything right.

Symantec C++ 7.21

Symantec C++ is an alternative to the Borland and Microsoft C++ compilers. As such, it offers compatibility in both directions and adds speed, portability, and extensive compilation options. Usability has improved over previous versions, and the current Symantec C++ IDDE is comparable to or better than most. As a special feature, Symantec C++ sports NetBuild -- the ability to distribute the build process across networked computers.

Symantec C++ 7.21 comes with Windows 16- and 32-bit versions of its environment (including all tools), and targets DOS 16 bit and 286 and 386 extender models and Windows 16- and 32-bit applications. As a result, you can use every combination of Windows host and target platforms for program development. The package includes debuggers for all platforms (the famous MultiScope debugger), text and resource editors, and features such as Make and Link tools. The latter is the latest version of OPTLINK (another Symantec tool) and links executables in unbelievably short times.

As far as frameworks are concerned, Symantec C++ 7.21 includes MFC 2.52 (16 bit) and 3.2 (32 bit), and comes with AppExpress and ClassExpress to help you on the way.

Symantec C++ is fast. Speed permeates from the startup of the IDDE, through all compilation and build tools, to the final executables, which are among the smallest and fastest of the lot. In addition, the wide range of target platforms (and host platforms, for that matter), make cross-platform development almost fun. The IDDE user-interface model is different from the usual MDI (multidocument interface) environment, in that the basic IDDE consists of only a title bar, menus, tool bar, and a set of tabs to select a work space (Edit, Debug, Output, and the like). The actual content windows float on the normal Windows desktop, which remains visible all the time.

In theory, this is a nice idea, since it allows more visual integration with, and easier access to, other applications on the desktop. In practice, I found it tended to increase visual clutter and made it difficult to tell at a glance which windows belonged to which apps.

In daily work, the general pattern of operations follows the pre-4.x Microsoft Visual C++ Workbench model, including accelerator-key assignments. There are standard provisions for Debug and Release versions of each project, for subprojects, and for migration of options between projects. On the whole, however, I didn't find it as easy to use as the Visual C++ 4.x environments.

The C++ compiler is very quick, but somewhat too permissive. It lets many questionable constructs pass without warning, and the diagnostics it generates are not always self explanatory or accurate. This becomes a bigger problem with errors in templates; in these cases, the diagnostic often refers to the last line of the file containing the template definition, rather than to the offending line within that file. Templates also seem to be a problem where compilation speed is concerned: While Symantec C++ normally is quick to compile C++ source code, the presence of template expansions dramatically increases overall compilation time.

The Symantec C++ IDDE includes a Symantec Basic scripting language that can be used to automate tasks from within the IDDE. Although the feature looks enticing and the language familiar, I never used it. If you rely mainly on the Symantec C++ IDDE, you may find it worthwhile to invest your time in developing scripts for it.

Watcom C++ 10.6

Watcom has a reputation for cross-platform development tools. With Watcom C++ 10.6, you get compilers that target DOS 16 bit, 286, and 386 extended; Windows 16- and 32-bit applications; OS/2 1.x through 3.x; and Novell NetWare. As far as hosting is concerned, you can run all command-line tools on all platforms (excluding NetWare) and all GUI tools on Windows and OS/2.

The package includes supporting tools (debuggers and profilers), libraries and toolkits (such as MFC 2.52 and 3.2), and a SOM toolkit for OS/2. Blue Sky's Visual Programmer is also included for developing MFC apps. The compilers and linkers are all command-line tools, but they are normally run from within the IDE.

"Looks don't count for machos" seems to apply to the Watcom IDE, which acts as the central switchboard and project manager. While it is very usable, it does not look as slick as some of the others. Of course, this is partly because Watcom uses the same IDE across all GUI-supporting platforms, and that precludes using some of the spiffier features of each platform.

Having said that, I find the Watcom designers masters at achieving maximum effect with minimal means. For example, when I first tried establishing build dependencies between subprojects, I could not find the right options to do so; moments later, I discovered that the project manager will automatically establish the correct relationships, based on knowledge of target types (.lib, .dll, and so on). This also applies to issues such as a Build All, which is no big deal to the IDE -- just set the date/time stamps of all output files to 1/1/80 and perform a regular Make action. Changes to compilation options (C or C++) are likewise recognized and dealt with. While this is not as sophisticated as, say, Borland's project manager, it is more effective and reliable.

All tools, including the editor, are external to the IDE and are run as separate processes. This makes for a fast-loading IDE, and it's easy to substitute for default tools. It does take more time to load a text file, but it is acceptable. What I found bothersome was the overall lack of speed in the compiler. For small programs, this was not a problem, but it became noticeable in larger projects. Compiling the MFC 3.2 sources from scratch took 10:14 (minutes:seconds) for the Watcom C++ compiler, but only 2:49 for Symantec. The same pattern consistently recurred for all but the smallest programs. In addition, compiler output is verbose, even if you tell it not to be. Four lines produced by a single diagnostic is not uncommon; the problem is exacerbated by the nonintuitive and selective way that warnings are suppressed by pragmas in the source code.

Watcom source-code compatibility with Borland and Microsoft C++ compilers is not as good as I had hoped. Furthermore, the compiler does not yet support C++ run-time type information and new-style typecasts. Finally, I had some problems with the profiler (which is implemented as separate sampling and presentation tools) -- somehow, I managed to crash one or the other every time I used them.

IBM VisualAge for C++for Windows 3.5

With VisualAge for C++ for Windows, Version 3.5, IBM entered the Win32 C++ development-tool fray. According to IBM, one goal was to offer portability to all platforms IBM sees as important -- Win32, OS/2, AIX, and MVS. VisualAge for C++ comes with the compiler for Win32 targets, editors for text and resources (the resource editors are a Win32 incarnation of Borland's Resource Workshop), debugger, profiler, Visual Builder, and a WorkFrame to integrate the lot. To ease the transition from OS/2, the Workframe and several other GUI tools have a distinct OS/2 look and feel.

VisualAge for C++ for Windows (VA) must be hosted on a Win32 platform and targets only Win32 (although source-code compatibility is maintained with other IBM platforms as long as you don't use platform-specific APIs). The Open Class library, a collection of C++ classes addressing data structures, user-interface elements, and the like, is not a true application framework, nor are there any "AppExperts" or similar wizard-like tools available for it. (IBM does provide "Project Smarts," though.)

Of the Win32 environments I examined, VisualAge for C++ for Windows is the least productive and most intrusive toolset I came across. For instance, ever since installing the tool, my Windows 95 system opens all folders in Small Icon view with a toolbar, regardless of what the settings used to be. I don't mind VA having its own preferences, but I don't want it to change my system without asking. On a Windows 95 system, VA requires no less than 20 KB of environment space. On both Windows 95 and Windows NT systems, VA adds 22 different environment variables, with room for extensions, to communicate settings between the various tools.

Simply starting the WorkFrame IDE takes time. To improve load times for the second and subsequent runs, VA leaves six processes running when the WorkFrame session ends. According to IBM, this is accepted practice on OS/2. It was a nuisance on my Win32 systems. Having started the WorkFrame, you can then choose from various application models through the use of Project Smarts. A static library is not among the options, as I found during one of my tests. The wizard-like approach for project configuration takes you through a number of tabbed dialog pages (like OS/2, with spiral binders), which look out of place on Windows platforms. Moreover, elementary options such as a Browse button to navigate to a project directory are missing.

Once you complete the wizard sequence, the regular WorkFrame window appears. Where C++ environments commonly consider a "project" to be a repository containing references to the actual source files (wherever those may be located), a WorkFrame project is the combined contents of all directories that you tell it to consider as project directories. Consequently, your WorkFrame window shows you all files indiscriminately, be they source files, object files, makefiles, or backup files. For any sizable project, this will be several hundred files.

Before compiling anything, you must use tools such as MakeMake to designate which files are source files and which aren't, tell the environment that it should use the C++ compiler for .cpp files and the resource compiler for .rc files, and specify compilation options for all these types. MakeMake will then generate a makefile to compile your sources. Well, sometimes it does, and sometimes it doesn't. MakeMake adds superfluous items (such as "echo Compile" to every compilation target's build rules), and it is weak in transferring the right compilation options to the makefile (as a rule, they were not transferred correctly) and generating the header dependencies for each source file. This point was underscored when I tried to compile MFC 4.1 sources -- VA took almost 18 minutes to generate all the dependencies, compared to less than 5 minutes for a straightforward Makedep tool, about 1.5 minutes for the Symantec C++ compiler, and 35 seconds for an optimized MkDep tool I subsequently developed. Furthermore, after each change to compilation options, MakeMake insists on a rescan of the dependencies, which takes, in the MFC case, another 15 to 18 minutes. In the end, I used my hand-crafted makefiles if I had to test the compiler.

Tool Summaries

Table 1 summarizes the most important features of the C++ development systems reviewed here. Note that every compiler is assumed to address both Windows 95 and NT as host and target platforms. Only features that come in the box are taken into account.

In addition to the general experience reports, I conducted a number of tests to see how different compilers behave with different tasks. The requirement to run the across-the-board tests necessitated some limitations (compatibility is good enough to allow you to compile larger projects on all compilers without substantial extra effort, which I couldn't afford).

Test results as reported apply to Windows NT. Unless otherwise noted, compiler settings were:

• Pentium code generation, with 8-byte alignment for data structures.
• Optimize for speed, fast function call model (arguments passed in registers), fast floating point.
• No standard stack frame, no stack overflow checking.
• Function-level linking, dead code elimination, unreferenced function removal (if available).
• C++ exception handling and RTTI enabled, if available, with destructor cleanup.
• No debug or browser information; file section alignments at their smallest allowable value.
• Run-time libraries were statically linked, using the nondebug, single-threaded versions.
• Precompiled headers were neither generated nor used.
• All build times refer to a full rebuild, either from the compiler's IDE (if possible) or from a separate make file.

For the Borland compiler, I ran each test for both the Borland and Intel back end and reported the results separately. All tests, including the build processes, were run several times to obtain the final results, interspersed with unrelated applications to flush NT's disk caches. Background activity on the test system was reduced to a minimum. The test results turned out to be highly reproducible.

Test 1: Hello World

The "Hello World" test, in both C (using <stdio.h> and printf()) and C++ (using <iostream.h> and cout), checks the minimal executable size; see Table 2. Optimizations were set for size rather than speed.

Even small programs (particularly C++) end up with lots of code in the executable because of features such as exception handling and the overhead of C++ class libraries. For example, the IOStreams library is often implemented in terms of the C stdio facilities. Compilers with function-level linking have an advantage in this test.

Test 2: BYTEMark Version 2

For the BYTEMark test, I cleaned up the code somewhat and added support for the compilers that were not supported in the original BYTE source code. The program is fully C based.

Table 3 refers to separate subtests of the benchmark program and are normalized relative to BYTE's standard system for these tests -- a Dell 90-MHz Pentium, and a benchmark program compiled with Watcom C++ 10.0. Given that I ran my tests on a Pentium/120, you'd expect benchmark scores of about 1.33. Any other results should be attributed to a nonlinear scaleup of the computer hardware and differences in code generated by the compilers.

The Watcom C++ 10.6 results, which are supposedly comparable to BYTE's Watcom C++ 10.0 results, indicate that as far as the hardware is concerned, the scaling effect holds to within a few percent. If you take this as the reference point for the other compilers, you find that only Microsoft's compiler performs consistently better. As far as integer scores are concerned, Watcom and Borland are better (in particular, with the Intel back end), but Microsoft heads the list. Floating point is where the most dramatic differences are to be found. Borland and Symantec perform poorly, and Watcom and Microsoft perform as expected.

Test 3: Memory Allocation

The memory-allocation test (see Table 4) measures the performance of the memory-allocation routines in the compiler's C run-time library (used to implement C++ memory allocators). In a program based on Arthur Applegate's article "Rethinking Memory Management" (DDJ, June 1994), I let each program allocate and free (in some random order) a number of memory blocks of varying sizes. The net result is a somewhat fragmented heap. As an extension to the original program, I varied the allocation pattern both with respect to block size and overall number of bytes allocated.

I chose the two dimensions -- block size and total allocation size -- because I noticed different behavior between them. For Symantec C++, the best performer in the test, the only noticeable effect was from increased block size: With larger blocks, it spends more time per block regardless of the overall allocation size. This also holds true for Borland C++, the second-best performer. I attribute the block size effect to the compilers implementing their own suballocators, thereby amortizing expensive system-allocation calls over a number of subblocks. With larger subblocks, each system-allocation call serves fewer subblocks, incurring a greater cost per block.

With the other compilers, however, time per block varied within a given block-range size. In those cases, I presume their suballocators deteriorate as the heaps become more fragmented during the test and free-pool search times become more and more important. For example, Microsoft's compiler is notorious for its poor memory allocator. Starting with 4.2, Microsoft included an improved suballocator for small blocks and its effect can be seen in the table. Apart from lower times per block for small blocks, they are also virtually constant for a given block-size range.

Test 4: TLX Class Library

The TLX class library contains about 80 C++ classes and class templates and was developed to provide commonly used data structures and algorithms for research. The source code is portable across some 15 C++ compilers for PC and UNIX systems, which makes it suitable for compiler comparisons.

The main objective of this test is to compare build times for nontrivial (about 25,000 lines) amounts of C++ source code, and the resulting library size. The latter is only an indirect indication of the size of the object code, because .LIB files also contain extra information such as dictionaries.

The Symantec compiler is the fastest compiler and generates the most compact library file. Microsoft's compiler generated the largest .LIB file. The .EXE file in Table 5 is a small program that does nothing but print out the library's version and build information. It is used as a rudimentary test of the proper compilation of the library. In this case, it indicates that as far as overall executable size is concerned, .LIB-file sizes are no indication at all.

Test 5: Constrained Optimization

The constrained-optimization test builds on the previous TLX test and applies the TLX library classes to a real optimization problem. It tests the compiler's ability to generate code from class templates and (at run time) the performance of the resulting code in situations that do not have the locality of reference of, for example, the BYTE benchmarks. As such, this test is probably more representative of real programs.

As Table 6 shows, the .EXE sizes are fairly similar across the board, with the exception of a large Intel-generated executable for Borland and a small executable for Microsoft. Compilation times were different, though; the worst performer was the Intel back end. Considering also the errors in the generated code, my advice is to not use the Intel back end. Of the other compilers, Watcom was the slowest, although the resulting code was reasonably compact and among the fastest.

Run times to the first solution were evenly matched, as were the run times until 100,000 problem nodes had been explored. The Visual C++ 4.1 compiler had a code-generation problem that created an endless loop in the program; 4.2 generated correct code and completed the test in the fastest overall time. Still, the results indicate that for programs that spend time evenly performing various activities (control structures, memory allocations, calculations), different compilers behave more alike than focused tests such as the BYTE benchmark would indicate.

Ad Hoc Tests

I also experimented with source code such as MFC. However, these tests were not performed consistently across all compilers, so take the results with a grain of salt.

Compiling MFC 4.1 source code with Borland C++ to a nondebug DLL generated a BFC40.DLL file of 1,343,488 bytes (Microsoft's MFC40.DLL for MFC 4.1 is 921,872 bytes). Compiling MFC 3.2 source to a nondebug DLL with Symantec C++ resulted in a DLL file of 375,808 bytes; Watcom's was 526,848 bytes. Watcom contained more function entry points in its import library; however, I'm not sure if the DLLs contained the same classes. Microsoft's DLL for this version measures 322,822 bytes. Build time for Symantec was 2:49, while Watcom was 10:14.

Conclusion

Overall, C++ programmers developing for Win32 platforms are better off than ever before. As the tests I ran for this article indicate, however, most vendors still have work to do. For the most part, I've tried to select tests that highlight significant differences between compilers. You may have other tests that produce different results. If so, I'd like to hear from you.

DDJ