This article contains the following executables: ISETL386.ZIP
Andrew is a software engineer in Cambridge, Mass., working on CD-ROM network applications, and is also a contributing editor for DDJ. He can be reached at 32 Andrew St., Cambridge, MA 02139.
Over two years ago, the cover of the July 1987 issue of Dr. Dobb's carried the title "386 Development Tools Within Your Lifetime" a photograph of a skeleton that rotted away in front of its computer while waiting for decent 386 tools, which summed up everyone's feelings about programming for the Intel 80386 microprocessor.
Things have improved a great deal since that issue. Watcom C7.0/386, for instance, produces 32-bit code (such as MOV EAX, 12345678h, and MOV FS:[EAX], ESI) while staying keyword and library compatible with the de facto 16-bit industry standard, Microsoft C 5.1 (MSC51). Even weird low-level routines such as intdosx( ), _dos_setvect( ), _dos_keep( ), and _chain_intr( ) do the right thing in 32-bit protected mode.
Of course, Watcom C7.0/386 (WAT-386) has many of the same features as Watcom's 16-bit C compiler (see "Examining Room," DDJ September 1989). This includes Watcom's famous register-based parameter passing. Many of Watcom's innovations involve the reduction, and sometimes elimination, of function call overhead. Any block of code that takes input from registers and puts output into registers is effectively a functional object, and WAT386 takes advantage of this fact in several places, including the nifty #pragma aux feature.
Buying In
WAT386 produces very different code from either Microsoft C or Turbo C (neither of which has an option to generate 386 instructions, much less 32-bit code). Yet, this compiler will fit seamlessly into your current work habits. Unlike MetaWare's High C 386 compiler, using WAT386 does not produce "culture shock."
Still, all is not rosy. It will cost you over $1000 in software to get into 386 development. WAT386, like High C, costs $895, and you will also need a 32-bit DOS extender, like the industry-standard Phar Lap 386 toolkit, which costs $495.
Further, the new Watcom C7.0/386 compiler is just that -- new. While writing this review, I found a number of bugs in the compiler and its standard library. Watcom was undoubtedly under pressure from its major client, Novell, to get the 386 compiler out the door. By the time you read this review, though, a second, more stable, release of WAT386 should be available.
Primarily because of its newness, WAT386 in some ways is not as good a product as MetaWare's High C 386, which has been around for two and a half years. Still, there is value in WAT386. For many PC programmers, this will be a much easier product to use than MetaWare's High C. WAT386's Microsoft compatibility is very important. On the other hand, the next release (1.6) of High C 386, in addition to many other changes, is scheduled to have what a MetaWare press release calls "86% compatibility with Microsoft's C libraries."
32 Bits!
WAT386 generates code for 32-bit protected mode. Thus, sizeof(int) and sizeof(unsigned) are each 4 bytes, not 2 bytes. Likewise, sizeof(void *) is 4 bytes. Note that sizeof(void near *) is also 4 bytes.
The all-important ANSI C identifier size_t, which is the unsigned type of the result of the sizeof( ) operator and the type used by function parameters that accept the size of an object, is also 4 bytes (typedef unsigned size_t).
C standard library functions such as malloc( ), fwrite( ), and strncpy( ) all take size_t parameters, and strlen( ) returns a size_t. These standard library functions deal in quantities between O and UINT_MAX. In the 16-bit code generated by PC compilers like MSC51, UINT_MAX is OxFFFF (65,535), yielding the familiar 64K limit on PC array lengths, string lengths, malloc blocks, and so on.
But in 32-bit code, UINT_MAX is OxFFFFFFFF, or 4,294,967,295 -- the magical upper "limit" of 4 gigabytes! In the native mode of the 386, this is the upper bound set on array lengths, string lengths, and malloc blocks. Effectively, no limit at all.
The Environment
If fwrite( ) can write 4 gigabytes at a time (which might be handy if you're working with CD-ROM or some other form of mass optical storage), how can it possibly work with MS-DOS? DOS is a 16-bit operating system. (So is OS/2.) The DOS Writefunction (INT 21, function 40H), which fwrite( ) must eventually call, expects the number of bytes to write in the 16-bit CX register. The maximum is 64K. How can WAT386, or any 32-bit C compiler for DOS, produce code that's compatible with 16-bit DOS?
The answer is that 386 C compilers (for DOS) produce code to be run under a 32-bit DOS extender. Programs such as Phar Lap's 386|DOS-Extender and Eclipse Computer Solutions' OS/ 386 do not replace DOS. Instead, they (almost invisibly) manage the interface between 16-bit real-mode DOS and your 32-bit protected-mode program.
In the example of fwrite( ), the 32-bit code produced by WAT386 or High C (which MetaWare actually calls "High C for MS-DOS/386") continues to call INT 21, function 40H. But now, the number of bytes to write goes into the full 32-bit ECX register rather than the 16-bit CX register.
A DOS extender takes over INT 21 (as well as other software interrupts like INT 10, INT 16, and so on), handles some functions itself, and passes others on to DOS. A program running under a 32-bit DOS extender is effectively running under "MS-DOS/386," because, for example, a call to write 640K is really going to write 640K. The DOS extender will invisibly break this up into multiple calls to the "real" INT 21, function 40H.
Another interesting example is malloc( ). If your 386 computer came with 4 gigabytes of memory, you could grab it all with a single call to malloc( ). As in 16-bit real-mode C compilers, the C memory manager eventually calls INT 21, function 48 (allocate memory). Here, however, the DOS extender provides a complete replacement, not a front end, for the DOS routine. There is one difference between Phar Lap and Eclipse: 386|DOS-Extender expects in EBX the number of 4K pages to allocate, where OS/386 more closely mimics DOS, expecting the number of 16-byte paragraphs. The WAT386 standard library detects which DOS extender it is running under and allocates memory appropriately.
By default, WAT386 produces code to be run under Phar Lap Software's 386|DOS-Extender. The Phar Lap toolkit (DOS extender, linker, assembler, and debugger) must be purchased separately, however.
Oddly, you don't need a 386 machine or a DOS extender to run the WAT386 compiler. By the time you read this review, Watcom should be shipping a 32-bit protected-mode version of the compiler. In the version I reviewed, however, all compiler components were 16-bit real-mode programs. To avoid "Not enough memory to fully optimize procedure" warnings, I had to specify that the compiler use a large-model version of the code generator. Pretty crazy for a 386 development system!
Presumably, if your customers had 386s but you didn't (which is probably the exact opposite of the real situation), you could use these 16-bit tools to generate 386 code on your AT.
Programs compiled with WAT386 and linked with Phar Lap's 386|LINK will only run on 386-based machines. To sell such programs, and to acquire a program that will "bind" the DOS extender into the executable so that your customers don't need to know anything about the DOS extender, you must acquire a redistribution package from Phar Lap. This costs an extra $1000 for unlimited distribution.
So the entrance fee for 386 development is still pretty steep. What do you get in return? A lot: Code that runs several times faster than 16-bit code; the elimination of 64K limits on array sizes or function parameters; and the elimination of the 640K boundary, allowing you to use all physical memory in the machine.
Note that this "MS-DOS/386" gives you big memory, but not virtual memory (VM). This is an important difference from OS/2. However, a VM manager (386|VMM) is available for $295 from Phar Lap, and WAT386 code, like High C code, runs without change under 386|VMM.
WAT386 code runs under one other environment: Novell's new 32-bit network operating system, NetWare 386 (see the accompanying box).
The Code
How can a 32-bit C compiler such as WAT386 produce code that runs several times faster than 16-bit code run on the same machine? Consider the following two lines of code:
extern char*Env; char*p=EnvCompiling under the "large model" (which is what most commercial PC software uses), any 16-bit C compiler, including Watcom's non-386 compiler, produce code something like that shown in the first portion of Example 1, in which the 4-byte far pointers are transferred piecemeal from one location to another.
Example 1: 32-and 16-bit code generated under the large memory model
16-bit code: mov es, seg _Env mov ax, word ptr es:_Env mov dx, word ptr es:_Env+2 mov word ptr _p, ax mov word ptr _p+2, dx _______________________________ 32-bit code: mov eax, _Env mov _p, eaxBecause mov mem, reg takes 2 clock cycles on a 386 and mov reg, mem takes 4 cycles regardless of whether the compiler uses the 8-bit (AL), 16-bit (AX), or 32-bit (EAX) form of the register, this takes (2*2) + (3*4) = 16 cycles. In contrast, the 32-bit equivalent takes 2 + 4 = 6 cycles (shown in the second portion of Example 1). More Details. The 32-bit code is similar to the code that would be generated by a 16-bit compiler working with 2-byte near pointers:
mov ax, _Env mov word ptr _p, axIn fact, "flat model" 32-bit code and "tiny model" 16-bit code are very similar. The only difference is that the 16-bit code can handle quantities up to 64K, whereas the 32-bit code can handle quantities up to 4 gigabytes. Right now, WAT386 supports flat model and small model. In the flat memory model, the application's code and data must total less than 4 gigabytes in size. In the small memory model, your code and data are each "limited" to 4 gigabytes. By default, WAT386 uses the flat model. When linking with the Lahey linker (LINK-EM/32) provided with OS/ 386, you must compile with the small model. Because an offset into a segment is 4 bytes while the segment registers are still 2 bytes, sizeof(void far *) is 6 bytes (an FWORD, not a DWORD). But because a near pointer is a 4-byte quantity, you almost never have to deal with far pointers. When a segment takes a 4-byte offset, even the most sloppily written, bloated program in the world should do fine with the flat model. Once loaded, DS and CS stay constant. Effectively, this is a linear address space.
Real-World Benchmarks
Interpreters are better for benchmarking compilers than the tiny programs that are usually used. Such benchmarks usually involve a fair amount of source code. The C source code for several interpreters is readily available, and to execute one line in the interpreted language, the interpreter needs to crunch through a lot of C code. In the remainder of this review, I'll describe using WAT386 (and MetaWare High C) to port a larger program to the 386: ISETL (Interactive Set Language), written in C by Gary Levin (Dept. of Mathematics and Computer Science, Clarkson University, Potsdam, N.Y.). ISETL is an interpreter for working with sets, tuples, propositions, several different types of functional objects, matrices, and other constructs useful for studying the mathematical foundations of computer science. It is described in the book Learning Discrete Mathematics with ISETL by Nancy Baxter, Ed Dubinsky, and Gary Levin (New York: Springer-Verlag, 1989). ISETL deserves a full discussion, but for now I'll just describe the process of producing ISETL/386. Due to space considerations, the ISETL/386 listings are not included in this issue. They are available through DDJ (see the end of this article for information). The ISETL implementation consists of 29 .C files and 14 .H files, and totals about 13,000 lines of code. Some of the code is YACC output. When I tried to produce a 386 version of this real program, my opinion about WAT386 vs. High C nearly reversed. As long as I was working on small one- or two-module programs, WAT386's similarity to Microsoft C and Turbo C made it preferable to MetaWare High C. But once I started working on ISETL/386, with more source code, written by someone else, my allegiance shifted to High C. High C provides better warning messages than WAT386; the High C compiler is faster than WAT386 (remember, the WAT386 compiler I used was a 16-bit real-mode program); surprisingly, High C seems to produce better overall code than WAT386; and, most important, High C and its standard library isn't buggy like WAT386. I should mention that Watcom has terrific technical support. If you call up with a problem, you get to talk to the person responsible for the library or the compiler. Watcom is quick to find and fix bugs and, with the WPATCH utility that comes with WAT386, they have made the patch a fine art. Watcom runs a well-organized BBS. On the other hand, I don't even know how good MetaWare's technical support is, because I never needed to use it. At one time or another, we've all thought we've found a compiler bug only to discover that in fact we have a bug in our own code. But after working with WAT386 for about a month, I found that nearly every time it was a compiler or library bug. First of all, one of the key switch statements in ISETL was behaving bizarrely. The value of the variable being switched on was correct, we would jump to the correct case label, but a function call to Emit(42) wasn't working. The problem is that any constant (for example, 42), used (anywhere) inside a switch statement is scrambled if that constant happens to match the number of case labels in the switch statement! This bug should be fixed by the time you read this. If you have this same release of the compiler, you can download a patch from the Watcom BBS. Another problem occurs because the ISETL initialization file opens the DOS device CON (to implement a pause( ) routine for use in ISETL programs) and tries to read from this device. The problem is, when reading from any of the DOS device files (CON, AUX, and so on), the WAT386 library gets confused between binary and text mode; a call to wait for one character actually waits for 512 characters, that makes it seem like the machine is hung. In another project, I found that intdosx( . . . ) was not working, even though int386x(0x21, . . . )worked fine. If this has not been corrected by the time you read this, a patch is available from the Watcom BBS. In that same project, I found an obscure bug in Watcom's use of the "interrupt" keyword that had to do with calling an interrupt function rather than generating an interrupt. Basically, functions defined with void interrupt (far *f)( ) work. But functions defined with void (interrupt far *f)( ) (note the placement of parentheses) don't do a PUSHFD when you call them. There is one problem that's not Watcom's fault: Debugging with the 386 flat memory model is hardly better than debugging in real mode. With one single segment working as a linear address space, it is nowhere as easy to catch bugs as when you have lots of little segments (for example, a 286-based protected-mode DOS extender such as DOS/16M). In fact, to debug ISETL/386, I found it necessary to create a DOS/16M version (ISETL/286). This shows that segmentation is not such a bad idea, after all, it's crucial for genuine memory protection. The ideal situation is to use lots of segments for development, and then switch over to the flat model for production. The only assistance you get in catching memory protection violations from the WAT386 flat memory is the Phar Lap linker's OFFSET switch, which allows you to load code or data starting at some offset other than zero. This way, you get page faults when dereferencing bad pointers, though you often won't know where they come from.Benchmarking with ISETL/386
Once ISETL/386 was up and running with WAT386, I was able to write some ISETL programs and use them for benchmarking the compilers. In addition to contrasting WAT386 and High C, I was able once again to compare 32-bit code with 16-bit code, using the Turbo C-produced executable from the ISETL distribution. Figure 1 shows the results for two different ISETL programs to generate prime numbers, for an ISETL program to generate the first 1000 Fibonacci numbers, and for an overall test of ISETL operations.Figure 1: ISETL test execution times in seconds (Watcom and High C run times using Phar Lap 386\DOS-Extender)
WAT386 HIGH C 386 TURBO C -------------------------------------------------------- PRIME.SET 2000 18.0 16.3 24.8 PRIME.SET 4000 42.6 40.0 N/A PRIME.TUP 2000 1:03.7 52.7 1:11.3 PRIME.TUP 4000 4:14.9 3:27.6 N/A FIB.SET 1000 15.0 14.1 20.4 FIB.SET 1200 18.0 17.0 N/A overall test 1:05.3 59.4 1:30 total 477.5 407.1 N/A ISETL filesize 133K 148K 209K ISETL full compile 12:52 min. 11:45 min. 3:30 min.Rather than use explicit loops, the ISETL prime number program in Listing One (page 115) uses set notation. This program creates the set of all odd numbers less than n, takes the union of this set with the singleton set {2}, then takes the difference between the resulting set and the set of all odd composite numbers less than n. The resulting set is the set of all primes <= n. This can be expressed in a few lines of ISETL code. Listing Two (page 115) performs the same operation, but uses ordered tuples (sets are, of course, unordered). I had to choose a small number n because, even with garbage collection, ISETL gobbles up a lot of memory. Listing Three (page 115) is a program to generate the first 1000 Fibonacci numbers. This relies on ISETL's support for assignment to the return value of a function (which allows one to write functions that "remember" past values-dynamic programming) and ISETL's arbitrary-precision arithmetic. Fibonacci(1000) is a 209-digit number. ISETL/386 takes 15 seconds to compute the first 1000 Fibonacci numbers in the WAT386 version and 14 seconds in the High C version. The 16-bit Turbo C ISETL takes 20.4 seconds. The High C 386 version of ISETL was faster than the WAT386 version in every case tested. Overall, the High C version was about 15 percent faster than the WAT386 version. This is somewhat surprising since, as is well known, MetaWare produces High C by using an automatic compiler-compiler (which MetaWare markets separately as the Translator Writing System). Profiling with the DOS/16M protected-mode debugger from Rational Systems (DOS/16M currently has the only decent protected mode C source-level debugging tools available), I found that ISETL generally spends 50 percent of its time in only four routines. Perhaps this test is somewhat lopsided. Any real program, on the other hand, will have similar "hot spots."
The Future
Over the next few months, both Watcom and MetaWare are planning major upgrades that may be out by the time you read this. One obvious change in High C is that while the 1.5 libraries are missing functions such as open( ), fdopen( ), dup( ), fileno( ), and signal( ), High C 1.6 is scheduled to include both a Microsoft-compatible standard library (including _dos_keep( ), int86x( )), a 32-bit version of the GFX graphics library, and a 32-bit version of the Sterling Castle C library. WAT386's new release should include a 32-bit protected mode source-level debugger, a 32-bit version of Watcom's graphics library (which is identical to the MSC51 graphics library), a 32-bit version of the WAT386 compiler, and a 32-bit version of Watcom's Express in-memory quick compiler. The source-level debugger is urgently needed, and should put Watcom ahead in the 386 development tool race. A 386 compiler war may indeed be starting. While WAT386 itself is not fully mature, its arrival is a sign of the growing strength of the market for 386 development tools. And about time too, now that the first 486s are rolling off the assembly line. But remember, even an 80586 will not save you from bad code.Product Information
Watcom C7.0/386 Watcom 415 Phillip Street Waterloo, Ontario, Canada N2L 3X2 800-265-4555 Price: $895 Requirements: 386-based PC- or PS/2 compatible, MS-DOS 3.1 or higher, 386 DOS extender toolkit: 386|DOS-Extender (Phar Lap) or OS/386 (Eclipse Computer Solutions) C Network Compiler/386 Novell Development Products P.O. Box 9802 Austin, Texas 78766 512-346-8380 Price: $995Watcom and Novell
In addition to producing code for Phar Lap's 386|DOS-Extender and, with some difficulty, for Eclipse's OS/386, the 32-bit Watcom C compiler also works with Novell's new network operating system, NetWare 386. In fact, Watcom C7.0/386 is being repackaged by Novell as its C Network Compiler/ 386. (This is the subject of Novell's strange "See Dick and Jane" ads.) NetWare 386 is a 32-bit operating system, and this allows for several performance leaps over the existing 286-based NetWare. Instead of the current limit of 100 users per file server, which is dictated by the single 64K data segment available in "medium model" (used in 286-based NetWare), the new NetWare 386 allows 250 simultaneous users per file server. Likewise, Novell claims that network throughput is two to three times greater than its already zippy throughput figures. In NetWare 386, the lack of segmentation in "flat model" is taken to its logical (but scary) extreme -- no memory protection. Novell baldly states that, "There is no memory or other application-level protection: All applications and device drivers run in kernel mode" (NetWare Technical Journal, July 1989). When used with NetWare 386, the Watcom C compiler produces server applications -- programs that run in file-server memory (the so-called "file server" thus becomes a generic server). These server applications are called "NetWare Loadable Modules," or NLMs, and are somewhat like value-added processes (VAPs) in pre-386 NetWare; except unlike VAPs, NLMs can be loaded or unloaded at any time, without taking down the file server. NLMs, in fact, are dynamic-link libraries and, in addition to providing services to clients on the network, can provide functions to be called by other NLMs. For instance, when calling a C standard library such as open( ) from an NLM, you are actually calling a routine in CLIB.NLM, which is the C standard library provided as a dynamic-link library. The code for open( ) is not linked into your executable. To produce such an NLM, use the NLMLINK provided by Novell rather than the Phar Lap linker. Similar to the OS/2 linker, NLMLINK requires a .DEF file with import statements. The module produced by the Novell linker essentially contains unresolved externals that are resolved when the NLM is loaded into file server memory (either by invoking the LOAD command at the file server console, or by spawning one NLM from within another). The library included with C Network Compiler/386 includes many functions not available in the standard Watcom library. Naturally, functions are provided to support network communications with Novell's IPX and SPX. The Btrieve data management library is provided as BTRIEVE.NLM. The Novell library includes functions (for example, TestAndSetBit( ) and BitScan( )) to interface to the 386-bit test instructions. Network servers are inherently multitasking (multiple operations must be in progress simultaneously on behalf of multiple clients), so the library contains functions for "execution threads," such as Begin Thread( ), EnterCritSec( ), ExitCritSec( ), SuspendThread( ), and so on. There are also functions to manage semaphores and queues. While this part of the Novell API seems modeled on OS/2, it is important to note that NetWare 386 uses non-preemptive multitasking. Inside a "big job," it is therefore necessary to call a routine such as delay( ) or ThreadSwitch( ) so that other threads are not starved. The library that Watcom provided for Novell contains a few modifications to support multiple threads. Global variables such as errno are in fact allocated on a prethread basis. Static data such as used by the notorious strtok( ) function is also handled differently than in a single-threaded library. No new keywords (such as private, used in Lattice C 6.0 for OS/2) have been added, however. -- A.S. INSIDE WATCOM C 7.0/386 by Andrew Schulman [LISTING ONE]$ PRIMES.SET $ ISETL program to find number of primes <= n, using set notation size := 1000 ; sqrt_size := fix(sqrt(size)) ; composites := {i*j : i in {3,5..sqrt_size}, j in {i..size div i}} ; primes := {2} + {3,5..size} - composites ; print size ; print #primes ;[LISTING TWO]
$ PRIMES.TUP $ ISETL program to find number of primes <= n, using ordered tuples $ tuple difference operator diff := func(t1, t2); return [i : i in t1 | i notin t2 ] ; end; size := 1000 ; sqrt_size := fix(sqrt(size)) ; composites := [i*j : i in [3,5..sqrt_size], j in [i..size div i]] ; primes := [2] + [3,5..size] .diff composites ; print size ; print #primes ;[LISTING THREE]
$ FIB.TUP $ ISETL program to find Fibonacci numbers, using dynamic programming $ uses log(): only accurate up to 308 digits digits := func(x); if (x = 0) then return 1 ; else return 1 + floor(log(abs(x))) ; end; end; $ use "dynamic programming" to assign to fib() fib := func(x); fib(x) := fib(x-1) + fib(x-2) ; return fib(x) ; end; fib(0) := 1 ; fib(1) := 1 ; fibonacci := [fib(x) : x in [1 .. 1000 ] ] ; print fibonacci(1000) ; print digits(fibonacci(1000)) ;
