Implementing NLM-Based Client/server Architectures

NetWare Loadable Modules (NLMs) are applications which are executed by the NetWare 3.x operating system. As such, they are 32-bit protected-mode programs able to take full advantage of the multitasking, multithreaded architecture of the NetWare operating system.

This article presents a distributed file manager made up of two modules--ENGINE.NLM, an NLM running on a NetWare 3.x server, and CLIENT.EXE, a DOS-based front end running on the client. ENGINE.NLM is not a full-featured file manager; rather, it's a basic implementation designed to illustrate several key aspects of NLM development:

• Implementing client-server communications using IPX.
• Using thread-control mechanisms for NLMs, including semaphores.
• Informing other servers about the NLM via the Service Advertising Protocol (SAP).
• Using transaction-tracking services (TTS) to ensure the integrity of the data file.

The most accurate way to view NLMs is as extensions to the NetWare operating system. The NetWare loader and linker essentially bind a loaded NLM to the operating system, and thereafter do not make a distinction between that NLM and the OS itself. Like the NetWare OS, NLMs use a flat 32-bit memory addressing model.

While it's possible to develop interactive, client-style applications as NLMs, it isn't appropriate to do so. NLMs should be "server" modules, providing "services" to clients located on other network stations. Moreover, NLMs should provide services that take advantage of the NetWare OS architecture. For example, a distributed file manager can take advantage of the fast 32-bit NetWare file system. It wouldn't make sense to develop an NLM for rendering bit-mapped graphics. Not only would this NLM slow file service to NetWare clients, but the NetWare OS offers no inherent advantage to this type of application over other 32-bit operating systems.

Even though NLMs are protected-mode programs, you don't need to worry about protected-mode issues; the NetWare operating system and loader handle the low-level details for you. The NetWare 386 runtime interface is a superset of ANSI C, so there's no need to delve into the more complex NetWare APIs unless you are particularly compelled to do so. Finally, NLMs use a flat memory space, which means you don't have to worry about segmentation, near or far pointers, and the like.

The most oft-repeated criticism of NLMs is that they run on an operating system that lacks memory protection. An errant NLM can therefore bring down the NetWare server by corrupting memory it doesn't own. Because NLMs don't run on a memory-protected OS, some say that they're not a suitable platform for application development. I believe this criticism has been blown out of proportion. The implied assumption is that bugs are acceptable unless they bring the server down--even if they corrupt something like an accounting database. The fact remains that buggy programs can do serious damage even when running on a memory-protected OS. Protection or not, it's the developer's responsibility to ensure the program is bug free and robust. The lack of memory protection hasn't kept developers from writing thousands of successful applications for unprotected-environment platforms like DOS and the Macintosh OS.

Memory protection benefits the developer by allowing detection of errant memory operations early in the development process, before releasing the program for QA test and production. You can, of course, trap NLM memory errors during the development cycle using tools like Nu-Mega Technologies' NLM Developers Kit, which includes a profiler, both network and NLM memory checkers, and a low-level debugger.

The NetWare 3.x environment is different from most mainstream commercial operating systems because it is highly optimized for networking operations. Therefore, it lacks certain features, such as memory protection and preemptive multitasking, found in general-purpose operating systems. (Multitasking in NetWare 3.x is non-preemptive.)

Other aspects of NetWare 3.x present opportunities and challenges to NLM developers. For example, NetWare uses all its free memory to cache data files, thus speeding network file service. This has interesting implications for my distributed record manager, discussed later.

NetWare performs load-time linkage of NLMs to external routines. This reduces memory consumption by NLMs because multiple NLMs may share the same libraries. NLMs may also export routines for linkage by other NLMs when they load. The external routines called by ENGINE.NLM are all exported by the NetWare C Interface (CLIB.NLM).

To create NLMs, you need Novell's software development kit (SDK) for NetWare 3.x. The SDK documents the NetWare C interface and provides the Watcom 32-bit C/386 compiler, the Novell linker (NLMLINK), header files, and miscellaneous utilities. You don't need to ship runtime support with your NLM because CLIB.NLM, which provides the actual public symbols of the NetWare C interface, ships with the standard NetWare package purchased by end users. For debugging, there's the NetWare internal debugger, an assembly language debugger which is part of the NetWare OS, and the Watcom Debugger, which runs on DOS.

NLMs must be compiled to support a flat memory model (using the /mf compiler switch for Watcom C 8.0 and earlier), to use stack-based calling conventions (using the /3s compiler switch), and to generate object files in Phar Lap's Easy-OMF format (using the /ez compiler switch).

The Novell linker (NLMLINK) requires a definition file that describes the characteristics of the NLM being linked, including which symbols it imports from CLIB, which symbols it exports to other NLMs, and so on. All NLMs must be linked to a special object file called PRELUDE.OBJ, which provides startup code for the NLM.

ENGINE.NLM is the back end of a distributed record manager. Because of its size (over 1000 lines of C source code), the program is only available electronically; see "Availability" on page 5. The operations supported by ENGINE are: adding a new record to the database; editing an existing record; reading an existing record; and marking an existing record as deleted. ENGINE performs all the file I/O itself, on behalf of the client, 32 bits at a time. ENGINE is designed to clearly demonstrate NLM programming methods; as such, it's too simple a record manager for practical use.

ENGINE performs the following basic operations:

1. Initialization.
2. Listening for request packets from clients.
3. Spawning a worker thread to perform the client's request when a packet comes in.
4. Returning to step 3.

  typedef struct recordHeader {
      unsigned long status;
      unsigned long offset;
      unsigned long hashkey;

     unsigned long recordNumber;
      unsigned long transactionNumber;
      unsigned char key [128];
  } rHeader;

  typedef struct recordData {
      time_t creationTime;
      time_t lastReferenceTime;
      time_t lastUpdateTime;
      unsigned char nodeAddress[10];
      unsigned long objectID;
      unsigned char data[128];
  } rData;

  typedef struct record {
      rHeader header;
      rData data;
  } rec;

The status field of the recordHeader structure indicates the state of the record. A value of 0 means the record is deleted or free, 1 means the record is occupied, and 2 means that the record is the special database header always located at offset 0 of the data file. The offset field gives the offset of the record within the data file. The hashkey field is not used in this version of the record manager, but will be included in a future version which supports hashed keys. The transactionNumber field is used in conjunction with TTS, a feature of the NetWare OS which preserves the integrity of data files in the face of I/O errors. The recordNumber and key fields of the record header are self explanatory.

The first three fields of the record's data portion indicate when the record was created, when it was last read, and when it was last updated. The nodeAddress field gives the network address of the last station which requested an update of the record. The objectID field gives the NetWare ID of the user who last updated the record. Finally, the data field contains the record's data.

Note that an entire record is designed to fit within a single IPX packet, which may be up to 576 bytes in size (including the 30-byte IPX header). This simplifies things considerably, because ENGINE does not need to construct multipacket messages in order to send an entire record to the client, nor does the client need to defragment them. (Multipacket-message support is more easily implemented using NetWare's Sequenced Packet eXchange, or SPX.)

CLIENT.EXE is the front-end component of the distributed record manager. Like its ENGINE.NLM counterpart, CLIENT.EXE is available electronically; see "Availability" on page 5. CLIENT provides all data input and display for the user. CLIENT doesn't perform file I/O; it simply makes requests for ENGINE.NLM to do so on its behalf and presents the results to the user. This lets the application take full advantage of the speed and robustness of the NetWare file system.

CLIENT.EXE's basic operations are:

1. Initialization.

  typedef struct ipxheader {
       WORD checkSum;
       WORD length;
       BYTE transportControl;
       BYTE packetType;
       LONG destNet;
       BYTE destNode [6];
       WORD destSocket;
       LONG  sourceNet;
       BYTE sourceNode [6];
       WORD sourceSocket;
  } IPX_HEADER;

  typedef struct shortPacket {
       WORD responseCode;
       BYTE operation;
       rHeader header;
  } sPacket;

  typedef struct longPacket {
       WORD responseCode;
       BYTE operation;
       rec record;
  } lPacket;

Add a Record

1. CLIENT allows the user to input the record key and data fields.
2. CLIENT sends an AddRecord request to the ENGINE. The packet includes the entire record to be added.
3. ENGINE attempts to add the record to the data file.
4. ENGINE sends a response packet back to the client containing the record header.
5. CLIENT infers from the status field of the record header (contained in the response packet) whether or not the record was successfully added.

1. CLIENT allows the user to input a record key.
2. CLIENT sends a FindRecordKey request to the ENGINE. The packet includes only the record header.
3. ENGINE attempts to find the matching record.
4. ENGINE sends a response packet back to the client containing the entire record.
5. CLIENT infers from the status field of the record header (contained in the response packet) whether or not the record was found.
6. If the record was found, CLIENT displays the record.

Steps 1 through 5 are same as "Read a Record" above.

6. If the record was found, CLIENT displays the record and allows the user to edit the record's data field.

7. CLIENT sends an EditRecord request to ENGINE. The request packet contains the entire edited record.

8. ENGINE updates the record by writing the edited record to the data file.

9. ENGINE sends a response packet back to the client containing the updated record's header.

10. CLIENT infers from the updated record's status field whether or not the edit operation was successful.

Delete a Record

Steps 1 through 5 are same as "Read a Record" above.

6. If the record was found, CLIENT changes the record's status field to 0.

Steps 7 through 10 are the same as "Edit a Record" above.

  typedef struct fragment {
      void far *fragAddress;  // DOS version.  All pointers
      WORD fragSize;          // are NEAR for an NLM
  } ECBFragment;

  typedef struct ecb {        // DOS version.  NLM version
      void far *linkAddress;  // is slightly different.
      void far *ESRAddress;
      BYTE inUseFlag;
      BYTE completionCode;
      WORD socket;
      BYTE IPXWorkspace[4];
      BYTE driverWorkspace[12];
      BYTE immediateAddress[6];
      WORD fragCount;
      ECBFragment fragList[2];
  } IPX_ECB;

Not all ECB fields must be initialized when you send or receive a packet. For example, the immediateAddress field contains the network address of the nearest router which knows the path to the ultimate destination of the packet. You only need to initialize the immediateAddress field when you send a packet.

The most important ECB field is the fragment-descriptor field which contains the address and length of the buffers that make up the packet. Buffer 0 must always be the IPX header. Other buffers can be anything the application defines. Combined, they make up the data field of the packet.

When you send a packet, IPX copies the buffers described in the sending ECB's fragment-descriptor field and combines them into a packet, which it sends over the network. When you receive a packet, IPX fragments to packet and copies the different components to the buffers described in the receiving ECB's fragment-descriptor field.

Note that when you send a packet, you must initialize the packet's IPX header with the packet-type code (a value of 4 for IPX packets) and with the packet's destination address.

Once you've initialized the ECB and (if necessary) the IPX header, you can post the ECB for sending or receiving by calling either IPXSendPacket(IPX_ECB *) or IPXListenForPacket(IPX_ECB *).

ENGINE uses three primary threads to accomplish its work. The first thread is main, which allocates required resources from the OS; registers functions with the OS for cleaning up the environment when the NLM is unloaded; begins the other two threads; and then goes to sleep until the user unloads ENGINE.

An important item within main is the call to AdvertiseService, which causes the operating system to send broadcasts every 60 seconds informing other servers on the network of the name, type, and network address of the ENGINE.NLM. Novell calls this feature the service-advertising protocol (SAP). Once ENGINE is advertising itself, CLIENT.EXE can discover the name and location of the ENGINE.NLM by scanning the bindery of any server on the network.

The second thread is InitMain, whose only job is to listen for query packets from CLIENTs wishing to begin a session with the ENGINE. After initializing itself, InitMain goes to sleep by calling WaitOnLocalSemaphore. The OS wakes up InitMain as soon as a query packet comes in from a client. InitMain then sends a query response packet back to the client and goes back to sleep. The entire purpose of InitMain is to provide a starting point for the client-server dialogue.

The third thread is EngineMain--the workhorse of the entire NLM. It listens for request packets, evaluates the op code of packets it receives, and spawns worker threads to accomplish the appropriate tasks on behalf of the client.

Like InitMain, EngineMain sleeps when there are no packets for it to process. As soon as a request packet comes in, the OS awakens EngineMain. Because most of the client/server traffic is in the form of request packets, EngineMain is able to handle up to six incoming packets per execution cycle. Moreover, it can spawn a worker thread and begin to listen for additional packets even before the worker thread has started to perform the task requested by the client.

ENGINE starts all its threads by making a call to BeginThread. This call requires a pointer to the function that ENGINE wishes to execute as a thread, and a pointer to a stack for the thread. Alternately, if you pass a NULL stack pointer to BeginThread, the OS allocates a stack for the new thread.

Because NetWare is a non-preemptive multitasker, it is possible for a thread to execute in a tight loop without relinquishing the CPU, thus shutting other threads out and slowing overall server performance. For this reason, each thread in ENGINE.NLM makes calls to the ThreadSwitch function within the body of loops. ThreadSwitch merely moves the calling thread to the back of the kernel's run queue, giving other threads a chance to run.

ENGINE continues to run until a user unloads it using the Unload command at the NetWare server console. When this occurs, the OS calls two functions in the NLM which were registered by main at run time. The first function, UnloadCleanUp, was registered using the signal API. The second function, ShutdownCleanUp, was registered using the atexit API.

The OS calls UnloadCleanUp as soon as the user has issued the UNLOAD ENGINE command. At this point, all threads continue to run. UnloadCleanUp sets the global shutdown variable to 1, causing threads to exit their loops and return. Next, UnloadCleanUp awakens sleeping threads by calling SignalLocalSemaphore, and returns. When the threads wake up, they see that the shutdown variable is equal to 1, and they kill themselves by exiting to main.

The OS calls ShutdownCleanUp after it has killed any threads which didn't kill themselves. At this point, the only thing the NLM can do is free OS resources it previously allocated, such as semaphores and sockets. ShutdownCleanUp does this and then dies, allowing the NLM to unload cleanly.

Although ENGINE.NLM is admittedly simple, it does sport some advanced features. For example, it supports an unlimited number of concurrent clients. It also uses NetWare's TTS to ensure the integrity of the data file. I've designed the architecture of ENGINE.NLM so it's easy to extend into a more sophisticated record manager without substantial changes to the program's structure. For example, defining the record header and data structures separately allows for easy migration to an indexed record manager (which uses a separate index file). In such a case, TTS support would be essential, because updating a record requires updating two separate files.

Despite the simplicity of ENGINE.NLM's record-handling components, the basic design scales up well because of its intelligent use of the NetWare OS. For efficiency, all threads sleep when they don't have any work to do. Each client request is handled by a different worker thread, which distributes the total workload evenly, meaning that no single client may consume a disproportionate share of computing resources. These are NLM design fundamentals that can apply to any industrial-strength NLM.

If I were extending ENGINE.NLM to become a full-fledged record manager, I'd be tempted to construct an index of the data file in memory, thus speeding access to records. However, this approach can backfire in an NLM because NetWare uses all its free memory to cache data files. An index file for ENGINE.NLM would, over a short period of time, become cached by the OS. Therefore, it would be as though the index were buffered in memory. However, the OS caches data files on a per-block basis using an intelligent LRU algorithm. If I were to buffer the index file myself, I would be overriding the OS caching algorithm with a brute-force approach. This would be less effective overall than simply allowing the OS to cache the index file for me.

Copyright © 1992, Dr. Dobb's Journal