Numerous C++ libraries are available that ease the burden of programming graphical user interfaces (GUIs) in a cross-platform environment. However, few libraries address the interprocess communications (IPC) facilities built into today's sophisticated PC and workstation operating systems. Interprocess communication is a vital part of client/server computing, and processes and threads running on the same system can communicate in many ways other than the familiar clipboard, DDE, and OLE standards.
In this article, I'll summarize the common techniques for IPC and present one way to build cross-platform C++ libraries. In doing so, I'll write an example library that implements semaphores in a platform-independent manner to allow signaling or controlling of shared resources between processes and threads. Implementations are presented for OS/2, AIX, and Windows NT.
Communication Mechanisms
Clipboard, DDE, and OLE facilities are familiar to many programmers because they are supported by Windows. However, Windows does not support the IPC capabilities built into 32-bit operating systems such as AIX, OS/2, and Windows NT. These powerful APIs enable sophisticated access to and sharing of information between processes and threads running in the same environment. The most common of these communication mechanisms are semaphores, shared memory, queues, and named pipes.
A semaphore is really just a flag indicating that an event has occurred or that data is locked and should not be changed by another process. While you can certainly define your own flag as a global variable in a DLL that other processes can use for signaling or resource locking, problems can arise. For instance, in a preemptive-multitasking environment, a task may be preempted at any time, even while trying to access or change a flag. This can cause synchronization problems unless the flag is controlled at the kernel level of the operating system in cooperation with the task scheduler. Thus, operating systems provide a semaphore API for creating and controlling semaphores.
There are two basic types of semaphores: mutex (mutual exclusion) and event. Mutex semaphores are used primarily to control access to some kind of shared resource: shared memory, database records, files, and the like. Event semaphores signal that an event has occurred, allowing other processes waiting on that event to continue. This facilitates synchronizing tasks that need to cooperate or allowing a process to wait until some kind of required initialization is complete.
Shared-memory address space is a block of memory created by one process or thread and made available to other running processes. Any process or thread that has access to the memory can use it just as if the memory were a part of the application's own address space. Memory is usually accessed through a unique name known to all processes needing to use it. Obviously, if two or more processes are writing to the shared-memory space simultaneously, the resulting data could be garbled, so a mutex semaphore is typically used to ensure that access is sequential.
Figure 1 schematically shows how two processes share the same address space. Shared memory typically offers the best performance in sharing data among applications, but other techniques may be more appropriate for client/server programming.
Figure 1.
Message queues are familiar to GUI programmers who use event-driven message loops to dispatch messages to window procedures. Windowing systems typically provide functions for posting messages to windows in the same or other applications, but these functions do not work for character-mode programs. However, a message-queue communication mechanism exists that is valid for both character-mode and window-mode applications: A queue may be created by a server application, which then reads messages from the queue and processes them one at a time. Multiple client applications can then access the queue and write messages to the server, and the operating system synchronizes the queue so that messages originating from different sources will not intermingle; see Figure 2.
Figure 2
Pipes are buffer areas which allow processes to communicate with each other just as if they were reading and writing files. Pipes can be either named or unnamed. Some operating systems (such as OS/2) allow named-pipe communications not only between applications on the same system, but also between applications running on different systems connected via a network. A server process creates a named pipe, and one or more client processes open it and establish connections; see Figure 3. Once the connection is established, the pipe is treated as a regular file handle, and the standard file I/O methods are used to transfer data across the pipe.
Figure 3.
Writing Cross-Platform Classes
OS/2, AIX, and Windows NT implement the IPC features described up to this point. Unfortunately, there is no standard for these features' functional interface, so each vendor provides a custom API. What programmers need is an API that is consistent across all platforms to minimize the operating-system-dependent part of their code.
The preprocessor definitions, macros, and data types for each operating system are all different. To avoid having users of your communication library deal with this at the API level, you can separate the library-class APIs from the implementation of the classes, which must necessarily be different on each platform. While there are many ways of doing this in C++, I'll undertake the separation for this library by defining a single interface class (InterfaceClass) that contains an instance of the operating-system-specific implementation class (ImplementationClass).
InterfaceClass is packaged as a class declaration (in ifclass.h) as in Example 1(a). The file ifclass.C contains method invocations which call the corresponding method in the ImplementationClass instance; see Example 1(b). Now the implementation-class header (defined in imclass.h) can still be operating-system independent (or #ifdefs can be included to encapsulate the dependent parts); see Example 1(c). Furthermore, as Example 1(d) shows, you can define three separate implementations of the constructor and destructor ImplementationClass to handle the individual operating systems (see os2impl.C, aiximpl.C, and winimpl.C).
Example 1: (a) InterfaceClass packaged as a class declaration; (b) method invocations which call the corresponding method in ImplementationClass; (c) implementation of class header can still be operating-system independent; (d) defining three separate implementations of the constructor and destructor ImplementationClass.
(a)
// forward declaration
class ImplementationClass;
// interface class declaration
class InterfaceClass {
friend class ImplementationClass;
public:
// constructor and destructor
InterfaceClass();
virtual ~InterfaceClass();
void SomeMethod(); // invoked by class user
protected:
ImplementationClass *myImpl; // pointer to implementation
};
(a)
void InterfaceClass::SomeMethod()
{
myImpl->SomeMethod();
}
(a)
class ImplementationClass {
public:
// constructor and destructor
ImplementationClass();
virtual ~ImplementationClass();
void SomeMethod(); // invoked by InterfaceClass
};
(a)
// os2impl.C
#define INCL_DOS
#include <os2.h>
void ImplementationClass::SomeMethod()
{
// call OS/2 specific functions
}
// aiximpl.C
#define <sys/ipc.h>
void ImplementationClass::SomeMethod()
{
// call AIX specific functions
}
// winimpl.C
#define <windows.h>
void ImplementationClass::SomeMethod()
{
// call Windows NT specific functions
}
To package the class library for distribution, compile the ifclass.C module and the appropriate implementation module for the current operating system (for OS/2, compile os2impl.C). You then either link the modules as a DLL or make a static library. The distribution files are simply the ifclass.h file and the library module for each operating system. Each platform uses the same header file containing the class declaration, so platform-independent applications can be written using the InterfaceClass.
Semaphores
To illustrate, I'll implement named semaphores because AIX requires a name for the ftok() function I'll use later. Also, you may need to query for the name and type of semaphore and perhaps access the semaphore id created by the underlying operating system. You can implement all of these generic methods in an abstract base class, and derive specialty classes for the mutex and event semaphores from the base class.
The Abstract Base Class
Listing One shows the class declaration for the abstract base class ipcSemaphore. All of the interface classes and data types are given an ipc prefix to avoid possible namespace collisions when linking with other libraries. Two enumerated types are defined in this file: ipcSemaphoreType holds the type of semaphore (mutex or event), and ipcSemaphoreOp defines whether the semaphore is being created by the owner of the semaphore (semcreate) or already exists and is being accessed (semaccess).
Listing One
// ****************************************************************************
// Module: ipcsem.h -- Author: Dick Lam
// Purpose: C++ class header file for ipcSemaphore
// Notes: This is an abstract base class. It is the interface class for
// semaphores used in signalling between processes and threads.
// ****************************************************************************
#ifndef MODULE_ipcSemaphoreh
#define MODULE_ipcSemaphoreh
// semaphore type designation and operation type
enum ipcSemaphoreType { unknown = 0, mutex = 1, event = 2 };
enum ipcSemaphoreOp { semcreate = 0, semaccess = 1 };
// forward declaration
class osSemaphore;
// class declaration
class ipcSemaphore {
friend class osSemaphore;
public:
// constructor and destructor
ipcSemaphore(const char *name, // unique name for semaphore
ipcSemaphoreType type, // mutex or event
ipcSemaphoreOp operation); // create or access the semaphore
virtual ~ipcSemaphore();
// methods for getting semaphore parameters [name, semaphore id, type of
// semaphore (mutex or event) and whether this is the owner (creator)
// of the semaphore]
char *Name() const;
unsigned long ID() const;
ipcSemaphoreType Type() const;
int Owner() const;
// pure virtual query method for number of requests made for the semaphore
// (must be redefined in derived classes)
virtual unsigned long Query() = 0;
// class version and object state data types
enum version { MajorVersion = 1, MinorVersion = 0 };
enum state { good = 0, bad = 1, badname = 2, notfound = 3 };
// methods to get the object state
inline int rdstate() const { return myState; }
inline int operator!() const { return(myState != good); }
protected:
osSemaphore *myImpl; // implementation
state myState; // (object state (good, bad, etc.)
private:
// private copy constructor and operator= (define these and make them
// public to enable copy and assignment of the class)
ipcSemaphore(const ipcSemaphore&);
ipcSemaphore& operator=(const ipcSemaphore&);
};
#endif
The base-class constructor takes arguments corresponding to a unique name for the semaphore, type, and operation. Processes or threads that access the semaphore only need to know the name and type of the semaphore to use it.
Methods are also defined for returning the semaphore name, underlying operating-system semaphore id (as an unsigned long), semaphore type, and whether the current process or thread is the owner (creator) of the semaphore. The last method, Query(), is a pure virtual method that queries the semaphore and returns the number of requests pending on it.
The last member is a pointer to the osSemaphore class. This friend class has its declaration in ossem.h (Listing Two). osSemaphore is the operating-system-independent implementation class corresponding to the imclass.h mentioned earlier.
Listing Two
// ****************************************************************************
// Module: ossem.h -- Author: Dick Lam
// Purpose: C++ class header file for osSemaphore
// Notes: This is a base class. It contains general implementation methods
// for semaphores used in signalling between processes and threads.
// ****************************************************************************
#ifndef MODULE_osSemaphoreh
#define MODULE_osSemaphoreh
#include "ipcsem.h"
// class declaration
class osSemaphore {
public:
// constructor and destructor
osSemaphore(ipcSemaphore *interface, const char *name,
ipcSemaphoreType type, ipcSemaphoreOp operation);
virtual ~osSemaphore();
// methods for getting semaphore parameters [name, semaphore id, type of
// semaphore (mutex or event) and whether this is the owner (creator)
// of the semaphore]
char *Name() const;
unsigned long ID() const;
ipcSemaphoreType Type() const;
int Owner() const;
// mutex semaphore methods
void CreateMutex();
void OpenMutex();
void RequestMutex();
void ReleaseMutex();
unsigned long QueryMutex();
void CloseMutex();
// event semaphore methods
void CreateEvent();
void OpenEvent();
void PostEvent();
void ResetEvent();
void WaitEvent();
unsigned long QueryEvent();
void CloseEvent();
protected:
ipcSemaphore *myInterface; // pointer to the interface instance
char *myName; // semaphore name, id and type
unsigned long myID;
ipcSemaphoreType myType;
int isOwner; // flag indicating whether this is owner
private:
// private copy constructor and operator= (define these and make them
// public to enable copy and assignment of the class)
osSemaphore(const osSemaphore&);
osSemaphore& operator=(const osSemaphore&);
};
#endif
Listing Three is the interface class. The ipcSemaphore constructor simply creates an instance of osSemaphore that is deleted by the destructor. The other member functions use this instance of the implementation class to call the corresponding function in that class. For example, the ipcSemaphore::Name() method calls myImpl-->Name().
Listing Three
// ****************************************************************************
// Module: ipcsem.C -- Author: Dick Lam
// Purpose: C++ class source file for ipcSemaphore
// Notes: This is an abstract base class. It is the interface class for
// semaphores used in signalling between processes and threads.
// ****************************************************************************
#include "ipcsem.h"
#include "ossem.h"
// ****************************************************************************
// ipcSemaphore - constructor
ipcSemaphore::ipcSemaphore(const char *name, ipcSemaphoreType type,
ipcSemaphoreOp operation)
{
// init instance variables
myState = good;
myImpl = new osSemaphore(this, name, type, operation);
if (!myImpl)
myState = bad;
}
// ----------------------------------------------------------------------------
// ~ipcSemaphore - destructor
ipcSemaphore::~ipcSemaphore()
{
delete myImpl;
}
// ----------------------------------------------------------------------------
// Name - returns the name of the semaphore
char *ipcSemaphore::Name() const
{
if (!myImpl)
return 0;
return myImpl->Name();
}
// ----------------------------------------------------------------------------
// ID - returns the semaphore id
unsigned long ipcSemaphore::ID() const
{
if (!myImpl)
return 0L;
return myImpl->ID();
}
// ----------------------------------------------------------------------------
// Type - returns the type of semaphore
ipcSemaphoreType ipcSemaphore::Type() const
{
if (!myImpl)
return unknown;
return myImpl->Type();
}
// ----------------------------------------------------------------------------
// Owner - returns 1 if this is the owner (creator), and 0 otherwise
int ipcSemaphore::Owner() const
{
if (!myImpl)
return 0;
return myImpl->Owner();
}
Mutex Semaphores
Mutex semaphores control access to some resource. They have two basic functions which our class must implement--request and release. When a process (or thread) requests a mutex semaphore, the process either obtains "ownership" of the semaphore immediately or is blocked until another process releases the semaphore.
The semaphore is then "owned" by the requesting process, and requests for the semaphore by other processes are blocked until the "owner" releases the semaphore.
Listing Four shows the class declaration for ipcMutexSemaphore, which is derived from our abstract base class. The Query() method provides an implementation for the pure virtual method in the base class, and Request() and Release() methods for the allowed operations on mutex semaphores.
Listing Four
// ****************************************************************************
// Module: ipcmutex.h -- Author: Dick Lam
// Purpose: C++ class header file for ipcMutexSemaphore
// Notes: This class is derived from ipcSemaphore. It is an interface class
// for mutex semaphores that can be used to control access to a shared
// resource across processes or threads.
// ****************************************************************************
#ifndef MODULE_ipcMutexSemaphoreh
#define MODULE_ipcMutexSemaphoreh
#include "ipcsem.h"
// class declaration
class ipcMutexSemaphore : public ipcSemaphore {
public:
// constructor and destructor
ipcMutexSemaphore(const char *name, ipcSemaphoreOp operation = semcreate);
virtual ~ipcMutexSemaphore();
// query method for number of requests made
virtual unsigned long Query();
// request and release methods (to lock and unlock resources)
virtual void Request();
virtual void Release();
private:
// private copy constructor and operator= (define these and make them
// public to enable copy and assignment of the class)
ipcMutexSemaphore(const ipcMutexSemaphore&);
ipcMutexSemaphore& operator=(const ipcMutexSemaphore&);
};
#endif
The implementation of this class is given in ipcmutex.C (Listing Five). In the constructor, the ipcSemaphoreOp operation is checked and used to call either CreateMutex() or OpenMutex() in the osSemaphore class instance myImpl (defined as "protected" in the base class). The destructor calls the myImpl-->CloseMutex() method, and the other three methods call their corresponding methods in myImpl. Listing Two shows prototypes for these five member functions included in the public section of osSemaphore.
Listing Five
// ****************************************************************************
// Module: ipcmutex.C -- Author: Dick Lam
// Purpose: C++ class source file for ipcMutexSemaphore
// Notes: This class is derived from ipcSemaphore. It is an interface class
// for mutex semaphores that can be used to control access to a shared
// resource across processes or threads.
// ****************************************************************************
#include "ipcmutex.h"
#include "ossem.h"
// ****************************************************************************
// ipcMutexSemaphore - constructor
ipcMutexSemaphore::ipcMutexSemaphore(const char *name,ipcSemaphoreOp operation)
: ipcSemaphore(name, mutex, operation)
{
// check the state of the object
if (myState != good)
return;
// create or open the semaphore
if (operation == semcreate)
myImpl->CreateMutex();
else if (operation == semaccess)
myImpl->OpenMutex();
}
// ----------------------------------------------------------------------------
// ~ipcMutexSemaphore - destructor
ipcMutexSemaphore::~ipcMutexSemaphore()
{
// close the semaphore
if (myState == good)
myImpl->CloseMutex();
}
// ----------------------------------------------------------------------------
// Query - returns the number of requests made of the semaphore
unsigned long ipcMutexSemaphore::Query()
{
if (myState == good)
return myImpl->QueryMutex();
return 0L;
}
// ----------------------------------------------------------------------------
// Request - requests the semaphore
void ipcMutexSemaphore::Request()
{
if (myState == good)
myImpl->RequestMutex();
}
// ----------------------------------------------------------------------------
// Release - releases the semaphore
void ipcMutexSemaphore::Release()
{
if (myState == good)
myImpl->ReleaseMutex();
}
Event Semaphores
Event semaphores synchronize operations between processes or threads. The terminology is different from mutex semaphores, and there are three basic operations instead of two. For example, if application A wants to start application B and wait for B to perform some operation before continuing, A first creates and resets an event semaphore. B is then started, and A does a wait on the event semaphore. When B is done with its operation, it accesses the event semaphore and posts it, causing A to unblock and continue execution.
Listings Six and Seven show the declaration and implementation for ipcEventSemaphore. This closely parallels the structure of ipcMutexSemaphore, except that the Request() and Release() methods are replaced by Reset(), Wait(), and Post(). The constructor also tests the operation flag argument and calls either CreateEvent() or OpenEvent(), and the destructor calls CloseEvent(). Note that all of these functions are also included in the public section of the declaration for osSemaphore. Now that the interface to the two semaphore types has been defined, we are ready to look at the actual implementations on each operating system.
Listing Six
// ****************************************************************************
// Module: ipcevent.h -- Author: Dick Lam
// Purpose: C++ class header file for ipcEventSemaphore
// Notes: This class is derived from ipcSemaphore. It is an interface class
// for event semaphores that can be used to signal events across
// processes or threads.
// ****************************************************************************
#ifndef MODULE_ipcEventSemaphoreh
#define MODULE_ipcEventSemaphoreh
#include "ipcsem.h"
// class declaration
class ipcEventSemaphore : public ipcSemaphore {
public:
// constructor and destructor
ipcEventSemaphore(const char *name, ipcSemaphoreOp operation = semcreate);
virtual ~ipcEventSemaphore();
// query method for number of requests made
virtual unsigned long Query();
// post, reset and wait methods
virtual void Post();
virtual void Reset();
virtual void Wait();
private:
// private copy constructor and operator= (define these and make them
// public to enable copy and assignment of the class)
ipcEventSemaphore(const ipcEventSemaphore&);
ipcEventSemaphore& operator=(const ipcEventSemaphore&);
};
#endif
Listing Seven
// ****************************************************************************
// Module: ipcevent.C -- Author: Dick Lam
// Purpose: C++ class source file for ipcEventSemaphore
// Notes: This class is derived from ipcSemaphore. It is an interface class
// for event semaphores that can be used to signal events across
// processes or threads.
// ****************************************************************************
#include "ipcevent.h"
#include "ossem.h"
// ****************************************************************************
// ipcEventSemaphore - constructor
ipcEventSemaphore::ipcEventSemaphore(const char *name,ipcSemaphoreOp operation)
: ipcSemaphore(name, event, operation)
{
// check the state of the object
if (myState != good)
return;
// create or open the semaphore
if (operation == semcreate)
myImpl->CreateEvent();
else if (operation == semaccess)
myImpl->OpenEvent();
}
// ----------------------------------------------------------------------------
// ~ipcEventSemaphore - destructor
ipcEventSemaphore::~ipcEventSemaphore()
{
// close the semaphore
if (myState == good)
myImpl->CloseEvent();
}
// ----------------------------------------------------------------------------
// Query - returns the number of requests made of the semaphore
unsigned long ipcEventSemaphore::Query()
{
if (myState == good)
return myImpl->QueryEvent();
return 0L;
}
// ----------------------------------------------------------------------------
// Post - posts the semaphore
void ipcEventSemaphore::Post()
{
if (myState == good)
myImpl->PostEvent();
}
// ----------------------------------------------------------------------------
// Reset - resets the semaphore
void ipcEventSemaphore::Reset()
{
if (myState == good)
myImpl->ResetEvent();
}
// ----------------------------------------------------------------------------
// Wait - waits for a semaphore event to be posted
void ipcEventSemaphore::Wait()
{
if (myState == good)
myImpl->WaitEvent();
}
The osSemaphore Implementations
The modules that implement the osSemaphore class for OS/2 (os2sem.C), AIX (aixsem.C), and Windows NT (winsem.C) are available electronically. Each module includes the header files specific to that operating system, plus ossem.h, the class declaration for osSemaphore. Because the operating-system dependencies are buried here, users of these classes need not compile with complicated preprocessor definitions and include files that specify the target operating environment.
The constructors for all three implementations are almost identical. OS/2 requires system semaphores to have a pathname such as \SEM32\name, and AIX requires a name for the ftok() function, which returns a key required to obtain UNIX interprocess-communication identifiers. Therefore, both os2sem.C and aixsem.C define a semPath string constant at the top of the module, and semPath and the name argument are used to form the full semaphore name stored in the myName instance variable. Also, a file with the name of the semaphore must exist on AIX, and it is created in aixsem.C if the ipcSemaphoreOp type is semcreate.
Note that the constructor takes a pointer to the ipcSemaphore interface instance that creates the osSemaphore. Because osSemaphore is also declared as a friend of ipcSemaphore, the osSemaphore methods can change the myState variable (declared in ipcsem.h). The user can then call rdstate() (or use the ! operator) to get information on the state of the semaphore.
The destructors delete the memory allocated in the constructors for the full semaphore pathname, and the AIX destructor also deletes the file and removes the semaphore using a call to semctl(). The other methods for returning the name, id, type, and owner are all the same in each file, and could arguably be put in the interface base-class implementation. However, this would require exposing the protected variables representing these values to the user of the semaphore classes. You could have also placed the common code in an abstract implementation base class and derived specific implementation classes from it, but the current structure is adequate for our needs.
CreateMutex(), OpenMutex(), and the like are the remaining methods, and they simply call the API functions appropriate for each operating system. For example, DosCreateMutexSem() is the OS/2 function that creates a mutex semaphore, while the ftok() and semget() functions are required under AIX. For Windows NT, macros are defined in winbase.h that must be undefined to avoid conflicts with the CreateMutex(), OpenMutex(), CreateEvent(), and OpenEvent() methods. Then the Win32 functions ::CreateMutexA(), ::OpenMutexA(), and the like are called directly.
Conclusion
I have defined a cross-platform implementation of mutex and event semaphores with an interface class that has no operating-system dependencies. You only need to compile the modules on the appropriate system and distribute the object module or DLL and the three header files ipcsem.h, ipcmutex.h, and ipcevent.h.
The test programs semtest1.C and semtest2.C (available electronically) can be compiled on all three systems using the code presented in this article. These programs illustrate how the mutex and event semaphores can synchronize two processes.
Dick is a member of the research staff at IBM's T.J. Watson Research Center. He can be contacted at [email protected].
