Channels ▼
RSS

Open Source

Shared Memory and Message Queues


In the article Cross-Platform Communication Classes (DDJ, March 1995), I presented a method of separating a C++ class interface from the underlying implementation details when writing cross-platform classes for event and mutex semaphores. Although there are several ways to separate the interface and implementation, I'll continue with the same approach, applying it here to the cross-platform coding of named shared memory and message queues. In doing so, I'll support interprocess communication (IPC) mechanisms for OS/2, AIX, and Windows NT.

Shared Memory

Shared memory is a single address space allocated as a block of memory by some process (or thread). This process gives the memory to one or more additional processes, and all processes then use the memory as if it were part of their normal address space. To gain access to an existing shared-memory block, processes can either be given a pointer or handle to the block, or they can reference the block by a name agreed upon beforehand.

If the shared memory is unnamed, the memory pointer must be passed from the creating process to any other process that wishes access to the shared memory. This can be done using other forms of IPC such as DDE, a message queue, or a pipe. In this article, I'll consider only named shared memory--a shared-memory block with a specific name that allows any process which knows the name of the block to gain access to the memory.

The interface to the generic shared-memory class is shown in Listings One and Two. There are two constructors for ipcSharedMemory. One is used by the process or thread which actually creates the memory block, and it takes the name of the block and the desired size of the block in bytes as input arguments. The second constructor is used by other processes or threads which need access to an existing block, and this constructor requires only the block name as a parameter.

Listing One

// ****************************************************************************
// Module:  ipcshmem.h  -- Author:  Dick Lam
// Purpose: C++ class header file for ipcSharedMemory
// Notes:  This is a base class. It is the interface class for creating and
//     accessing a memory block that is sharable between processes and threads.
// ****************************************************************************

#ifndef MODULE_ipcSharedMemoryh
#define MODULE_ipcSharedMemoryh

// forward declaration
class osSharedMemory;

// class declaration
class ipcSharedMemory {

friend class osSharedMemory;

public:

   // constructor and destructor
   ipcSharedMemory(const char *name,    // unique name for creating block
                   long blocksize);     // requested size (in bytes)
   ipcSharedMemory(const char *name);   // name of block to open
   virtual ~ipcSharedMemory();

   // methods for getting memory block parameters [name, pointer to the block,
   // and whether this is the owner (creator) of the block]
   char *Name() const;
   void *Pointer() const;
   int Owner() const;


   // class version and object state data types
   enum version { MajorVersion = 1, MinorVersion = 0 };
   enum state { good = 0, bad = 1, badname = 2, notfound = 3 };

    // methods to get the object state
   inline int rdstate() const { return myState; }
   inline int operator!() const { return(myState != good); }
protected:
   osSharedMemory *myImpl; // implementation
   state myState;          // (object state (good, bad, etc.)
private:
   // private copy constructor and operator= (define these and make them
   // public to enable copy and assignment of the class)
   ipcSharedMemory(const ipcSharedMemory&);
   ipcSharedMemory& operator=(const ipcSharedMemory&);
};
#endif
Listing Two

// ****************************************************************************
// Module:  ipcshmem.C -- Author:  Dick Lam
// Purpose: C++ class source file for ipcSharedMemory
// Notes:  This is a base class.  It is the interface class for creating and
//   accessing a memory block that is sharable between processes and threads.
// ****************************************************************************

#include "ipcshmem.h"
#include "osshmem.h"

// ****************************************************************************
// ipcSharedMemory - constructor for creating

ipcSharedMemory::ipcSharedMemory(const char *name, long blocksize)
{
     // init instance variables
   myState = good;
   myImpl = new osSharedMemory(this, name, blocksize);
   if (!myImpl)
      myState = bad;
}
// ----------------------------------------------------------------------------
// ipcSharedMemory - constructor for accessing
ipcSharedMemory::ipcSharedMemory(const char *name)
{
   // init instance variables
   myState = good;
   myImpl = new osSharedMemory(this, name);
   if (!myImpl)
      myState = bad;
}
// ----------------------------------------------------------------------------
// ~ipcSharedMemory - destructor
ipcSharedMemory::~ipcSharedMemory()
{
   delete myImpl;
}
// ----------------------------------------------------------------------------
// Name - returns the name of the memory block
char *ipcSharedMemory::Name() const
{
   if (!myImpl)
      return 0;
   return myImpl->Name();
}
// ----------------------------------------------------------------------------
// Pointer - returns a pointer to the start of the memory block
void *ipcSharedMemory::Pointer() const
{
   if (!myImpl)
      return 0;
   return myImpl->Pointer();
}
// ----------------------------------------------------------------------------
// Owner - returns 1 if this is the owner (creator), and 0 otherwise
int ipcSharedMemory::Owner() const
{
   if (!myImpl)
      return 0;
   return myImpl->Owner();
}

Two member functions return the block name and a flag indicating whether the process or thread creates ("owns") or accesses the block. The Pointer() member function returns a void * pointer to the start of the memory block. The implementation of the member functions simply refers to the corresponding member functions in the implementation class osSharedMemory.

Listing Three is the header file used to create the implementation code for shared memory on individual operating systems. The only difference in the constructor arguments to osSharedMemory is the additional pointer to the ipcSharedMemory interface class. This pointer is kept so that the myState variable in the interface class can be modified by the implementation-level member functions. Note that osSharedMemory is a friend of ipcSharedMemory so the myState variable can be set directly in case an initialization error occurs.

Listing Three

// ****************************************************************************
// Module:  osshmem.h  -- Author:  Dick Lam
// Purpose: C++ class header file for osSharedMemory
// Notes:  This is a base class.  It contains general implementation methods
//         for memory blocks shared between processes and threads.
// ****************************************************************************

#ifndef MODULE_osSharedMemoryh
#define MODULE_osSharedMemoryh

#include "ipcshmem.h"

// class declaration
class osSharedMemory {

public:
   // constructor and destructor
   osSharedMemory(ipcSharedMemory *interface,const char *name,long blocksize);
   osSharedMemory(ipcSharedMemory *interface, const char *name);

   virtual ~osSharedMemory();

   // methods for getting memory block parameters [name, pointer to the block,
   // and whether this is the owner (creator) of the block]
   char *Name() const;
   void *Pointer() const;
   int Owner() const;
protected:
   ipcSharedMemory *myInterface;        // pointer to the interface instance
   unsigned long myID;                  // id of memory block
   char *myName;                        // shared memory block name
   int isOwner;                         // flag indicating owner

   void *myBlock;                       // pointer to the memory block

   // methods for handling the memory block
   void CreateBlock(long blocksize);
   void OpenBlock();
   void CloseBlock();
private:
   // private copy constructor and operator= (define these and make them
   // public to enable copy and assignment of the class)
   osSharedMemory(const osSharedMemory&);
   osSharedMemory& operator=(const osSharedMemory&);
};
#endif

The implementation header file also defines a block id required for the AIX and Windows NT implementations, along with CreateBlock(), OpenBlock(), and CloseBlock() methods that call the corresponding operating-system-specific shared-memory API functions.

The OS/2 implementation, os2shmem.C, is available electronically. The OS/2 API requires that all named shared-memory blocks have a name which starts with "\SHAREMEM\" (for example, "\SHAREMEM\TEST", "\SHAREMEM\MYBLOCK", and so on). Thus, memPath is defined at the top of the module as a constant string containing the name prefix, and this is prepended to the block name passed to the constructors to form the complete shared-memory-block name. The OS/2 API functions DosAllocSharedMem(), DosGetNamedSharedMem(), and DosFreeMem() are called to create, access, and close the memory block.

For AIX, the shared-memory API is handled similarly to the semaphore implementation--ftok() is called on a unique filename to get a key for use by the AIX IPC functions. In aixshmem.C (the AIX implementation is available electronically) the constructors prepend the string "/tmp/" to the input block name and then create a file with the full block name.

To create the block under AIX, the shmget() routine with an IPC_CREAT flag is used, and the memory is attached to the process with the function shmat(). Accessing the block is carried out the same way, except that the creation flag is omitted in the call to shmget(). The CloseBlock() member function calls shmdt() to detach the shared memory from the process, and calls shmctl() to remove the shared-memory id from the system. The osSharedMemory destructor also deletes the temporary file created in the constructor if the shared-memory owner is destroyed. Windows NT implements shared memory via file mapping, which allows you to treat a file as a block of memory. The CreateBlock() member function in winshmem.C (available electronically) calls the NT function CreateFileMapping() with an input handle argument of 0xFFFFFFFF. This tells the system to use the system swap file rather than an actual disk file to create a file-mapping object. The function returns a mapped file-object handle which is passed to MapViewOfFile() to get a pointer to the block of shared memory. The OpenFileMapping() function is called to access an existing shared-memory block; the memory is freed by calling UnmapViewOfFile() and CloseHandle().

Message Queues

The message queues I deal with here are quite distinct but similar in function to the event queues used in Windows or OS/2. These event queues work only for windowed applications, whereas message queues are also valid in character-mode sessions. OS/2 and AIX provide direct API support for message queues on the same platform, but Windows NT provides an alternative mailslot API (also available on other platforms) that I'll use. Mailslots can be used for intersystem communications, but I will limit this ipcMessageQueue implementation to IPC.

There is a distinction for message queues between the owner or creator of the queue, which in general is the server process, and the clients that access the queue. We allow clients of the queue write-only access, and queue owners read-only access. Member functions in the ipcMessageQueue interface class (see Listing Four and Listing Five) are also provided for queue owners to peek at the number of messages currently waiting in the queue, and to purge the queue of all messages.

Listing Four

// ****************************************************************************
// Module:  ipcqueue.h  -- Author:  Dick Lam
// Purpose: C++ class header file for ipcMessageQueue
// Notes:  This is a base class.  It is the interface class for creating and
//    accessing a message queue that handles messages between processes.
// ****************************************************************************

#ifndef MODULE_ipcMessageQueueh
#define MODULE_ipcMessageQueueh

// forward declaration
class osMessageQueue;

// class declaration
class ipcMessageQueue {

friend class osMessageQueue;

public:
   // constructor and destructor
   ipcMessageQueue(const char *name);       // unique name to create queue
   ipcMessageQueue(const char *name,        // name of queue to open
                   unsigned long powner);   // process id of queue owner
   virtual ~ipcMessageQueue();

   // methods for accessing the queue and queue parameters [name, queue id,
   // queue owner process id, and whether this is the owner (creator)]
   char *Name() const;
   unsigned long ID() const;
   unsigned long Pid() const;
   int Owner() const;
   // read/write methods for the queue (only a queue owner may read from
   // the queue, and only queue clients may write to a queue)
   virtual int Read(void *data, long datasize, int wait = 0);
   virtual int Write(void *data, long datasize);

   // methods to examine and remove messages from the queue (owner only)
   virtual unsigned long Peek();
   virtual int Purge();

   // class version and object state data types
   enum version { MajorVersion = 1, MinorVersion = 0 };
   enum state { good = 0, bad = 1, badname = 2, notfound = 3, notowner = 4,
           notclient = 5, readerror = 6, writeerror = 7, badargument = 8 };
   // methods to get the object state
   inline int rdstate() const { return myState; }
    inline int operator!() const { return(myState != good); }
protected:
   osMessageQueue *myImpl; // implementation
   state myState;     // (object state (good, bad, etc.)
private:
   // private copy constructor and operator= (define these and make them
   // public to enable copy and assignment of the class)
   ipcMessageQueue(const ipcMessageQueue&);
   ipcMessageQueue& operator=(const ipcMessageQueue&);
};
#endif



Listing Five

// ****************************************************************************
// Module:  ipcqueue.C  -- Author:  Dick Lam
// Purpose: C++ class source file for ipcMessageQueue
// Notes:  This is a base class.  It is the interface class for creating and
//   accessing a message queue that handles messages between processes.
// ****************************************************************************

#include "ipcqueue.h"
#include "osqueue.h"

// ****************************************************************************
// ipcMessageQueue - constructor for server
ipcMessageQueue::ipcMessageQueue(const char *name)
{
   // init instance variables
   myState = good;
   myImpl = new osMessageQueue(this, name);
   if (!myImpl)
      myState = bad;
}
// ----------------------------------------------------------------------------
// ipcMessageQueue - constructor for clients
ipcMessageQueue::ipcMessageQueue(const char *name, unsigned long powner)
{
   // init instance variables
   myState = good;
   myImpl = new osMessageQueue(this, name, powner);
   if (!myImpl)
      myState = bad;
}
// ----------------------------------------------------------------------------
// ~ipcMessageQueue - destructor
ipcMessageQueue::~ipcMessageQueue()
{
   delete myImpl;
}
// ----------------------------------------------------------------------------
// Name - returns the name of the queue
char *ipcMessageQueue::Name() const
{
   if (!myImpl)
      return 0;
   return myImpl->Name();
}
// ----------------------------------------------------------------------------
// ID - returns the queue id
unsigned long ipcMessageQueue::ID() const
{
   if (!myImpl)
      return 0L;
   return myImpl->ID();
}
// ----------------------------------------------------------------------------
// Pid - returns the process id of the Queue owner
unsigned long ipcMessageQueue::Pid() const
{
   if (!myImpl)
      return 0L;
   return myImpl->Pid();
}
// ----------------------------------------------------------------------------
// Owner - returns 1 if this is the owner (creator), and 0 otherwise
int ipcMessageQueue::Owner() const
{
   if (!myImpl)
      return 0;
   return myImpl->Owner();
}
// ----------------------------------------------------------------------------
// Read - reads a message from the queue (queue owner only)
int ipcMessageQueue::Read(void *data, long datasize, int wait)
{
    if (!myImpl)
       return bad;
    return myImpl->Read(data, datasize, wait);
}
// ----------------------------------------------------------------------------
// Write - writes a message to the queue (queue clients only)
int ipcMessageQueue::Write(void *data, long datasize)
{
    if (!myImpl)
       return bad;
    return myImpl->Write(data, datasize);
}
// ----------------------------------------------------------------------------
// Peek - returns the number of entries in the queue
unsigned long ipcMessageQueue::Peek()
{
   if (!myImpl)
      return 0L;
   return myImpl->Peek();
}
// ----------------------------------------------------------------------------
// Purge - removes all entries from the queue
int ipcMessageQueue::Purge()
{
   if (!myImpl)
      return bad;
   return myImpl->Purge();
}



In case direct access to the queue is required, the ID() member function returns the operating-system-specific queue handle. Also, there is a Pid() function that returns the process id of the queue owner. This is provided because the OS/2 queue API functions require clients to know the queue owner's process id in the call to DosOpenQueue(). Therefore, the ipcMessageQueue constructor for clients includes both the queue name and the process id of the server, while the server constructor only needs the queue name.

As before, the implementation details are delegated to an instance of osMessageQueue, which calls the system-specific API functions.

In addition to requiring the server-process id, the OS/2 queue API uses an event semaphore that can inform the server that a client has posted data to the queue. Consequently, the implementation-class declaration in Listing Six includes a pointer to an ipcEventSemaphore, which is created in the server constructor.

Listing Six

// ****************************************************************************
// Module:  osqueue.h  -- Author:  Dick Lam
// Purpose: C++ class header file for osMessageQueue
// Notes:  This is a base class.  It contains general implementation methods
//         for message queues for sending messages between processes.
// ****************************************************************************

#ifndef MODULE_osMessageQueueh
#define MODULE_osMessageQueueh

#include "ipcqueue.h"

// forward declaration
class ipcEventSemaphore;

// class declaration
class osMessageQueue {

public:
   // constructors and destructor
   osMessageQueue(ipcMessageQueue *interface, const char *name);
   osMessageQueue(ipcMessageQueue *interface, const char *name,
                  unsigned long powner);
   virtual ~osMessageQueue();

   // methods for accessing the queue and queue parameters [name, queue id,
   // queue owner process id, and whether this is the owner (creator)]
   char *Name() const;
   unsigned long ID() const;
   unsigned long Pid() const;
   int Owner() const;

   // read/write methods for the queue (only a queue owner may read from
   // the queue, and only queue clients may write to a queue)
   virtual int Read(void *data, long datasize, int wait);
   virtual int Write(void *data, long datasize);

   // methods to examine and remove messages from the queue
   virtual unsigned long Peek();
   virtual int Purge();
protected:
   ipcMessageQueue *myInterface;        // pointer to the interface instance
   unsigned long myPid;                 // process id of queue owner
   unsigned long myID;                  // id of queue
   char *myName;                        // queue name
   int isOwner;                         // flag indicating owner

   ipcEventSemaphore *mySem;            // required for OS/2 only

   // methods for handling the message queue
   void CreateQueue();

   void OpenQueue();
   void CloseQueue();
private:
   // private copy constructor and operator= (define these and make them
   // public to enable copy and assignment of the class)
   osMessageQueue(const osMessageQueue&);
   osMessageQueue& operator=(const osMessageQueue&);

};
#endif

There are also three protected member functions declared in osMessageQueue for creating, opening, and closing a queue.

The OS/2 implementation (os2queue.C, available electronically) of message queues requires that all queue names begin with "\QUEUES\", so a constant string variable (queuePath) is defined at the top of the OS/2 module. A call to DosGetInfoBlocks() in the server constructor is used to retrieve the server's process id, which is stored in the myPid variable. Finally, the event semaphore is created (using the same name as the queue input argument) and DosCreateQueue() is called. The client constructor simply forms the complete queue name and calls DosOpenQueue() to access the queue.

The osMessageQueue::Read() member function resets the event semaphore and then calls DosReadQueue() with either a DCWW_WAIT or DCWW_NOWAIT flag, depending on the value of the Read() function's wait input argument. The data are then copied to the input buffer, and the memory is freed. The Write() member function sets up an unnamed shared-memory block, gives the server process access to the block, and copies the data to be written into the block. The shared-memory block pointer, rather than a message-buffer structure, is then written directly to the queue. For each client write operation, the operating system automatically posts the event semaphore used in DosReadQueue().

For AIX, the server process creates a file by prepending "/tmp/" to the input name and calling creat()--this name is used in the ftok() function call (see aixqueue.C, available electronically). The message queue is created or opened through a call to msgget() and closed with msgctl(). The read/write operations are handled by allocating a special msgbuf structure that is passed to msgsnd() for writing and msgrcv() for reading.

Windows NT message queues are implemented using the mailslot API. Mailslots must have a name that begins with "\\.\mailslot\" for use on the same machine, so this string is defined at the top of the NT-implementation module (see winqueue.C, available electronically). The CreateQueue() member function calls the NT function CreateMailslot(), and the client constructor's OpenQueue() function call accesses the mailslot through a call to CreateFile().

Reading and writing to the mailslot is handled simply by calls to the NT functions ReadFile() and WriteFile(). The Peek() member function calls GetMailslotInfo() to return the number of waiting messages, and Purge() reads and discards each message until no further messages remain.

Test Programs

To test the shared-memory implementation, I've written a number of test programs (which are available electronically). The file, mbtest.h, defines a SharedVariables structure. The mbtest1.C program creates an ipcSharedMemory block, "myblock," large enough to hold the SharedVariables structure. The Pointer() function retrieves the pointer to the block of memory; the structure fields are initialized and their values are printed. The mbtest2.C program can then be started in a separate session to access the existing memory block, change the values of the structure fields, and print the results.

In practice, processes should synchronize their access to shared-memory blocks using a mutex semaphore. This controls access to the shared data to help provide data integrity and consistency.

The QMsg structure, defined in the file qtest.h, represents the contents of messages that will be written to the server process qtest1 in the file qtest1.C. The server test program creates a message queue named myque and starts the client program qtest2.C, passing the process id of the server as a command-line argument to the client. The client then opens the queue and writes several messages to the server, which reads messages and purges the queue before ending.

Summary

Shared-memory blocks are perhaps the fastest IPC mechanism, especially for transferring large structures between processes. However, they require careful synchronization, or subtle bugs can occur in complex programs or systems. Message queues are quite useful for one-way communications between a server (say a display process) and a number of clients (data-collection processes, for example). But the practical size of queue messages may be limited (particularly on AIX) to small chunks of information.


Dick is a member of the research staff at IBM's T.J. Watson Research Center. He can be contacted at rblam@watson.ibm.com.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video