Ron, author of the Tarma Simulation Framework, can be contacted at [email protected].
Imagine students peering into computer monitors and arguing over investments, price settings, and wages--all the time pointing to charts that show how well (or badly) their company is faring in a simulated economy. The game I'm describing is a distributed application that my company developed for a Dutch university. Its object is to teach students the causes and effects of both micro- and macro-economic decisions. It consists of a set of Windows-hosted applications, interconnected through a LAN (Novell NetWare, Banyan VINES, NetBIOS, and a shared-memory network simulation are supported), which form a simulated national economy, consisting of 30-50 companies, a government, and a banking sector.
The game is played in rounds that last anywhere between 1 and 30,000 seconds. A "normal" round lasts for 60-120 seconds. During each round, each company analyzes its position by means of its balance sheet, profit and loss statements, and production and inventory data. Spreadsheets and charts allow analysis of the corresponding historical data. Based on this information, its board of directors decides on its prices, investments, and so on. At the end of each round, current individual prices, investment and labor demands are fed into an economic model (created by the university for which the game was developed), and sales, labor allocations, and order stocks are distributed amongst the companies. The process then continues to the next round. A game normally lasts about two to three hours (or approximately 120 rounds).
In this article, I'll focus on the design of the game software and the network communications that allow it to run on top of several different protocols. As a bonus, I'll share a trick for dynamically loading C++ classes from DLLs.
The Software Design Process
The game software is based on an object-oriented model in which companies, the national economy, and the game controller are all objects. Each object is responsible for the part of the economic model that it represents: Companies keep track of their own inventories, production, balance sheets, and so on, while the national economy consolidates summary data from individual companies, and adds its own monetary policies and market-distribution mechanisms. The game-controller object is responsible for overall control of the game and the login/logout process. Clearly, communication between objects is required.
Figure 1 illustrates the basic communication pattern between the national-economy object and individual companies. Time runs from left to right. At the end of each round, the economy object uses up-to-date information from the company objects to perform the first series of calculations (which involve allocation of sales and labor, among other things) and relays this information back to the companies. On receipt of this information, each company finalizes the then-expired period, updates its balance sheet, inventory, and so forth, and prepares for the next round. Meanwhile, the economy waits until all companies have finalized the period, then performs its own finalization.
Publishers and Subscribers
The pattern in Figure 1 is appropriate for objects that live on a single computer, but less so for a truly distributed system. Network communications between objects living on different machines (or simply in different address spaces) require a more elaborate approach. As in many other distributed applications, communication is managed using a form of "proxies." Each company object becomes a remote publisher of information, while the economy object communicates with local subscribers to that information. Figure 2 shows this modified communication pattern; note that the economy object and the company subscribers live on one computer, while the company publishers live on several other different computers.
For efficiency, company subscribers cache the most important information from their publishers, which is updated as necessary by those publishers. This means that information pertaining to the current round is frequently updated as players make decisions regarding their company's policy, while at the end of a round, the information is frozen and added to the historical data that both publishers and subscribers maintain.
Publishers and subscribers are not restricted to a one-to-one relation; in general, a publisher can have any number of subscribers. Figure 3 shows a publisher's publication channel, to which subscribers may connect. The channel has a bus-like structure that allows the publisher to send updates to its subscribers by means of a multicast. This is not the only communication pattern, however. Individual subscribers may also request specific information from their publisher (as in a classic client/server relationship), and conversely, a publisher may request information from one of its subscribers. In this situation, the subscriber under consideration acts as an "author" of information. In the game software, this is primarily used when a previous game is being reopened from the central game repository: The company subscribers (acting as authors) located on the economy's computer are used to initialize their publishers to the state where the game left off. This approach is feasible for other applications: for example, a "blackboard" architecture in which several parties contribute to a set of common knowledge that is maintained by the publisher.
The publisher/subscriber approach is useful for other purposes, too. Each participating computer also displays a summary of the national economic data and, of course, the state of the game at large (current round, progress in the current round, and so on). Similar to the companies' information, economic and game information is distributed by the publisher/subscriber mechanism. The national economy object is turned into an economy publisher and all other computers are equipped with economy subscribers. The same applies to the game controller, whose subscribers display the simulated game date, time, and news bulletin messages broadcast by the "government." To keep the information from different sources separated, there are as many different publishing channels as there are publishers; in this game, this amounts to n+2, where n is the number of companies and the two additional channels are for the economy and the game controller.
Figure 4 illustrates a hypothetical situation involving four computers. The top computer is operated by the instructor. It contains publishers for game and economic information, and subscribers for all companies. Below it are two computers of teams that participate in the game; each has one subscriber for the game information and one for the economic data. In addition, they both house a publisher for their respective companies. Finally, at the bottom of the figure is a computer used by an outside observer, that subscribes to the game, the economic data, and all companies, but publishes nothing itself.
Publisher and Subscriber Classes
To capture the commonality of the publisher/subscriber pattern, I have developed a small hierarchy of classes; see Figure 5. The top-level class is called Actor and captures the common aspects of both publishers and subscribers, including the ability to communicate across the network through the use of a Port--an object that represents an endpoint in a network connection, similar to like-named entities in network protocols (sometimes called "sockets").
Derived from the Actor base class are the Publisher and Subscriber classes. Their purposes are reflected in their interfaces: A Subscriber object is aware of the network address of its Publisher, while the converse is not true. A representative part of the C++ declarations of these classes is shown in Listing One. Class cActor, contains a cPort object that takes care of the actual network communications, a number of functions relating to this port and the associated communication channel, and some functions for datagram management. These functions are, in fact, central to the purpose of the cActor hierarchy, which is to communicate across the network. At this level, communication is performed by means of a datagram abstraction (not shown here). A datagram contains fields to indicate the message type (for example, information update, shutdown notification), its class (request/response, broadcast, acknowledgment, and so on), some routing information, and the message contents. Internally, datagrams are kept in a preallocated pool, which is why cActors and their derivations must explicitly acquire and release the cDatagram instances.
Dispatch of incoming datagrams in the cActor hierarchy is done through message maps similar to those in Borland's OWL or Microsoft's MFC frameworks. The cActor class contains the dispatching mechanism proper, as well as a default message map that deals with a few predefined message types (such as diagnostics). Derived classes can add their own message handling or override existing mappings by including message maps in their own class declarations. Messages that do not appear in any map are handled by cActor::UnknownMessage(); by default, this function simply releases the datagram (in the debug version, it also produces a trace message to that effect). A similar function, cActor::IgnoreMessage() can be used to explicitly indicate that the message type is known, but needs no further handling. It, too, will release the datagram to its pool.
The derived classes cPublisher and cSubscriber specialize the default behavior by adding support for the maintenance of a publication channel, and the subscription to it, respectively. In particular, functions for the orderly shutdown of a channel are implemented for both parties. Only the cSubscriber class needs a message handler to cover this possibility.
Finally, you may have noticed that the actual allocation of publication channels is not covered by one of the cActor-derived classes; instead, this responsibility is delegated to a helper class of the game controller which oversees channel management in general and is not part of the cActor hierarchy.
Combining Actor Classes with Domain Classes
The cActor hierarchy knows nothing about companies, the economy, or any domain-specific information. To create specific publishers and subscribers, you must combine the cActor functionality with domain-specific classes. I'll use the company classes as an example; the other domain classes are dealt with in much the same way.
There are two basic company classes, cCompany and cCompanyPlayer; see Figure 6. Class cCompany represents the basic structure of a company, including its historical data. Class cCompanyPlayer extends this functionality by adding detailed information about the machine inventory and the company's policy, both of which are only available to the companies themselves, not to outside observers. Class cCompany is derived from a document-type base class provided by the application framework that we used to interface to the Windows environment; I used Borland's OWL, but other frameworks with similar document/view architectures would work.
To this basic company functionality, the publisher and subscriber functionality was added using the cPublisher and cSubscriber classes, respectively, as mix-in classes in a multiple-inheritance setting. Since there was no chance that both of them would appear as a base class for a further derived class, their common ancestor, cActor, didn't need to be virtual. The resulting classes are cCompanyPublisher (a player that also is active as a publisher), and cCompanySubscriber (an observer that tracks updates from its publisher). Each derived class has its own message map, which takes care of properly dispatching initialization and update broadcasts (in the case of the subscriber), and information requests (in the case of the publisher). In all, 14 different message types are currently defined for communication across a company publication channel. A similar number applies to both the economy and the game-control channels.
Network Communication Classes
As far as actors are concerned, a Port object is all they need to know about the network. Behind the scenes, however, a lot goes on to make this abstraction work. For one thing, the game has to run on top of several different LANs (Novell NetWare, Banyan VINES, generic NetBIOS, and occasionally AppleTalk) and several different operating systems. For the time being, I'll restrict my discussion to 16-bit Windows. Most of the implementation of the network layer runs on most of the platforms. Furthermore, the network protocol is freely selectable (given the presence of the corresponding network) and new protocols may even be added at run time.
We were able to accomplish this through abstraction and encapsulation in a number of classes. In effect, we designed our own transport layer that operates in terms of an abstract network protocol. For each of the network protocols that must be supported, we provided concrete implementations of the abstract protocol in terms of the API of the protocol under consideration--we used IPX for Novell NetWare, IPC for Banyan VINES, and datagrams for NetBIOS and AppleTalk. When we implement a TCP/IP version, we'll use UDP. Figure 7 provides for an overview of the relationships among the classes. Listing Two presents the corresponding class declarations.
Class cTransportManager takes responsibility for overall communication management. It exposes its services to the actors by means of the intermediate cPort objects that we first encountered in the cActor hierarchy. As shown in Listing Two, the cPort class offers its clients the ability to send datagram messages in several ways. Conversely, when a datagram is received, the cPort object will call back its cActor object and let it dispatch the datagram as dictated by the actor's message maps. Class cTransportManager does not create or destroy cPort objects, since they are normally assumed to be part of other objects, but does provide the means to connect them to and disconnect them from the network as appropriate. Furthermore, the cPort objects can use the cTransportManager::Send...() functions to forward the datagrams that are submitted by their own clients.
On the network side, cTransportManager uses the abstract interface of class cNetProtocol to get the datagrams from and to the port objects across the physical network. Class cNetProtocol is responsible for implementing some basic network services present in all network protocols considered. In the concrete derivations of the abstract cNetProtocol class, functions such as cNetProtocol::SendBroadcast() and cNetProtocol::SendMessage() map almost immediately to corresponding-protocol API functions as indicated earlier. We have also implemented a shared-memory network simulation, which allows us to test the network classes on a single computer. Originally, we did this for testing only, but this pseudonetwork protocol turned out to be quite useful for stand-alone demonstrations of the simulation game, and is now a standard part of the software distribution.
The final two classes in Figure 7 are cDatagram and cDatagramPool. Class cDatagram represents the actual datagram, as mentioned earlier; class cDatagramPool assists cTransportManager in the maintenance of a pool of these objects. There are several reasons for this pool. Datagram buffers must normally be present during (network) interrupts, since several of the network protocols use some kind of event-service routine on receiving or transmitting a datagram. In the 16-bit Windows environment, this implies that those buffers must be page locked. Since we need them often and without delay, it makes sense to preallocate an ample number of them and let them be managed by a separate class. Instead of new and delete, we use an acquire/release protocol to manipulate datagram buffers. (We could have overloaded operators new and delete, but they were already overloaded to allocate page-locked memory chunks, and we also didn't want frequent calls to constructors and destructors.)
Implementation of Network Services
The trio cPort/cTransportManager/cNetProtocol offers the following types of datagram transmission:
- Multicast to all ports connected to a given channel. Publishers use this to announce updates or send other information to all their subscribers at once.
- Point-to-point request and reply with guaranteed delivery, used by subscribers and publishers in a client/server fashion (where the publisher itself sometimes assumes the role of a client to an authoring subscriber).
- Point-to-point informational message without reply. This is used in particular during the shutdown of a node to announce its demise to the game-control publisher, and for acknowledgment messages.
The request/reply protocol is a straightforward implementation of a "request/reply with acknowledgment" algorithm with retransmission after a time-out, described in detail in books such as Distributed Systems: Concepts and Design, Second Edition, by G. Coulouris et al. (Addison-Wesley, 1994). The idea is to attach a unique identifier and an expiration field to each request, keep transmitted requests around until the corresponding reply is received, and retransmit the request if a time-out period expires without a reply. This may be repeated for several expiration periods, after which the other party is deemed unreachable and an error indication (instead of a reply) is returned to the original submitter of the request. If a reply is received, however, the request is satisfied, and an acknowledgment is sent to the replying party, which allows it to release any resources it might hold for retransmissions of the reply. In practice, the length of the time-out period, the maximum number of retries, and the expiration time of replies (in case acknowledgments are lost) are subject to the quality of the underlying network, the overall network load, the desired response times, and the risk one is willing to take of falsely declaring a node unreachable. In our implementation, these are all parameters that may be preset and that, to some extent, will adapt dynamically to the network conditions.
Dynamically Loading New Classes
To load new network protocol classes at run time without linking them into the code, place the class code in a DLL. You can call virtual functions through a vtable, which is a glorified jump table. Suppose that we knew the address of that jump table, and knew that index #3 would point to one function, #4 to another, and so on. If you implement class cTransportManager in terms of the (virtual) interface of class cNetProtocol, put them both in the application's executable, and at run time provide a pointer to a cNetProtocol-derived object in a DLL, you have:
- The pointer to the jump table (the vtable; its address can be found at some offset from where the object's this pointer points).
- The indices of the various functions in that table, since the C++ compiler courteously translates calls to virtual functions to look-up operations in that same jump table.
Conclusion
In this article, I've covered a lot of material in a short space. Still, I hope that I've shed some light on yet another distributed computing design, and perhaps also shown some useful patterns and implementation techniques for immediate application.
Figure 1: Communication between national economy and company objects.
Figure 2: Communication between economy and companies in a distributed environment.
Figure 3: Communication channel between a publisher and its subscribers.
Figure 4: Distribution of publishers and subscribers over participating computers.
Figure 5: Hierarchy of publisher and subscriber classes.
Figure 6: cCompany class hierarchy.
Figure 7: Class diagram for network communications.
Listing One
// Assume declarations of the following classes:
class cPort; // Network port abstraction
class cNetAddress; // Generic network address
class cDatagram; // Datagram message
// Abstract base class for Publisher & Subscriber
class cActor {
public:
// Public virtual destructor; anyone can delete an Actor.
virtual ~cActor();
// Functions to obtain network information
uint16 ChannelNo() const;
void NetAddress(cNetAddress &adr) const;
// Functions to interrogate & change connection state
void DisconnectPort();
bool IsConnected() const;
virtual void Shutdown() = 0;
virtual bool IsPublisher() const = 0;
protected:
// Constructor for use by derived classes
cActor();
// Access to the port object for derived classes
cPort & Port();
// Functions relating to datagram management
cDatagram * AcquireDatagram();
void ReleaseDatagram(cDatagram *);
virtual void IgnoreMessage(cDatagram *);
virtual void UnknownMessage(cDatagram *);
// Implementation of message dispatcher
bool DispatchMessage(cDatagram *);
// Default message table
DECLARE_MESSAGE_TABLE(cActor);
private:
// Actors own ports for their network communications.
cPort mPort;
};
// Publisher class
class cPublisher: public cActor {
public:
cPublisher();
// Functions to manage the publication channel
void BroadcastChannelDown();
virtual void Shutdown();
// Implementations of other cActor functions
virtual bool IsPublisher() const { return true; }
// Signature of the publisher
uint16 Signature() const;
};
// Subscriber class
class cSubscriber: public cActor {
public:
cSubscriber();
// Functions that set the server address of the client.
const cNetAddress &PublisherNode() const;
void SetPublisherNode(const cNetAddress &);
// Implementations of other cActor functions
virtual bool IsPublisher() const { return false; }
virtual void Shutdown();
protected:
// Default message responders
void OnChannelDown(cDatagram *);
virtual void ChannelDownAction() {}
// Subscriber message table
DECLARE_MESSAGE_TABLE(cSubscriber);
private:
// We keep the node address of our publisher
cNetAddress mPublisherNode;
};
Listing Two
// Network endpoint abstraction
class cPort {
public:
cPort(cActor *);
~cPort();
// Access to information regarding this port
cTransportManager *Manager() const;
void NetAddress(cNetAddress &) const;
uint16 ChannelNo() const;
// Function to check the connection state of the port
bool IsConnected() const;
void Disconnect();
// Functions to send messages to our peers in other nodes.
void SendBroadcast(cDatagram *);
void SendInfo(cDatagram *, const cNetAddress &);
void SendRequest(cDatagram *, const cNetAddress &);
void SendReply(cDatagram *);
private:
friend class cTransportManager;
// cPort instances are managed by the transport manager,
// organized by channel number.
cTransportManager *mManager;
uint16 mChannelNo;
// Pointer to the actor to be called back by the port.
cActor * mActor;
// Function to handle incoming datagrams
void ReceiveDatagram(cDatagram *);
};
// Transport manager
class cTransportManager {
public:
cTransportManager();
~cTransportManager();
// Interface to start the network connection.
int StartGroupAdmin(const char *);
int StartGroupMember(const cGroupInfo &);
// Stopping the network occurs in two phases
void StartShutdown();
void FinishShutdown();
// Function to enumerate the active groups.
int EnumGroups(cGroupList &);
int LookupGroup(cGroupInfo &);
// Node-level information functions.
bool IsActive() const;
bool IsAdmin() const;
const char * GroupName() const;
// Low-level protocol information functions.
cNetProtocol *Protocol();
const char * ProtocolName() const;
void NetAddress(cNetAddress &);
const cNetAddress *GroupAdmin() const;
// Interface to attach and detach ports.
void ConnectPortAt(cPort *, uint16);
void ConnectNextPort(cPort *);
void DisconnectPort(cPort *);
uint16 NextChannelNo();
// Functions for datagram buffer maintenance.
cDatagram * AcquireDatagram();
void ReleaseDatagram(cDatagram *);
// Functions to send datagrams.
void SendBroadcast(cDatagram *);
void SendRequest(cDatagram *, const cNetAddress &);
void SendInfo(cDatagram *, const TLNetAddress &);
void SendReply(cDatagram *);
void SendAck(cDatagram *);
// On reception of a datagram, ReceiveDatagram() is called.
void ReceiveDatagram(cDatagram *);
void DispatchDatagram(cDatagram *);
private:
// Current network protocol
cNetProtocol *mProtocol;
// List of connected ports and next available channel
cPtrArray<cPort> mPortList;
uint16 mNextChannel;
// A pool of datagrams is maintained by a subobject.
cDatagramPool mPoolMgr;
// Queues for reply and request transactions
int16 mNodeID; // Unique node ID
int16 mNextTid; // Transaction counter
cDataQueue mRequestQ; // Request queue
cDataQueue mReplyQ; // Reply queue
cDataQueue mSendQ; // Pending send queue
// Protocol maintenance
int OpenProtocol();
bool CloseProtocol();
bool IsProtocolOpen() const;
// Internal port maintenance
void DisconnectAllPorts();
// Function to maintain send and receive queues
void PostSends();
void CheckTransactions();
};
// Abstract base class for network protocols
class cNetProtocol {
public:
// Virtual destructor to cater for derivation
virtual ~cNetProtocol();
// Functions to initialize and terminate the protocol
virtual int InitProtocol() = 0;
virtual int TermProtocol() = 0;
virtual bool IsInited() const = 0;
virtual const char *ProtocolName() const = 0;
// Functions to open and close the network connection.
virtual int OpenConnection(const cNetAddress * = 0) = 0;
virtual int CloseConnection() = 0;
virtual bool IsConnected() const = 0;
virtual void NetAddress(cNetAddress &) = 0;
// A node must be able to advertise its address.
virtual int StartAdvertising(const char *) = 0;
virtual int StopAdvertising(const char *) = 0;
// Function to enumerate the active groups.
virtual int EnumGroup(cGroupList &) = 0;
virtual int LookupGroup(const char *, cNetAddress &) = 0;
// Functions to send datagram messages.
virtual bool SendBroadcast(cDatagram *) = 0;
virtual bool SendMessage(cDatagram*, const cNetAddress&)=0;
protected:
// Back pointer to transport manager
cTransportManager *mManager;
// Constructor for derived classes
cNetProtocol(cTransportManager *);
};
