Examining the SmartSockets Toolkit

Visual tools take the pain out of network IPC development

Doug is president of DC Systems and can be contacted at [email protected].

SmartSockets is a message-oriented, rapid application development toolkit for developing systems that require interprocess communication (IPC). SmartSockets makes it possible to embed messaging capabilities in your apps, allowing programs to communicate across different operating systems. The development environment, which runs on Windows 3.1/95/NT, OS/2, UNIX, and OpenVMS, includes visual point-and-click tools for developing, testing, and controlling networked, distributed applications. Processes can communicate across LANs, WANs, and the Internet; and SmartSockets takes care of network interfaces, handles communication protocols, and deals with recovery after system/network problems. The SmartSockets SDK includes an API, C++ class library, message router, visual point-and-click interface (for monitoring and debugging), 100-plus predefined reusable message types, and a command language.

My company recently faced the task of porting a major piece of software from DOS to Windows NT. As part of the conversion process, we used SmartSockets. Other tools included Visual C++ 2.2 (and 4.x when it became available) and Visual Source Safe for version control, all running under Windows NT 3.51 on an AST Premmia GX P133 with 32 MB of RAM. In this article, I'll focus on the SmartSockets toolkit, detailing the problems it solved for us.

Our Project

My company designs and implements real-time data-acquisition systems for the electric-utility industry. These systems collect large amounts of data from a wide geographic area and transport the data to a centralized control center. From a hardware perspective, these systems are truly distributed-the remote locations contain data-collection computers, and the control center uses a series of computers linked via a LAN. The control center and the remote locations are linked via a variety of media, such as leased phone lines, radio, microwave, and so on. The control center computers are a mix of PCs and UNIX workstations.

One of the components in the system is the communication front-end processor (FEP), a computer responsible for interfacing with the remote sites. The FEP is a PC with up to 64 serial ports attached to modems. In turn, each modem is connected to a remote site. The FEP also contains a network card that links it to the rest of the control-center computers.

In the first version of the FEP, we used TCP/IP sockets for the data link between the FEPs and the other control center computers. Unfortunately, we made the mistake of oversimplifying the task of managing a socket-based IPC mechanism. During the design phase, we thought, "No problem, we'll just build our own IPC using sockets." We then spent roughly two years learning about the nuances and pitfalls of this decision. Here are some of the lessons we learned:

• Different computer architectures have different data types. An int on a PC is different from an int on a UNIX workstation. An int on Win32 is even different from an int on Win16. An effective IPC scheme must manage the conversion of data between different architectures. (We looked at XDR, but decided against it since it more than doubles the number of bytes in a data stream.)
• Even though sockets provide reliable delivery, they do not guarantee delivery. If the socket connection is lost, you have no way to determine how much, if any, of the data made it through before the connection broke. In our case, part of the data we were transporting was alarm data, and it was not acceptable for an alarm to be lost or for duplicate alarms to arrive at a destination.
• As the number of distributed processes grew, the complexity of managing socket connections between them grew exponentially. If a socket is maintained between every process, the number of sockets grows out of hand quickly. This issue is also compounded by the need to manage socket connections dynamically. In our system, users on remote machines are logging on and off regularly.
• Managing the flow of data and determining how to distribute the data while making best use of the network bandwidth is a very complex issue. Our system handles a large amount of incoming data. It is not unusual to see data rates of 200 values-per-second on a sustained basis, and 1000 values-per-second during disturbances. We found it necessary to manage the distribution of this data from the FEP to the other control-center computers. This was especially important for users that were linked via WAN, with low-speed connections.

Additionally, one major user of the system required that the FEP provide self-diagnostics and contact us (via the Internet) when a problem occurred. To compound matters, NT required that we deal with Winsock for the socket interface.

Internal Data Types

SmartSockets solved our data format problems with its internal data types. It provides data types such as T_INT2 (16-bit int), T_INT4 (32-bit int), T_REAL4 (4-byte float), T_REAL8 (8-byte double), and so on. Using these data types, there was no worry about different machine data interpretations. In addition, when one of these data types is sent between machines with different byte orders, SmartSockets automatically converts the data so that when a floating-point number is put in at the PC end, it is read correctly when it arrives at the UNIX end. While this may seem trivial, it becomes complex when data is being sent to platforms as diverse as Windows NT, Windows 3.1, SunOS, Solaris, HP-UX, AIX, and the like.

Messages

SmartSockets uses the paradigm of a message as the basic unit transported between two processes. Messages can be used to notify another process of an event, request a RPC service, or transport data. A message contains fields for source, destination, priority, and zero or more data fields. While SmartSockets includes a number of predefined message types, we had to create a series of user-defined message types to efficiently transport our data. We found that we could transport any data structure by breaking it down into its member variables and inserting them into the data fields of a message. By using different message types, it was easy for both ends of the connection to know how to load and unload messages. In one case, we had to transport a large, opaque binary field; the SmartSockets binary field type allowed us to do this.

Building a user-defined message type was straightforward. First, a unique message-type identifier is declared, along with a name for the message and the grammar (data-field layout) of the message. The grammar is provided to support internal logging capabilities, which, during debug, will create a log file containing messages and their contents. Once the message type has been defined, an actual message of this type is created. Data is then loaded into the message, the destination is set, and the message is sent; see Listing One. Using SmartSockets, we were able to reduce the entire data-transfer task to simply constructing and sending messages.

Managing Data in the Application

One of the problems we encountered in our initial implementation was the task of managing the distribution of data from the FEPs to other processes that required the real-time data. This problem manifested itself in two ways:

• The FEP had to manage socket connections to every other process that required data. These connections had to be managed dynamically, since other processes could start and stop while the FEP was running.
• The FEP had to determine how to distribute data to the various processes. The FEPs maintain many different types of data, including device values, alarms, device status, and so on. All the data is not required by all processes. In fact, if we sent all the data to all the processes, we would quickly eat up available network bandwidth, especially to processes accessed via a WAN.

SmartSockets solved the connection-management problem via its IPC Manager. Rather than manage direct connections between all the various processes in the system, SmartSockets uses a central IPC manager. All processes create a single connection to the IPC manager. This single connection is used for all outgoing and incoming message traffic. The IPC manager also handles the connection and disconnection of processes dynamically.

We were able to solve the data distribution problem by making use of SmartSockets' "publish and subscribe" scheme. While an outgoing message can be directly addressed to the unique identifier of another process, it can also be addressed, or published, to a logical destination. Any process that is interested in receiving a particular type of data can subscribe to the logical destination. Thus, in our situation, all the FEP had to do was publish data to several logical destinations, such as device_data, alarms, history_data, device_status, and the like.

This scheme had the additional benefit of relieving the receiving processes of the need to poll for data. Typically, in this type of application, the client processes must continuously poll to get the latest data values. This chews up a lot of CPU time, especially since data is only changing at the rate of 1 percent per second. With SmartSockets, as data values change, the FEP publishes the new values. The SmartSockets IPC Manager then sends new values to all interested parties.

Making the Connection

To send and receive messages, the FEP has to make a connection to the SmartSockets IPC Manager. Once the connection is made, SmartSockets manages the connection and reconnects if it is broken. This is particularly important when the FEP is installed in a remote location and linked to the control center via a WAN. We found that socket connections over a WAN tend to break on a regular basis. Once we began using SmartSockets, we could stop worrying about this because the toolkit handled such situations.

Delivering the Data

When a remote field device changes state, its new value and the time of the change-down to the millisecond-are transmitted to the FEP. The FEP must then send this data to the processes that need it. If a piece of data gets lost, a critical operations error could result. We utilized the SmartSockets guaranteed message delivery (GMD) functions for critical data. When a message is sent with GMD, SmartSockets adds its own send/acknowledge layer to the transaction. If a message cannot be delivered, it is stored on persistent media, such as a disk file, until the connection can be reestablished. The message is then retransmitted. In addition to guaranteeing message delivery, SmartSockets GMD prevents duplicate messages from being delivered. This was important in the case of alarm messages. Listing Two presents the code for creating a connection and publishing data with GMD.

Receiving Messages

On the receiving end, SmartSockets was also easy to implement. In addition to acquiring data, the FEPs transmit control commands to the remote devices. Control commands are sent to the FEP, using SmartSockets messages. SmartSockets provides an API call that checks for the arrival of incoming messages and calls the appropriate message-type callback to unload the data from the message.

Within the FEP process, each serial channel is handled by a different thread. The connection to the SmartSockets IPC Manager is also controlled by a thread. We used thread-safe queues, built using the MFC collection classes, for interthread communication. Since SmartSockets is platform independent, it does not provide any Win32 event notification when a message arrives. This is one of the few limitations we found in the tool.

Consequently, we created a separate thread that performs a select on the socket connection and sets a Win32 event object when data arrives. The IPC manager thread is then able to use the Win32 WaitForMultipleEvent API to block until data arrives on either the IPC connection or the queue. The socket-select thread had to be synchronized with the socket-read thread, since the select thread must select for read until data arrives, notify the read thread, then wait for the data to be read. The read thread must notify the select thread once the read operation is complete. The select thread can then do another select for read.

The Internet Connection

One SmartSockets feature we did not recognize until near the end of the project is its ability to connect to an IPC manager that exists across a network, even the Internet. Since this is a mission-critical system, downtime had to be minimized. Using the SmartSockets Internet-connection feature, we were able to produce a self-diagnostic process that notifies us via the Internet if a problem occurs.

This was accomplished by running a SmartSockets IPC manager on our Internet server. In normal operation, the FEP is connected to a local IPC manager and data flows through the system. When a problem occurs, the self-diagnostic process connects to our IPC manager, via the Internet, and notifies us of the problem. Our technicians can then connect to the remote system via the Internet and perform additional tests to diagnose and fix the problem. In many cases, this can all happen before the customer is even aware that a problem exists.

For More Information:

Talarian Corp.

444 Castro St., Suite 140

Mountain View, CA 94041

415-965-8050

http://www.talarian.com

Listing One

#define USER_DEVICE_DATA 2000
T_IPC_MT userDeviceDataMsgType;

struct userDeviceDataBuffer {
  T_INT2 fepId;
  T_INT2 chan;
  T_INT2 devAdr;
  T_INT2 numDataFields;
  struct {
    T_INT2 pointNum;
    T_INT4 pointVal;
    T_INT4 quality;
    T_INT4 dataTime;
  } data[];
};
//----------------------------------------------------------------
// CreateDeviceDataMsgType
// Create a user defined message type
//----------------------------------------------------------------
void CreateDeviceDataMsgType(void)
{
// Create message type. Grammar is used to allow message logging to an ASCII
// file. Curly braces in grammar indicate fields that can be repeated.
// Message type: USER_DEVICE_DATA
// Purpose: Transport field device data
// Grammar:
//   Fixed fields:
//      int2 - FEP id
//      int2 - Channel
//      int2 - Device address
//   Repeated fields:
//      int2 - Point number
//      int4 - Point value
//      int4 - Data quality flags
//      int4 - Value time tag
userDeviceDataMsgType =
  TipcMtCreate("device_data",
               USER_DEVICE_DATA,
               "int2 int2 int2 {int2 int4 int4 int4}");
}
//----------------------------------------------------------------
// CreateUserDeviceDataMsg
// Create and return a UserDeviceData message.
//----------------------------------------------------------------
T_IPC_MSG CreateUserDeviceDataMsg(void)
{
T_IPC_MSG msg;
// Create a message to transport data
msg = TipcMsgCreate(userDeviceDataMsgType);
return msg;
}
//----------------------------------------------------------------
// LoadUserDeviceDataMsg
// Load a UserDeviceData message from the supplied buffer
//----------------------------------------------------------------
void LoadUserDeviceDataMsg(T_IPC_MSG msg, userDeviceDataBuffer* buf)
{
  TipcMsgAppendInt2(msg, buf->fepId);
  TipcMsgAppendInt2(msg, buf->chan);
  TipcMsgAppendInt2(msg, buf->devAdr);
  for (int x=0; x<buf->numDataFields; x++) {
    TipcMsgAppendInt2(msg, buf->data[x].pointNum);
    TipcMsgAppendInt4(msg, buf->data[x].pointVal);
    TipcMsgAppendInt4(msg, buf->data[x].quality);
    TipcMsgAppendInt4(msg, buf->data[x].dataTime);
  }
}

Listing Two

//--------------------------------------------------------------------------
// AcquireAndSendData. Get data from a channel and send it to other processes
//--------------------------------------------------------------------------
void AcquireAndSendData(void)
{
  T_IPC_MSG userDeviceDataMsg;
  userDeviceDataBuffer buffer;
  // create the new message type
  CreateDeviceDataMsgType();
  // create connection to the IPC manager
  TipcSrvCreate(T_IPC_SRV_CONN_FULL);
  // go get some data
  acquireData(buffer);
  // create a message to send data
  userDeviceDataMsg = CreateUserDeviceDataMsg();
  // load data into message
  LoadUserDeviceDataMsg(userDeviceDataMsg, buffer);
  // set message destination
  // this message goes to the logical "device_data" destination
  TipcMsgSetDest(userDeviceDataMsg, "device_data");
  // set GMD
  TipcMsgSetDeliveryMode(userDeviceDataMsg, T_IPC_DELIVERY_ALL);
  // send the message
  TipcSrvMsgSend(userDeviceDataMsg, TRUE);
}