Designing Servers with CPI-C
Achieving client/server portability
Peter J. Schwaller and John Q. Walker II
Peter, who currently develops ATM software in IBM's Networking Hardware Division, can be reached on CompuServe at 73602,3201. John also works in IBM Networking and can be reached on CompuServe at 72440,1544. They are the authors of CPI-C Programming in C: An Application Developer's Guide, published by McGraw-Hill.
Advanced Program-to-Program Communication (APPC), also known as LU 6.2, is software that enables high-speed communications between programs on different computers, from portables and workstations to midrange and host computers. APPC software is available for many different operating systems, either as part of the operating system or as a separate package.
APPC provides a rich set of functions for creating "conversations" between programs. Its original design, however, did not specify a common API for implementing these functions. Consequently, each operating system that originally supported APPC developed its own native APPC API. Until now, if you were designing APPC programs for different operating systems, you had to learn a distinctive verb syntax for each different platform.
The Common Programming Interface for Communications (abbreviated CPI-C, and pronounced "sip-ick") eliminates this problem. The CPI-C standard provides a consistent set of calls for all systems that support it. Although these calls correspond to APPC verbs, they are easier to use, since the names of the calls, constants, and variables are the same across all platforms and programming languages. Whether you are coding for Windows, OS/2, UNIX, AS/400, CICS, or MVS, you need to learn only one set of calls to write client/server applications for different systems.
Almost every CPI-C application is a client/server app. The client program starts the conversation by issuing a pair of CPI-C calls named Initialize_Conversation() and Allocate(); the server program connects by issuing an Accept_Conversation() or Accept_Incoming(). Often, many client programs want to connect to the same server program. In this article, we'll discuss server designs that handle multiple clients, even when the server's resources are constrained.
When the number of clients is small and the transaction rate low, it's okay to dedicate a server-program instance to each client. As the number of clients increases, all platforms will, at some point, run out of resources to support this operating model. Short conversations and accepting multiple conversations improve server-program throughput and work within resource constraints.
Using Short Conversations
Long conversations are maintained even when no work is being done. Although they can result in a lot of idle time, the startup cost of initializing long conversations is incurred only once. An application using short conversations, however, deallocates the conversation during idle times. This frees up the network and server resources for other clients or applications to use. The disadvantage of short conversations is the overhead of starting a conversation every time the client needs work from the server. However, that overhead is less than that of starting a new process.
On the server side, short conversations have the following advantages:
- Support for more clients. The server has a finite number of processes or threads that can be dedicated to serving clients. Short conversations let you "timeshare" available tasks to the clients. If all available tasks are busy, the next client waits for a task to become available. But, since you're using short conversations, the wait is almost always short.
- Increase in server throughput. To obtain the highest server throughput, we always want the server to have some work to perform. In fact, we would like a variety of tasks for the server to perform, to take advantage of the server platform's power (for example, disk I/O that can run in parallel with a calculation task). Short conversations reduce the amount of idle time in each server task. The chances of having useful work to perform increase because we don't have to dedicate a task to waiting on an inactive client.
- Decrease in the number of server-platform sessions. When many applications run between the client and server platforms by reusing sessions, but each conversation is active for only a short period of time, your application can use the session while another isn't using it. More applications can be run over fewer sessions. Sessions can also be brought down when they haven't been used for a period of time. This is done by configuring your connection as a limited resource.
- Recovery in case of connection failure. Less data must be recovered in the event of a short-conversation failure. In addition, the code to restart a conversation is already written as part of the short-conversation design, so the recovery and mainline logic are very similar. This results in more robust applications.
When breaking up long conversations, you should also determine the smallest transaction unit that can exist on its own in a single conversation. This transaction unit may span more than one request/reply, especially if the requests are related. The conversation startup should not become a significant portion of the total conversation time. If the conversations are too short, clients could spend most of their time starting conversations instead of getting work done. To illustrate how conversations can be broken up, let's examine a file-transfer program that sends a set of files from the client to the server. You could design this application with:
- Long conversations, where all files are sent on the same conversation: The client connects to the server, sends each file in succession, and requests confirmation of all files that were sent, and the conversation is deallocated. The server is tied up during the entire file transfer and cannot handle another client. If user input is required between files, there will be excessive idle time on the conversation. One problem in this design is error handling. If only one of the files sent cannot be written to disk, the server cannot interrupt the client without stopping all files already in transit. There are two choices for handling errors: Either the server uses Send_Error() whenever the error occurs and the client has to resend files that were already in transit; or the server has to receive the file that cannot be processed and discard the data, wasting network bandwidth if the file is large.
- Short conversations, where each file is sent on a separate conversation: The client connects to the server, sends a file, and requests confirmation that the file was stored successfully. The conversation is deallocated and the client goes through the previous steps for each file.
- Shorter conversations, where each data record of each file is sent on a separate conversation: The client connects to the server, sends a file data record, and requests confirmation. The conversation is deallocated and the client goes through the previous steps for each data record in the file, then repeats the process for each file. Since we're not sending very much data on each conversation and confirming after each send, the conversation overhead is likely not worth the cost.
- Excessively short conversations, where each file byte is sent on a separate conversation: The client connects to the server, sends a one-byte file-data record, and requests confirmation. The conversation is deallocated and the client goes through the previous steps for each byte in the file, then repeats the process for each file. The number of conversations started is equal to the number of bytes in all of the files combined (this is definitely not the way you should design your applications).
To correlate short conversations, use an existing data item as a correlator. In many instances, the resource that the server interacts with already has an identifier that could be used as a correlator; for example, a file server could use an operating-system file handle. If there is no acceptable existing data item to use, you may have to invent your own correlator. If so, consider using a combination of the client's LU name (from the Extract_Partner_LU_Name() call) and a unique integer ID generated by the server program.
One way to avoid correlating short conversations is to design a "stateless server," where each client request includes all of the information necessary to complete processing. Although this may result in more data in each request, the request can be handled independently of any other requests, past or future. In addition, the server is freed from having to maintain state information on each client. Thus, increasing the number of clients does not increase the server program's memory requirements.
Conversation-Startup Overhead
As we move toward using short conversations in servers, we start conversations more often. Thus, conversation-startup overhead becomes a bigger part of our performance concerns. To determine how to reduce it, let's look at the steps that occur when the client connects to the server and see how long each step takes. Assume the sequence of calls on the client and server shown in Example 1. At this point, the client has established a conversation and verified that the server program is running. The elapsed times assume a LAN transport and, thus, a short propagation delay.
The client's Initialize_Conversation() call pulls the necessary CPI-C parameters from a side information table. This is usually stored in memory while CPI-C is running and, therefore, is a very fast operation, usually on the order of tens of milliseconds, at most.
The client's Allocate() call first ensures that a session is available for use and allocates a conversation to it for use by the client program. The first time you Allocate() your conversation, session activation is performed, taking on the order of hundreds of milliseconds to complete. Subsequent Allocate() requests can reuse that session (serially, not simultaneously). Then, the only overhead of the Allocate() call is the matching of a conversation to an active session, which takes on the order of tens of milliseconds. Many configuration options exist to ensure that an active session will be available for use by your program. (Most programs are not concerned with session activation and have little control over it. Session activation is not normally a source of performance problems since it is usually done only once.)
Lastly, Allocate() puts an Attach into APPC's buffers to be sent to the server platform. The Attach contains all of the program-startup and security information for the conversation. In the client program in Example 1, Confirm() flushes the Attach and sends it to the server platform along with the confirmation request.
On the server platform, the processing of the Attach header itself is usually simple, taking only about 20 msecs. If the server program is already running, Accept_Conversation() gets the conversation ID, and we're off and running. If the server program is not already running, the server platform will have to start the program. The overhead to start a program varies among platforms, but a good rule of thumb is that program startup usually takes between 1 and 10 seconds to complete. In the server program in Example 1, Receive() and Confirmed() take about another 10 msecs to complete. Table 1 summarizes where the time is spent.
Program startup is the last major element of startup overhead, and its time varies from platform to platform. On a system like CICS, which was designed for quick program startup and takedown, program startup is likely to be less than 10 msecs. Although normal program-startup time on OS/2 is around 1 to 2 seconds, a slow PC running OS/2 with little memory could take minutes!
To limit program-startup time, it's best for the server program to be running when an Attach arrives from the client. Ideally, you would like to start one copy (or many copies) of the server program and have it accept one conversation after another without ending their processes.
Since we're looking for optimal performance and we're using short conversations, we cannot afford to start a copy of our server program for each conversation. (An exception is CICS, which is optimized to make program load blindingly fast.)
Starting the server program is usually the biggest part of conversation-startup overhead. To avoid program-startup costs, we would like to design our server program to accept multiple conversations without exiting.
Accepting Multiple Conversations
Starting with CPI-C version 1.2, programs have been able to accept multiple conversations within a single program. Your programs can now handle multiple conversations or multiple clients without the overhead of program startup for each conversation.
Accepting multiple conversations in CPI-C 1.2 is easy; just issue another Accept_Conversation() call. The easiest way to convert your programs to accept multiple conversations is to add a loop around your main processing. In Figure 1, for instance, the program should exit whenever an Accept_Conversation() call fails. An Accept_Conversation() failure usually indicates one of the following:
- The program is running on an old CPI-C platform that doesn't support accept multiple, so your program will never be able to accept a new conversation.
- The TP definition for the server program isn't set up correctly to accept multiple conversations. For example, on the OS/2 Communications Manager, a TP definition can specify that it is nonqueued, meaning that the attach manager should start a new instance of the program running for each incoming Attach.
- No incoming conversation arrived within a time-out period. There wasn't an incoming Attach in a specified time period. Rather than tying up resources longer than necessary, you should end your program and free up those resources. Let the attach manager start a new server program when necessary. The time-out period is usually a configuration option.
Although not specifically a CPI-C function, you can use multiple threads within your server to handle multiple conversations simultaneously. Using multiple threads allows your server to handle multiple clients without the overhead of multiple processes. More clients are serviced with fewer server resources. You can use multiple threads in your server programs in many different ways. We'll examine two:
- A main thread accepts conversations, then starts worker threads to process each conversation. This allows your server to process all the client conversations that arrive, up to the system thread limit. This technique is useful only if the number of conversations is not expected to grow beyond the thread limit. If your program does reach the thread limit, it is difficult to determine when threads are free to accept new conversations again.
- A set of N threads are started. Each accepts and processes conversations in a loop. This allows your program to explicitly specify how many threads and, thus, how much resource it will use up. The actual number of threads may be tuned to provide the best server throughput without overloading or thrashing the server platform. Listing Three is an example of the server program adapted to accept many conversations and start a thread to process each.
The disadvantage is the extra overhead required for nonblocking processing. Although the overhead will be less than that for implementing nonblocking using threads, a nonblocking call is more expensive than a normal procedure call. Furthermore, your program must supply and maintain parameters for each nonblocking call it issues. CPI-C keeps the addresses of your parameters until the nonblocking call completes. If your program issues four nonblocking Receive() calls, you must have four sets of Receive() parameters, including four Receive() buffers. If you are using nonblocking calls, we recommend using C structures to keep the sets of parameters together as one unit. Finally, your program must maintain complete state information for each conversation.
When the nonblocking call completes, you are only told the conversation ID and the return code. Your program must remember what CPI-C call actually completed and what call should be issued next on that conversation.
Conclusion
CPI-C is a powerful API for creating client/server applications. Early versions of CPI-C made it easy to build portable clients, but server programs were limited in their capacity. As CPI-C has become an industry standard, it has been enhanced to allow building powerful servers, as well.
References
Walker, John Q. II and Peter J. Schwaller. CPI-C Programming in C: An Application Developer's Guide to APPC. New York: McGraw-Hill, 1994. ISBN 0-07-911733-3.
The Best of APPC, APPN, and CPI-C. IBM CD-ROM #SK2T-2013.
Example 1: Usual sequence of startup calls. (a) Client; (b) server.
(a) Initialize_Conversation() Allocate() Confirm() (b) Accept_Conversation() Receive() Confirmed()
Table 1: Conversation-startup overhead.
Initialize <10 msec Session activation 100--1500 msec Conversation allocation <10 msec Attach About 20 msec Program startup About 1--10 sec
Figure 1 Accepting multiple conversations.
Listing One
/*---------------------------------------------------------------
* CPI-C example program, displaying received records
* server side (file SERVER1D.C)
*-------------------------------------------------------------*/
#include <cpic.h> /* conversation API library */
#include <stdio.h> /* file I/O */
#include <stdlib.h> /* standard library */
#include <string.h> /* strings and memory */
#define RECEIVE_SIZE (10) /* receive 10 bytes at a time */
static void process_incoming_data(unsigned char *conversation_ID);
int main(void)
{
unsigned char conversation_ID[CM_CID_SIZE];
CM_RETURN_CODE cpic_return_code;
setbuf(stdout, NULL); /* assure unbuffered output */
do {
cmaccp( /* Accept_Conversation */
conversation_ID, /* returned conversation ID */
&cpic_return_code); /* return code from this call */
if (cpic_return_code == CM_OK) {
printf("Accepted a conversation...\n");
process_incoming_data(conversation_ID);
}
else {
(void)fprintf(stderr,
"Return code %lu on CMACCP\n", cpic_return_code);
}
} while (cpic_return_code == CM_OK);
(void)getchar(); /* pause for a keystroke */
return(EXIT_SUCCESS);
}
static void process_incoming_data(unsigned char *conversation_ID)
{
unsigned char data_buffer[RECEIVE_SIZE];
CM_INT32 requested_length = (CM_INT32)sizeof(data_buffer);
CM_INT32 received_length;
CM_DATA_RECEIVED_TYPE data_received;
CM_REQUEST_TO_SEND_RECEIVED rts_received;
CM_STATUS_RECEIVED status_received;
unsigned done = 0;
CM_RETURN_CODE cpic_return_code;
while (done == 0) {
cmrcv( /* Receive */
conversation_ID, /* conversation ID */
data_buffer, /* where to put received data */
&requested_length, /* maximum length to receive */
&data_received, /* returned data_received */
&received_length, /* length of received data */
&status_received, /* returned status_received */
&rts_received, /* ignore this parameter */
&cpic_return_code); /* return code from this call */
/* replace the following block with the good algorithm
* that's shown in the program sketch in the text. */
if ((cpic_return_code == CM_OK) ||
(cpic_return_code == CM_DEALLOCATED_NORMAL)) {
/* write the received string to stdout */
(void)fwrite((void *)data_buffer, (size_t)1,
(size_t)received_length, stdout);
if (data_received == CM_COMPLETE_DATA_RECEIVED) {
(void)fputc((int)'\n', stdout); /* newline */
}
}
if (cpic_return_code != CM_OK) {
done = 1; /* CM_DEALLOCATED_NORMAL or unexpected */
}
}
}
Listing Two
/*-----------------------------------------------------------
* CPI-C example program, sending command-line parameters.
* Client side (file HELLO5.C)
*-------------------------------------------------------------*/
#include <cpic.h> /* conversation API library */
#include <string.h> /* strings and memory */
#include <stdlib.h> /* standard library */
#include <stdio.h> /* standard I/O */
/* this hardcoded sym_dest_name is 8 chars long & blank padded */
#define SYM_DEST_NAME (unsigned char*)"SERVERS "
int main(int argc, char *argv[])
{
unsigned char conversation_ID[CM_CID_SIZE];
CM_RETURN_CODE cpic_return_code;
cminit( /* Initialize_Conversation */
conversation_ID, /* returned conversation ID */
SYM_DEST_NAME, /* symbolic destination name */
&cpic_return_code); /* return code from this call */
if (cpic_return_code != CM_OK) {
printf("Error on CMINIT, RC was %ld\n",
cpic_return_code);
}
cmallc( /* Allocate */
conversation_ID, /* conversation ID */
&cpic_return_code); /* return code from this call */
if (cpic_return_code != CM_OK) {
printf("Error on CMALLC, RC was %ld\n", cpic_return_code);
}
{
/* send each command-line argument, one per send */
int index;
for (index = 0; index < argc; index++) {
CM_REQUEST_TO_SEND_RECEIVED rts_received;
CM_INT32 send_length = (CM_INT32)strlen(argv[index]);
cmsend( /* Send_Data */
conversation_ID, /* conversation ID */
(unsigned char *)argv[index], /* send this */
&send_length, /* length to send, no null */
&rts_received, /* ignore this parameter */
&cpic_return_code); /* return code */
if (cpic_return_code != CM_OK) {
printf("Error on CMSEND, RC was %ld\n", cpic_return_code);
}
}
}
cmdeal( /* Deallocate */
conversation_ID, /* conversation ID */
&cpic_return_code); /* return code from this call */
if (cpic_return_code != CM_OK) {
printf("Error on CMDEAL, RC was %ld\n", cpic_return_code);
}
return(EXIT_SUCCESS);
}
Listing Three
/*---------------------------------------------------------------
* CPI-C example program, displaying received records
* server side (file SERVER2D.C)
*-------------------------------------------------------------*/
#include <cpic.h> /* conversation API library */
#include <stdio.h> /* file I/O */
#include <stdlib.h> /* standard library */
#include <string.h> /* strings and memory */
#include <process.h>
#define RECEIVE_SIZE (10) /* receive 10 bytes at a time */
static void process_incoming_data(void *void_conversation_ID);
int main(void)
{
unsigned char * conversation_ID;
CM_RETURN_CODE cpic_return_code;
int thread_id;
setbuf(stdout, NULL); /* assure unbuffered output */
do {
conversation_ID = malloc(CM_CID_SIZE);
if (conversation_ID != NULL) {
cmaccp( /* Accept_Conversation */
conversation_ID, /* returned conv ID */
&cpic_return_code);
if (cpic_return_code == CM_OK) {
printf("Accepted a conversation...\n");
thread_id = _beginthread(
process_incoming_data,
NULL, /* have C allocate the */
/* stack for the thread */
8192, /* specify stack size */
(void*)conversation_ID);
if (thread_id == -1) {
perror("Error creating thread.");
}
}
else {
(void)fprintf(stderr,
"Return code %lu on CMACCP\n", cpic_return_code);
}
}
else {
printf("Error getting memory!\n");
cpic_return_code = -1;
}
} while (cpic_return_code == CM_OK);
(void)getchar(); /* pause for a keystroke */
return(EXIT_SUCCESS);
}
static void process_incoming_data(void * void_conversation_ID)
{
unsigned char data_buffer[RECEIVE_SIZE];
CM_INT32 requested_length = (CM_INT32)sizeof(data_buffer);
CM_INT32 received_length;
CM_DATA_RECEIVED_TYPE data_received;
CM_REQUEST_TO_SEND_RECEIVED rts_received;
CM_STATUS_RECEIVED status_received;
unsigned done = 0;
CM_RETURN_CODE cpic_return_code;
unsigned char * conversation_ID = (unsigned char *) void_conversation_ID;
while (done == 0) {
cmrcv( /* Receive */
conversation_ID, /* conversation ID */
data_buffer, /* where to put received data */
&requested_length, /* maximum length to receive */
&data_received, /* returned data_received */
&received_length, /* length of received data */
&status_received, /* returned status_received */
&rts_received, /* ignore this parameter */
&cpic_return_code); /* return code from this call */
/* replace the following block with the good algorithm
* that's shown in the program sketch in the text. */
if ((cpic_return_code == CM_OK) ||
(cpic_return_code == CM_DEALLOCATED_NORMAL)) {
/* write the received string to stdout */
(void)fwrite((void *)data_buffer, (size_t)1,
(size_t)received_length, stdout);
if (data_received == CM_COMPLETE_DATA_RECEIVED) {
(void)fputc((int)'\n', stdout); /* newline */
}
}
else {
printf("unexpected error %lu\n", cpic_return_code);
}
if (cpic_return_code != CM_OK) {
done = 1; /* CM_DEALLOCATED_NORMAL or unexpected */
}
}
free(conversation_ID);
}
Copyright © 1995, Dr. Dobb's Journal
