Distributed Computing Now: Development Environments
Middleware takes the pain out of distributed computing application development
Lowell S. Schneider
Lowell, a cofounder of Ellery Systems, has 25 years of software-development experience, most of which have been spent in distributed database and computing systems. He can be contacted at [email protected].
In the first (June 1993) installment of this article, we discussed both the NASA Astrophysics Data System (ADS) and the Earth Data System (EDS), applications typical of emerging distributed computing systems. This month, we'll look under the hood and examine the type of tools and techniques required to build large-scale distributed systems like the ADS.
The environment in which the ADS and EDS are written is the Ellery Open Systems (EOS), an interpreted runtime middleware that runs on top of the OSF standard Distributed Computing Environment (DCE), allowing access to existing programs as DCE servers. EOS was designed to provide programmers with scant knowledge of the DCE API a way to begin using DCE at minimum cost. Using a language called C-Lite (C-Like Interpreted Teleprocessing Environment), EOS gives DCE what Basic gave the PC over a decade ago: the ability to type a few lines of code, execute, and keep doing it until you get it right. The difference is that instead of controlling an 8088, you're in control of an entire wide-area network of high-speed workstations. EOS also provides a complete application-development environment (including a Motif user-interface server) and tools for source debugging, performance profiling, and test-coverage analysis.
The original EOS was based on a European DCE-like product called ANSA (Advanced Network System Architecture) designed by APM (Cambridge, U.K.) That version is still supported because: 1. Not everyone has a workstation that can support DCE (the DCE runtime requires at least 32 Mbytes of real memory and 100 Mbytes of swap space) and the ANSA version of EOS will run on an old 8-Mbyte SPARCstation1; and 2. not all vendors are yet ready to ship DCE. The architecture described in this article is the DCE version which, although very different from the ANSA version, allows applications written for EOS/ANSA to run on EOS/DCE without change.
With almost 500 functions, the DCE API can be daunting. That's not meant to knock DCE: All 500 functions are important. But most of the DCE API solves problems you don't have yet unless you're already into distributed computing. Furthermore, what if you start a project to develop a native DCE application from scratch and you find it's going to cost three times what you budgeted, or that DCE doesn't provide everything you need. Are you left with anything that you can use? Probably not. If you started from scratch, you wrote all your "manager code" (the DCE naming convention for that part of the server that implements the application) as pthreads created by the server stub, and it's not going to port back to straight C very easily.
The alternative is to write to a middleware environment like EOS which, depending on the scenario, lets you build a distributed DCE application with little startup and no throw-away costs.
EOS Server Architecture
The EOS server architecture is a runtime veneer on top of DCE that's both compatible with, and complements the functionality of, DCE. The principal EOS server runtime is the remote process invocation (RPI) daemon. It's a highly replicated, highly available process that fields client requests for server bindings, forwarding them as needed to other RPIs. An RPI determines whether it can satisfy a request based on a generalized property specification and constraint language. If an RPI satisfies a client's request, it forks the other EOS server runtime, the session-management server (SMS), to manage the binding for as long as it endures. Underneath the SMS are the servers themselves, which are merely ordinary UNIX filters that read stdin and write/flush stdout. Bindings to these servers can have up to six orthogonal attributes, all of which are dealt with by the EOS runtime transparently to the server:
- Local or remote.
- Modal or modeless.
- Session or sessionless.
- Protected or unprotected.
- Authenticated or unauthenticated.
- Authorized or unauthorized.
With RPI, servers run when needed by clients, as opposed to being started and administered as separate servers that run continuously. This is important in two regards. First, we've learned from experience that if the reason for distributed computing is to integrate already distributed applications, then there are (very quickly) a large number of servers available, most of which will be used only occasionally. This is in sharp contrast to the situation that arises when you build a specific application (such as an accounting system) in which there are one or a few servers being used continually. Second, we've learned that DCE servers are huge processes. The executables may be deceivingly small if they're linked with shared libraries, but in fact, a null DCE server (one that does nothing at all) will have a 2.5-Mbyte text image on an HP720. Consequently, if you're going to offer many different services and you don't have something like the EOS RPI, you're going to need a lot of machines to run them.
The next problem that arises when you have a large number of servers is finding the one you want. The name-service interface (NSI) API in DCE is well suited to the accounting system alluded to earlier, for multiple instances of the same service (such as print servers), or any similar situation in which you have a small number of interfaces; see Figure 1. This is because the cell-directory service (CDS) allows the client to search for servers based on interface type; see Figure 2. But it provides only minimal functionality for searching for servers based on attributes. That is, if you integrate a bunch of disparate databases through a common interface and you want to query one of the tables, knowing the interface type isn't enough. You have to have some way of asking whether the instance of the server offering that interface also has access to the table you want. With CDS, you can do this with an object uuid, but it's really difficult and is essentially just a place-holder for the DCE Common Object Request Broker Architecture (CORBA), which isn't part of DCE yet. Until it is, EOS adds some of this functionality by providing a context-free property specification and constraint language based on ANSA. (The property specification and constraint language was developed by Dr. J. Sventek of Hewlett-Packard as part of the Advanced Network Systems Architecture [ANSA] while he was seconded to APM Ltd., Cambridge, U.K.)
When an RPI is started, it looks for a .proplist file, an ordinary text file that might look something like Example 1(a). The properties (TABLE, for example) are any arbitrary name and can have singular string, string-list, or numeric values. The RPI reads this file when it starts, and adds to it this list of its runtime properties; see Example 1(b). It then searches for a number of other files to create two more built-in lists, the sblist (server body) list, and the sbdescrip (server-body description) list.
The first file it searches for is .rpirc, which lists the servers it's supposed to advertise. Unlike the example I'll be illustrating, where I start RPI with a flag that tells it to advertise all executables in its directory, it's more typically the case that: 1. You have an environment variable (EOSSERVERS) that's a path variable to all the different directories in which servers live; and 2. not all the executables in those directories are servers (for example, some are still in development). Then, for each server listed in .rpirc, it looks for a file of the same name, prefaced with a dot, that contains a brief text description of what the server does and/or how to get the equivalent of man pages on how to use the server.
On the client side, the PROPV[] component of the SERVER structure is initialized with a series of constraints, the logical AND of which will be the client's expression of what it wants; see Example 2. At this point, the client has two choices. It can merely broadcast this binding request by calling the C-Lite function rpi(); if a match is found, the RPI that matched will return its binding information, which the client will use on the subsequent INITIALIZE. When the INITIALIZE entry point is called, the RPI will fork/exec the session manager and return the binding information for the session manager to the client. This is the binding the client uses for the duration of that connection. If the client chooses this approach, almost everything that happens is behind the scenes, as viewed from C-Lite. The other choice is to call the C-Lite function servers(), optionally including a constraint expression, as in Listing One (page 100). This establishes the bounds of the search on the first call, and on all subsequent calls until a null binding is returned, to get the complete property list of every matching rpi. The client can programmatically (or interactively with a user) examine the other properties and choose a particular RPI that best meets its requirements. It then calls rpi(), with the only PROPV[] being the binding property that uniquely matches only one RPI. One possible reason for using this approach is to achieve load balancing.
Building an EOS-based Application
The example scenario I'll use here assumes you want to integrate two existing applications: The first is financial simulation that takes the value of your stock portfolio, the name of a stock, and some statistics about the stock's recent performance and returns the number of shares you should buy or sell (or 0 if you sit tight). It runs on a high-speed server on your network and the name of the executable is "finsim."
The second application is a broker application that takes the name of a stock and the number of shares to buy (or sell if the number is negative) and electronically delivers the buy/sell orders to your broker. In the new application, the client will read a table in which each row represents one stock in your portfolio, call the simulation model, and if it returns other than 0, will place a buy or sell order via the broker server. It runs on many different hosts and the executable is "sbroker." (For the time being, I won't address user-interface issues.)
The Servers
Building the servers is straightforward under EOS, particularly if the servers are already written in the UNIX paradigm--that is, they read a line from stdin and write a line to stdout. If the servers aren't UNIX-like but are written in a language that supports stdin/stdout (C, for instance) and you have access to the source, then you only need to change how the server reads its input parameters so that it uses gets() and how it writes its output parameters so it uses puts() (or readln()/writeln() in Pascal, or READ(5,*)/WRITE(6,*) in Fortran). If the server writes multiple lines of output per input line, you need to marshal these into a single output line, typically as a C initializer string such as {2,{{\"line0\",\"line1\"}}}) that C-Lite can ingest. If you don't have access to the source, or the language doesn't support stdin/stdout, then you'll need to write a simple C filter that fork()/exec()s the application and deals with user-interface peculiarities. An example would be a prompting interface that takes one line of input, produces multiple output lines, and then puts the prompt up without a newline, so you need to check the beginning of each line for the prompt and discard it before you gets() so you don't block. This isn't as difficult as it sounds. We've written hundreds of such wrapper programs, and the only hard part is figuring out the trick you need to play on the application to get it to do what you want. Once you know that, the whole wrapper (exclu
ding the trick) is less than a hundred lines of C. To illustrate, assume that the sbroker server is a program for which you don't have any source, and it implements a command-line prompting interface and a minimal command language, like Example 3, which is typical of legacy code.
Listing Two (page 100) is a C wrapper program that behaves like a filter; it reads commands from stdin and writes a file of orders. It also makes the input file one line per order instead of three, allowing us to call it as a server just once per order, so you can exec it from a shell with: sbroker<orders>&/dev/null. Debugging the servers is straightforward. Assuming you already have programs that read and write stdout, you can debug them as you would any filter--from the shell, using your debugger.
The Client
To build an EOS client, you use C-Lite. Becoming familiar with C-Lite isn't difficult, particularly if you're familiar with C syntax. C-Lite semantics, on the other hand, are Basic-like, and use a string as the basic unit instead of a character. Consequently, everything is passed by value unless you explicitly pass pointers, and memory is managed automatically.
The skeletal client code (see Listing Three, page 100) imports the model server and the broker server, then opens a table. For each row, the client calls the model server with an argc/argv structure representing the contents of the row, and if the model server returns anything other than 0, the client calls the broker with the name of the stock and the number of shares to trade. When it's done, it closes the table and returns. To keep it simple, there's no error checking.
To recap: I first declared some local variables. The next variable is in a new kind of storage class called "stable" that has a global scope; the value of this variable is maintained across sessions. I then declared two SERVER structures. When I initialized the components of the model SERVER structure, I set the TYPE=
112000, which means that the server binding is remote, modal, sessionless, and that there are no protection, authentication, or authorization requirements. Then I set CONTEXT (which is like the UNIX PATH variable) to /.:/hosts/ferrari /.:/ which means try to bind first on the host called "ferrari" (the fast one), and failing that, bind to any host in the cell. Finally, I set PROVP[0] to 'finsim' in sblist, meaning the only property required is that there be a finsim server there.
Next, the *model=rpi(*model) statement broadcasts this request, which is fielded by the first RPI daemon that satisfies our needs; that is, it can run a finsim server. When this call succeeds, the returned structure has a number of other components in the structure filled in with function pointers to the various entry points in the interface to the server, one of which is INITIALIZE. The next statement, (model->INITIALIZE)(), does three things: 1. It declares the function symbol "model" that will be bound to the server; 2. it starts an SMS to handle all the RPC and session semantics; and 3. it tells the SMS that the server's shutdown command is quit. The SMS, in turn, execs finsim hooked to pipes. The initialization of the broker structure is essentially the same, except that we give no host preference. We then change to the directory with the portfolio table and attach the table to a data window. When we do this, all the columns of the table become implicitly accessible using a table.column.row syntax. In this syntax, a missing component denotes the current component, in this case the row, which is made current by the nextrow() function. A * in the column component means all columns are marshaled as an argc/argv[] structure. If model() returns 0, we continue the loop, otherwise we call the broker with the value of the Name column in the table, and the number of shares model() returned. The last two statements instruct the session manager to shut down the server, and to shut itself down if it's not managing any other servers. And that's it: About 30 lines of C-Lite is interpreted into about 30,000 lines of C and DCE API.
Debugging the Client/Server Interface
Remote debugging is hard. With all the technology that leads up to and is now offered in the DCE, no more than trivial logging and tracing facilities are offered in this regard. While EOS provides a Motif mouse-driven source debugger (modeled on HP Softbench) for C-Lite, it leaves open the issue of how you debug the client/server interface and, instead, makes it a nonproblem. In the above example, I said the TYPE component specified a remote, modal, sessionless connection. If you change
foo->TYPE to 212000, it means a local, modal, sessionless binding, so the same server code you write can run remotely or locally without modification. When it's run as a local server, the session manager is fork()ed directly out of the client using ordinary IPC. So, you can run a distributed application as a completely local application. This means you can use whatever local debugger you would use ordinarily in concert with the C-Lite debugger to get everything working right, all on your own workstation.
Installing the Distributed Application
Assuming you debugged locally and you're not already running the RPI daemon on the server host, installing the debugged version as a distributed application requires three things:
- Change the first digit of the TYPE component back to 1 in the client
C-Lite.
- ftp your server executable and the RPI and SMS executables to the same directory on the server host.
- rlogin to the server host, cd to that directory, make sure that the files have execute permissions, and then start RPI like this: rpi_srv -a >&/dev/null &
Conclusion: What You Risked vs. What You Gained
If you already had the server code, you've still got it and, at worst, you had to write a C wrapper that turned it into a standard UNIX filter--a valuable thing to have done, anyway. If you had to write the server code, you wrote it as a standard UNIX filter and it's still a perfectly good program that you can run from the shell. The 30 lines of C-Lite in clientCB() are probably the only throw-away cost you've got. The bottom line is that you've spent a few days learning an already familiar language and maybe a week building a distributed application. If you like it, that's a huge return on investment. If you don't, most of what you wrote is still usable code. In either case, you have actually done some distributed computing instead of just thinking or talking about it. And you'll see both the advantages and pitfalls of distributed computing in a very different light once you've gotten your hands dirty.
Figure 1: Wide Area Network binding via the Name Service Interface, CDS.
Figure 2: Using asynchronous entry points with a session semantics.
Example 1: (a) Sample .proplist file; (b) RPI-generated additions to the .proplist file.
(a)
TABLE {'student' 'enroll' 'professors' 'course' 'dept'}
STUDENT {'stu_id' 'lastname' 'major' 'stu_desc'}
SERVERS {'sql_app' 'filexfer' 'syb2eos' 'eos2syb'}
MIPS 144
DISK 1000000000
AUTH {'Pete' 'Lowell' 'Geoff' 'Andrew' 'Kyle' 'Clark'}
(b)
HOST 'ferrari'
hostname 'ferrari.lri.com'
binding '187.95.103.78[1234]'
SECPAC '211'
Example 2: Client initialization.
foo->PROPV[ 0 ] = "(student in TABLES) or (enroll in TABLES)"; foo->PROPV[ 1 ] = "sql_app in sblist"; foo->PROPV[ 2 ] = "MIPS > 50"; foo->PROPC = 3;
Example 3: Command-line prompt.
% sbroker broker=> set stock IBM stock set to IBM broker=> set shares +20 order set to buy 20 broker=> execute placing order to buy 20 shares IBM broker=> quit
[LISTING ONE] (Text begins on page 64.)
typedef struct {
binding, /* the binding property */
princ_name, /* the rpi's principal name */
secpac, /* the rpi's security package */
ns_auth, /* the rpi's privileges in the NSI */
struct {
name, /* property name */
argc, /* number of values */
argv[16] /* array of values */
} prop[24], /* property list */
propc, /* number of properties */
} INQUIRE;
auto INQUIRE inq[ 24 ];
auto cexp = "(('student' in TABLES) or ('enroll' in TABLES ))", i = 0;
*inq[ 0 ] = servers(
"", /* subnet - "" means local subnet */
"/.:/", /* context - in this case every host */
cexp, /* constraint expression */
000, /* secpac - in this case none */
"" /* CDS group -- "" means default
which is /.:/eos */
);
while( inq[ i ]->binding )
*inq[ ++i ] = servers();
</PRE>
<P>
<h4><a name="01f7_0010"><a name="01f7_0011"><B>[LISTING TWO]</B><a name="01f7_0011"></h4>
<P>
<pre>
/* Implements a C wrapper around an executable we have no source for that turns it into a server. */
#include <stdio.h>
#define MAXBUF 65536
char stdinb[ MAXBUF ];
int prlen = 10;
char prompt[] = "\nbroker=> ";
char buf[ 3 ][] = { "set stock ", "set shares ", "execute\n" };
char *arg[ 3 ] = { NULL, NULL, NULL };
char null[] = "";
main( argc, argv )
int argc;
char *argv[];
{
char *c, *d, tmp[ 255 ];
int i, pid, j;
FILE *fp[ 2 ];
int fdin[ 2 ], fdout[ 2 ];
/* create output pipe to server body */
if ( pipe( fdout ) == -1 ) exit( 1 );
if (!( fp[ 1 ] = fdopen( fdout[ 1 ], "w" ))) exit( 2 );
/* create input pipe from server body */
if ( pipe( fdin ) == -1 ) exit( 3 );
if (!( fp[ 0 ] = fdopen( fdin[ 0 ], "r" ))) exit ( 4 );
/* name of server body to exec */
strcpy( tmp, "/usr/bin/sbroker" );
argv[0] = tmp;
/* start the a new process, which runs on its own. */
if ((pid = fork()) == -1) exit( 5 );
/* if in the server body */
if( pid == 0 ) {
/* dup our output pipe to servers stdin */
close( 0 );
dup( fdout[ 0 ] );
/* dup our input pipe to servers stdout */
close( 1 );
dup( fdin[ 1 ] );
/* execute sbroker program. */
execlp( tmp, tmp, NULL );
exit( -1 );
}
/* swallow the first prompt */
for( i = 0; i < prlen; i++ )
getc( fp[ 0 ] );
/* forever or until we're told to quit */
while( 1 ) {
/* wait for sms to call us */
if ( !fgets( stdinb, MAXBUF, stdin ))
break;
/* if caller wants us to quit */
if ( !strcmp( stdinb, "quit\n" ) ) {
fputs( stdinb, fp[ 1 ] );
fflush( fp[ 0 ] );
break;
}
/* parse arguments: should be stock, shares */
arg[ 0 ] = ( char * )strtok( stdinb, "," );
arg[ 1 ] = ( char * ) strtok( NULL, "," );
arg[ 2 ] = null;
/* begin the "trick" code which turns our single input */
/* line into three outputs to the server after each of */
/* which we discard the prompt */
for ( i = 0; i < 3; i++ ) {
fprintf( stdout, "%s %s\n", buf[ i ], arg[ i ] );
fflush( stdout );
d = c;
for( j = 0; j < prlen; j++ ) {
*c = getc( fp[ 0 ] );
if ( *c != prompt[ j ] ) {
c++;
break;
}
if ( *c == '\n' ) *c = '\0';
else
c++;
}
*c = '\0';
if ( !strcmp( d, ( char *) prompt + 1 ))
continue;
fgets( stdinb, MAXBUF, fp[ 0 ] );
}
/* this server is implemented as a void function */
fputs( "\n", stdout );
fflush( stdout );
}
exit( 0 );
}
</PRE>
<P>
<h4><a name="01f7_0012"><a name="01f7_0013"><B>[LISTING THREE]</B><a name="01f7_0013"></h4>
<P>
<pre>
function "clientCB"
{
auto dir = "/users/me/stock", table = "portfolio", shares, row;
stable current_portfolio_value;
static SERVER model;
static SERVER broker;
/* import the model server */
model->TYPE = 112000;
model->CONTEXT = "/.:/hosts/ferrari/ /.:/";
model->PROPC = 1;
model->PROPV[ 0 ] = "'finsim' in sblist";
*model = rpi( *model );
(model->INITIALIZE)( "model", "finsim", "quit" );
/* import the broker server */
broker->TYPE = 112000;
broker->CONTEXT = "/.:/";
broker->PROPC = 1;
broker->PROPV[ 0 ] = "'sbroker' in sblist";
*broker = rpi( *broker );
(model->INITIALIZE)( "trade", "sbroker", "quit" );
/* open the portfolio table */
chdir( dir );
wattach( table );
for ( row = 1 ;( row ); row = nextrow() ) {
shares = model( current_portfolio_value, table.* );
if ( !shares ) continue;
trade( sprintf( "%s,%s", table.stock, shares ));
}
(model->DISCARD)();
(broker->DISCARD)();
wdetach();
return();
}
End Listings
Copyright © 1993, Dr. Dobb's Journal
