Andy Nicholson

The high-performance parallel interface (HIPPI) is the first standard computer network architecture to use crosspoint switches as the basis for a local area network. Currently, HIPPI is focused on point-to-point, high-throughput data connections useful for applications such as real-time graphics. A HIPPI LAN not only offers very high throughput, but demonstrates production-quality technology for building bandwidth-scalable networks using components available today. Making the best use of this capability, however, requires that we rethink some of our fundamental assumptions about the architecture of computer networks and network applications. Ultimately, we must modify or replace the client/server model with peer-to-peer oriented applications to take advantage of the increased bandwidth offered by HIPPI and other crosspoint-switch based networks.

HIPPI, which has its roots in super-computer I/O interfaces, defines a standard for 100-Mbyte/sec and 200-Mbyte/sec point-to-point simplex links using 32-bit wide or 64-bit wide copper cabling. (An implementor's agreement also exists for building fiber-optic extenders that use two or four fibers.) A related standard defines the behavior of a cross-point switch used to support multiple interconnects between HIPPI interfaces on different hosts. These crosspoint switches are central to building a LAN with HIPPI technology. There's also a proposed Internet standard for implementing an IP network on HIPPI switches. These standards and commercially available components make it possible to build a production HIPPI LAN. At Cray Research, we used these standard components to build a sophisticated HIPPI LAN for production and development work.

A crosspoint switch (also known as an NxN or crossbar switch) connects hosts into a LAN such that each host may have a dedicated connection to at least one other host in the network. Each connection may transfer data at the full throughput capability of the connection media. A 32x32 crosspoint switch (Network Systems Corporation's PS32, the largest currently available) allows each of 32 hosts to maintain a full 200-Mbyte/sec throughput data connection. That's 6.4 Gbytes/sec (or 51.2 Gbits/sec) of total network bandwidth!

Two properties of crosspoint-switch based networks are very exciting compared to bus- or ring-based networks such as Ethernet or FDDI. First, a crosspoint-switch network has constant throughput per host. Second, a crosspoint-switch network has increasing bandwidth that scales to the number of hosts. Regardless of the number of hosts attached to the network, a crosspoint-switch network delivers constant throughput capability to each host. Because of this, the network bandwidth (the throughput times the number of hosts) scales up as the network grows in size.

This scalability is the most valuable feature of a crosspoint-switch based network. In a HIPPI network, it makes little sense to build a 200-Mbyte/sec data interface if contention over a shared media (like a bus or ring) lowers the average throughput to a fraction of what is possible. The interface is simply too expensive to waste. For example, with the 32x32 HIPPI crosspoint switch, all hosts can simultaneously transfer data at 200 Mbytes/sec. Thirty-two hosts attached to a 100-Mbit/sec FDDI ring network will achieve (at most) an average of 3 Mbits/sec throughput per host. Crosspoint-switch networks, therefore, are the most efficient users of the network interface.

The fundamental difference between scalable crosspoint-switch networks and traditional bus or ring networks is that, relative to the number of hosts, a cross point-switch network is throughput limited and bandwidth scalable, and a bus or ring is throughput decreasing and bandwidth limited. In short, the network itself is a bottleneck in bus and ring networks.

Before HIPPI technology, there was no standard networking hardware available that allowed us to build a bandwidth-scalable LAN. The availability of a proposed Internet standard for using the IP protocol over a HIPPI crosspoint-switch based LAN puts this technology into use in a production environment. Still, the current network computing model is not sufficiently sophisticated to take advantage of the greater capabilities of a band-width-scalable LAN.

Software designers used a model of a network as a shared resource when creating network applications. Client/server distributed computing is well suited to this network model because contention for the shared network resource hides contention for the shared server resource. Only one server is needed because only one host may access the network at any time. Building a LAN with HIPPI technology, however, exposes the limitations of this approach.

Bandwidth-limited network. The client/server architecture encourages balance on the bandwidth-limited network. The server must be capable of servicing requests only according to the network's capacity to carry requests and responses. Clients are limited to their fair share of the bandwidth available in the network. Additional capacity in the server is wasted, and additional capacity in the clients only increases contention over the network. Adding more clients also increases network contention.

Network contention encourages some interesting solutions to increasing aggregate system performance. One solution is to optimize use of the network. The server should respond to requests with minimum latencies, and clients should make a minimal number of requests. This has been accomplished in some networks by adding expensive local disks to previously diskless workstations. Of course, much of the same data is replicated (such as commonly used programs) across all of the local disks.

Another solution is to partition networks. Instead of adding capacity to each server, networks are broken up into multiple networks with a smaller number of clients and individual, dedicated servers. A variation on this strategy is to add multiple network interfaces to higher-capacity servers to support multiple-client networks on each server. These strategies increase the load on the servers because they become routers between client networks. Partitioning is more expensive in terms of server hardware, network hardware, routers, and network-management effort.

Crosspoint-switch network. A crosspoint-switch network has very different properties. Foremost, the network media is not bandwidth limited and is not a shared resource. The point of contention becomes the network interface at each host. This is a liability in a client/server architecture. The server can respond to requests only according to the capability of the network interface. However, the network is capable of carrying requests according to the number of clients on the network.

Not surprisingly, another solution to overloaded networks has appeared. Localtalk, Ethernet, and FDDI switches are available from Tribe Computer Works (Emeryville, California), Kalpana (Santa Clara, California), and Digital Equipment Corp., respectively. These products are designed and marketed specifically to offer scalability over typical partitioning approaches for increasing aggregate network bandwidth. However, their potential is limited by clients creating significant contention over single servers. The Localtalk, Ethernet, or FDDI switch must distribute network load to several servers to increase overall network bandwidth utilization.

Building a balanced client/server network using a crosspoint switch requires that a server service requests according to the capacity of clients to generate those requests. This requires a high-performance server, and it may require multiple network interfaces in order to bypass the bottleneck of contention for its network interface.

This is an advantageous approach to building a balanced system because it scales more easily than a bandwidth limited network. Adding new clients to the network requires only additional capacity at the server and not a wholesale replacement or repartitioning of the network. However, we can go further in balancing system resources in a bandwidth-scalable network.

Because a bandwidth-scalable network offers the same throughput to each network interface no matter how many interfaces are active in the network, the client/server model of distributed computing appears less useful. A peer-oriented model makes much more sense. As a first step toward a peer-to-peer model, the server functions should be split out to separate servers with a capacity well balanced to the function.

A typical server may provide remote disk, e-mail gateway, printer, dial-up, and wide area network (WAN) connectivity services in some combination. All of these services will typically be using the same network interface, yet their functions are not compatible with optimal use of the interface. For example, dial-up services will generate many small packets, printer and e-mail will be larger and bursty, disk will require a lot of large blocks, and WAN connectivity will generate all sorts of traffic.

A crosspoint-switch network allows each of these services to live in a separate server. Another possibility allows each service to reside behind a different interface in one high-capacity server. The system designer has great flexibility in designing a system that is balanced to make optimal use of system resources. (See the text box entitled, "Peer Programming: It's a Matter of Symmetry" for a discussion of peer-to-peer vs. client/server programming.)

Computer-supported collaboration (CSC) is a perfect example of an intrinsically peer-to-peer application that takes advantage of the enhanced capabilities of a bandwidth-scalable crosspoint-switch based LAN. The high-throughput requirements of real-time audio and video are multiplied by the participation of many people in a collaborative effort. The usefulness of a scalable bandwidth is obvious when you consider the possibility of people entering or leaving a collaborative session at random intervals. The network can adapt to the changing needs of the application.

HIPPI technology is not the only way to build bandwidth-scalable networks using available products. Nevertheless, HIPPI is the first network media in which crosspoint-switch technology is the only method of building a LAN, as well as providing a standard method for use of the crosspoint switch for the LAN.

This is an important distinction. A Localtalk, Ethernet, or FDDI switch exists to increase bandwidth in a bandwidth-limited network. The same networking model is still in use and there is little incentive to directly connect hosts to ports on the switch. The Ethernet or FDDI switch breathes new life into a client/server model on bandwidth-limited networks.

HIPPI technology offers the opportunity to re-examine the way we build LANs. We can build networks that have high throughput per host with scalable bandwidth and no network performance degradation due to the number of hosts. The client/server model as currently used must be modified or replaced with a peer-to-peer model to best use this capability in a network. We must challenge our fundamental assumptions about the organization of function and network topology. HIPPI technology creates our first opportunity to build fast networks with capabilities unavailable in the past. We can take full advantage of this opportunity only by recasting our thinking process and imagining new creative solutions to network computing problems unencumbered by the limitations of bus- and ring-oriented networks.

I want to thank John Renwick for getting me involved with HIPPI.

Adams, George B. III, Dharma P. Agrawal, and Howard Jay Siegel. "A Survey and Comparison of Fault-Tolerant Multistage Interconnection Networks." IEEE Computer (June 1987).

Bhuyan, Laxmi N., Qing Yang, and Dharma P. Agrawal. "Performance of Multiprocessor Interconnection Networks." IEEE Computer (February 1989).

High-Performance Parallel Interface--Mechanical, Electrical, and Signaling Protocol Specification (HIPPI-PH), ANSI X3.183-1991.

High-Performance Parallel Interface--Framing Protocol (HIPPI-FP), ANSI X3.210-199X.

High-Performance Parallel Interface--Encapsulation of IEEE 802.2 (IEEE Std 802.2) Logical Link Control Protocol Data Units (802.2 Link Encapsulation) (HIPPI-LE), ANSI X3.218-199X.

High-Performance Parallel Interface--Physical Switch Control (HIPPI-SC), ANSI X3.222-199X.

Kumar, V.P. and S.M. Reddy. "Augmented Shuffle-Exchange Multistage Interconnection Networks." IEEE Computer (June 1987).

Renwick, John, and Andy Nicholson. "IP and ARP on HIPPI." Proposed Internet Standard, Internet Working Group Request for Comments 1374, October 1992.

The primary difference between client/server and peer-to-peer network programming is the symmetry between the communicating parties. Client/server programming is asymmetrical and peer-to-peer is symmetrical. Peer programming means writing only one version of a program that does everything the network application requires.

When writing client/server programs, you first think in terms of the service the server will provide, and how that service will be made available. Then you write the server to implement the design. Then you think through how the client will access the service. Finally, you implement the client program.

Peer programming is similar, except that you put both kinds of functionality into the same program. It might seem more complex at first, because, when thinking it through, you realize that meaningful communication is taking place in two directions when peers are cooperating on the network. This is the same as running a client/server program in both directions on the same network. In this aspect, peer programming is simpler because you can ignore the differences between client and server programs.

<a name="016c_000f">

/* echo.h - header file for the echo example programs */

#define BUFSIZE         512
#define ECHOPORT        7500

#define ECHO_REQUEST    1
#define ECHO_REPLY      2





<a name="016c_0010">
<a name="016c_0011">

<a name="016c_0011">

/* echos.c  --  The echo service server program  */

#include <errno.h>

#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include "echo.h"

main()
{
        char buf[BUFSIZE];
        int s, c;
        struct sockaddr_in sin, peer;
        int peerlen;
        int mlen;

        /* get a socket */
        if ((s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1) {
                perror("can't open socket");
                exit(1);
        }
        /* bind to local address and echo port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        sin.sin_port = ECHOPORT;
        if (bind(s, &sin, sizeof(sin)) == -1) {
                perror("cannot bind");
                close(s);
                exit(1);
        }
        /* start listening for connections */
        if (listen(s, 5) == -1) {
                perror("cannot listen");
                close(s);
                exit(1);
        }
        /* accept and process incoming messages */
        while (1) {
                if ((c = accept(s, &peer, &peerlen)) == -1) {
                        perror("cannot accept");
                        close(s);
                        exit(1);
                }
                /* simple processing, read message, echo it back */
                if ((mlen = read(c, buf, BUFSIZE)) == -1) {
                        perror("read error");
                        close(c);
                        continue;
                }
                if (write(c, buf, mlen) != mlen) {
                        perror("write error");
                        close(c);

                        continue;
                }
                printf("%d byte message received and echoed\n", mlen);
                close(c);
        }
}  /* end of main */




<a name="016c_0012">
<a name="016c_0013">

<a name="016c_0013">

/* echoc.c -- The echo service client program  */

#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#include "echo.h"

main(argc, argv)
int argc;
char **argv;
{
        char buf[BUFSIZE];
        int s;
        struct sockaddr_in sin;

        if (argc < 2) {
                printf("must specify a server in internet dot notation\n");
                exit(1);
        }
        /* get a socket */
        if ((s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1) {
                perror("can't open socket");
                exit(1);
        }

        /* bind to local address and available port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        if (bind(s, &sin, sizeof(sin)) == -1) {
                perror("cannot bind");
                close(s);
                exit(1);
        }
        /* connect to echo server at echo port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        sin.sin_port = ECHOPORT;
        sin.sin_addr.s_addr = inet_addr(argv[1]);
        if (connect(s, &sin, sizeof(sin)) == -1) {
                perror("cannot connect");
                close(s);
                exit(1);
        }
        /* send a message to be echoed */
        if (write(s, buf, BUFSIZE) != BUFSIZE) {
                perror("write error");
                exit(1);
        }
        /* read the echoed message back */
        if (read(s, buf, BUFSIZE) != BUFSIZE) {
                perror("read error");
                exit(1);
        }
        printf("%d byte message echoed by echo server\n", BUFSIZE);
        close(s);
        exit(0);
}  /* end of main */




<a name="016c_0014">
<a name="016c_0015">

<a name="016c_0015">

/* echop.c -- The echo service peer program  */

#include <errno.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <sys/time.h>

#include "echo.h"

main(argc, argv)
int argc;
char **argv;
{
        int s;
        struct sockaddr_in sin;
        int status;

        int width;
        fd_set readfds, writefds, exceptfds;
        struct timeval timeout;
        if (argc < 2) {
                printf("must specify a peer in internet dot notation\n");
                exit(1);
        }
        /* setup passive socket */
        if ((s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1) {
                perror("can't open passive socket");
                exit(1);
        }
        /* bind to local address and echo port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        sin.sin_port = ECHOPORT;
        if (bind(s, &sin, sizeof(sin)) == -1) {
                perror("cannot bind");
                close(s);
                exit(1);
        }
        /* start listening for connections */
        if (listen(s, 5) == -1) {
                perror("cannot listen");
                close(s);
                exit(1);
        }
        /* initial select parameters */

        FD_ZERO(&writefds);
        FD_ZERO(&exceptfds);
        width = s+1;  /* only file descriptor we want to check */
        timeout.tv_sec = 0;  /* set timeval for polling */
        timeout.tv_usec = 0;

        while (1) {
                /* check for and process incoming connection requests */
                FD_SET(s, &readfds);
                while ((status = select(width, &readfds, &writefds,
                                     &exceptfds, &timeout)) > 0) {
                        do_respond(s);
                        FD_SET(s, &readfds);
                }
                if (status == -1) {
                        perror("select failed");
                }
                do_request(argv[1]);
        }
}  /* end of main */
int
do_respond(s)
int s;
{
        int c;
        struct sockaddr_in peer;
        int peerlen = sizeof(peer);

        if ((c = accept(s, &peer, &peerlen)) == -1) {
                perror("accept failed");
                return(-1);
        }
        do_echo(c);
        close(c);
        return(0);

}  /* end of do_respond */

int
do_request(dst)
char *dst;
{
        int s;
        struct sockaddr_in sin;
static  pid_t pid = 0;
        int status;
        /* Wait for child if we have already forked. */
        if (pid) {
                wait(&status);
                pid = 0;
        }
        /* get a socket */
        if ((s = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) == -1) {
                perror("can't open socket");
                return(-1);
        }
        /* bind to local address and available port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        if (bind(s, &sin, sizeof(sin)) == -1) {
                perror("cannot bind");
                return(-1);
        }
        /* connect to echo server at echo port */
        bzero(&sin, sizeof(sin));
        sin.sin_family = AF_INET;
        sin.sin_port = ECHOPORT;
        sin.sin_addr.s_addr = inet_addr(dst);
        if (connect(s, &sin, sizeof(sin)) == -1) {
                perror("connection failed");
                close(s);
                return(-1);
        }
        /* fork process to be form active request */
        if ((pid = fork()) == -1) {
                perror("Cannot fork");
                pid = 0;
                return(-1);
        }
        /* return if parent so we can go back and look for incoming
         * service requests. */
        if (pid > 0) {
                close(s);
                return(0);
        }
        /* child uses echo service */
        do_echo(s);

        close(s);
        exit(0);
}  /* end of do_request */

int
do_echo(s)
int s;
{
        char buf[BUFSIZE];
        int recvlen;
        int sendlen = BUFSIZE;
        /* send a message to be echoed */
        buf[0] = ECHO_REQUEST;
        if (write(s, buf, sendlen) != sendlen) {
                perror("sending: write error");
                exit(1);
        }
        printf("sent echo request\n");
        /* read an incoming message */
        if ((recvlen = read(s, buf, BUFSIZE)) == -1) {
                perror("reading message: read error");
                exit(1);
        }
        /* echo message if it is an echo request */
        if (buf[0] == ECHO_REQUEST) {
                buf[0] = ECHO_REPLY;
                if (write(s, buf, recvlen) != recvlen) {
                        perror("echoing: write error");
                        exit(1);
                }
                printf("received echo request - sent echo reply\n");
        } else {  /* must be reply */
                printf("received echo reply\n");
        }
        /* read another incoming message */
        if ((recvlen = read(s, buf, BUFSIZE)) == -1) {
                perror("reading message: read error");
                exit(1);
        }
        /* echo message if it is an echo request */
        if (buf[0] == ECHO_REQUEST) {
                buf[0] = ECHO_REPLY;
                if (write(s, buf, recvlen) != recvlen) {
                        perror("echoing: write error");
                        exit(1);
                }
                printf("received echo request - sent echo reply\n");
        } else {  /* must be reply */
                printf("received echo reply\n");
        }

        return(0);

}  /* end of do_echo */

Copyright © 1993, Dr. Dobb's Journal