Measuring Network Software Performance

While the measurement techniques I describe are useful (and in many cases essential), they are by no means designed to be comprehensive for all types of network software. Rather, they focus on the minimum set of performance criteria basic to almost all types of networking functions. Developers of certain network software may need to consider variations of these measurement techniques to more fully test their software. For example, developers of network device drivers or TCP/IP stacks may be satisfied with running just the tests I present here, while developers of middleware may need to run additional variations of the tests to evaluate things such as the overhead of guaranteeing certain types of quality of service.

Performance Measurements

The basic network performance criteria are latency and throughput. A key assumption in these tests is that the software has previously been successfully unit tested for basic functional integrity, such as being able to send/receive messages without losing or corrupting contents. Network performance tests are really higher level functional tests that should be carried out only after the initial code-level functional tests and debugs have been successfully completed. Because the test drivers used during the initial software development process can be easily modified to obtain latency and throughput measurements, the additional effort that you need to expend to carry out these tests should be relatively minimal. Because of the functional requirements and expectations that are inherent to network programs, the tests that are referred to here as "network performance tests/measurements" are actually as much a part of the network software development process as the initial code-level test and debug. In other words, the primary goal of these performance tests, as in any type of software test, is to determine whether there are problems with the network software, such as sudden failures or not meeting requirements. (I use the terms "measurement" and "test" interchangeably here.)

While there are a number of public domain, open-source network benchmarking programs, I'll use Netperf (http://www.netperf.org/) to illustrate how latency and throughput can be measured. In addition, you can use Netperf as a tool for evaluating a communication interface or network through a number of different protocols, including BSD-style TCP/IP. This tool is often used by network engineers to evaluate network configurations. (For more information on Netperf, see the accompanying text box entitled "Netperf.")

Latency

Latency is the time it takes for a message to travel from one task to another. This is also known as "one-way latency." The tasks, which may be threads or processes, may be on the same host or different hosts. For network software under test, latency is (in large part) determined by code efficiency. However, functional design can also play a major role. Consider, for example, two middleware designs where one only uses TCP/IP and the other intelligently chooses between TCP/IP and message queue. Assume that code efficiencies are equal. If the sender and receiver tasks are collocated in the same host, the middleware that uses only TCP/IP may have greater latency than the one that intelligently chooses message queue since TCP/IP is a roundabout way for the message to travel (that is, down and up the stack) while message queue takes a more direct path.

In theory, measuring latency is simple, and can be accomplished as in Figure 1(a). In practice, however, measuring latency is not as simple. One reason is that, when senders and receivers are located in different hosts, the time source for senders will usually be different from that of receivers, and in such cases, the time sources must be synchronized for accurate latency measurements. For many systems, maintaining constant, accurate time synchronization among different hosts is difficult, so latency in many cases is measured as in Figure 1(b).

In Figure 1(b) the two time stamps are recorded using the same time source, namely that of the sender, thus solving the time synchronization problem. However, the time difference computed in step 7 is actually two-way (or roundtrip) latency, because the message travels in one direction and then back in the other. Therefore, one-way latency is half of the time difference. For an accurate one-way latency measurement, the sender/receiver interaction must be symmetrical. In other words, when the receiver sends the message back to the sender, it must do so in the same manner with which the sender originally sent the message. Furthermore, if the test is between two different hosts, both machines should have identical configurations. Otherwise, if your resources are limited and do not allow identical configurations, then the one-way latency measured with Figure 1(b) will probably be the best estimate you can obtain. On the other hand, if your objective is to measure round-trip latency, then obviously symmetry would not necessarily be an issue.

There are two problems with Figure 1(b). First, for a very small message, the latency may actually be smaller than the time source resolution, making it immeasurable. For instance, the time clock may tick in 17-millisecond intervals, in which case the shortest duration it can measure would be 17 milliseconds, but the latency may be even smaller, perhaps 500 microseconds. Second, for latencies that are measurable, they will almost certainly contain so-called measurement errors. These are errors due to uncertainties, or random fluctuations, that are inherent in such measurements.

Figure 1(c) shows how the two problems can be solved essentially in one step. The latency is now a bulk average because it is measured over a number of roundtrips. (Actually, the one-way latency measured with Figure 1(b) is also an average, but over a smaller duration.) Num, the message count, should be chosen to be large enough (>1000) to not just account for the measurement errors, but also so that the elapsed time Diff will be long enough (>10 seconds) to compensate for the limited resolution of the time source. For a 60-Hz time source, as long as Num is at least 1000, Diff of 10 seconds or more should be sufficient.

For improved statistical results, the measurement should be repeated at least twice, for a total of at least three measurements, to determine variability among the measurements. Wide variabilities indicate that the measurements are not reliable and that either the test setup or the network configuration should be carefully reexamined for the cause of instability. Assuming that the test duration (Num or Diff) is sufficiently long, the source of instability can be as simple as an unauthorized user pinging in the middle of a test or as serious as poorly designed network software. If the variability is not excessive, the multiple measurements can be averaged to obtain the final latency value. A more precise measure of variability is standard deviation, where greater standard deviation corresponds to wider variability, but for most routine tests, simply eyeballing variability should be adequate. On the other hand, if you are required to formally document or present test data, you should compute the standard deviation.

The statistical treatments discussed so far may not be appropriate for all types of networks. One example is where a network is designed or claimed to have bounded latencies or real-time characteristics. For such a network, average latency measurements may have little use, and high-resolution time sources may, instead, have to be used to measure individual latencies for more useful performance assessment.

A variation on Figure 1(c) is to base the measurement on time duration rather than on message count. Listing One shows how Netperf implements both options on the Sender side when measuring TCP latency. (Netperf refers to the latency test as the "request/response" test. For simplicity, the source code listings I present here show only the relevant comments and code fragments.) In Listing One, trans_remaining is equivalent to Num in Figure 1(c) and test_time is the time duration. start_timer() sets up the timer and its signal handler. When the timer expires, the signal handler sets times_up. The timer signal also causes send() or recv() to terminate with the appropriate errno value. Thus, for a time-based execution, the test can exit out of the loop either because times_up was detected as having been set, or because send() or recv() terminated with the appropriate errno value. For an execution based on message count, the test will exit out of the loop only if trans_remaining decrements to zero. thruput is actually the number of roundtrips per second and Netperf reports this as the number of transactions per second. You will have to take the reciprocal of this result and then divide by 2 to obtain the average one-way latency; for example, average one-way latency = 1/(2*thruput). If requested, Netperf uses confidence interval to determine whether variability among repeated measurements is too large.

The latency measurement procedure as given thus far works fine when message delivery is guaranteed. However, there is a problem for an unreliable protocol, such as UDP. In such a case, if the latency measurement is based on message count, Num in Figure 1(c), and not on time duration, the sender and receiver tasks deadlock when a sent message gets dropped, because both sender and receiver will block on receiving messages. On the other hand, when a timer is used, the deadlock can at least be limited to the remainder of the timer duration, but it may still result in misleadingly high latency values. Though there is no definite method for determining whether a deadlock has occurred, one good way is to examine variability among repeated timer-based measurements. With deadlocks, the variability will most likely be large since message dropouts tend to occur at random points in time. Even so, with a properly functioning network system that is unloaded and quiescent (for example, no additional CPU load and free of data traffic except for those generated by latency tests), message dropouts should be rare. If you do suspect a message was dropped during a test, simply run the test several more times and examine the variability among the latency values. A wide variability indicates that messages are probably being dropped and that there could be an anomaly, perhaps either in the test setup or in the network software.

Throughput

Throughput is defined as the amount of messages transmitted per unit time. It is often expressed as bits per second or messages per second. The type of throughput that will be discussed here will be the peak throughput, that is, the maximum amount of messages that a communication pipeline is capable of successfully transmitting per unit time. The reference to successful transmissions is necessary because only completed message deliveries are counted in measuring throughput. In this article, the terms "throughput" and "peak throughput" are used synonymously.

A common misconception is that throughput can be computed directly from the reciprocal of latency. Throughput and latency are, in fact, conceptually different. Throughput refers to the information-carrying capacity and, for data networks, it is a function of spacing between data packets or messages. Latency refers to the information transit time and it is a function of the temporal length of the communication pipeline. Changing the length of the communication pipeline would affect the latency, but not necessarily the throughput. On the other hand, changing the spacing between messages would affect the throughput, but not necessarily the latency.

Given a stream of messages, a goal of the throughput test is to then measure how tightly the system (middleware, operating system, device drivers, and so on) can pack the messages during transmissions. In other words, the throughput is a function of the turnaround time between successive transmissions. The turnaround time is the time it takes for the system to return to the state of being able to send a new message after a message is sent and it is determined by events, such as buffer deallocation and other cleanup activities, that may occur after a message transmission. Such events would not necessarily affect the latency since the message would have already been on its way. For connection-based protocols such as TCP, it is sometimes argued that the throughput can be estimated from latency because, as the reasoning goes, the turnaround time essentially consists of the delay in receiving the Ack (acknowledgment) from the receiver. While this may be true in theory, it ignores the fact that the primary objective of the performance tests, as stated previously, is to determine whether there are problems with the software. Unfortunately, heavy communication loads, as seen in throughput tests, can cause the network software to degrade in unanticipated ways, exposing bugs and inefficiencies that would have otherwise gone unnoticed until after deployment. In rare instances, operating system crashes have been known to occur during TCP/IP throughput tests.

There are a number of ways of measuring throughput, some more preferable than others. The sequence of events in Figure 2(a), which is actually an interleaving of activities in two separate tasks, Sender and Receiver, shows one method for measuring throughput. The method incorporates several recommended features. First, making Num_Msg a user-specified input parameter that is communicated between the two tasks rather than being hardcoded in both tasks, significantly increases the ease and flexibility of performing the tests. Having the Sender subsequently wait for the Ready status from the Receiver ensures that the Receiver will be ready to receive messages when the test actually begins. Finally, having the Sender wait for the Received_All_Messages status before recording the second time stamp increases the accuracy of the measurement since this would account for any flushouts of data queues. The Sender records both time stamps for the same reason as that given for latency measurement, that is, the difficulties of time synchronization between hosts.

Listings Two and Three demonstrate how Netperf respectively implements the Sender and Receiver sides of the TCP throughput test. (The throughput test in Netperf is actually called the "stream" test. Also, the Netperf source code variously refers to the Receiver side as remote, remote end, or server side.) In overall code layout, they are similar to the latency test. One obvious difference is that, in each side, the loop contains either send() or recv(), but not both. The loops are relatively tight. On many CPUs, the loop in the Sender is in fact tight enough to easily overwhelm a 100 Mbits/sec. network interface. Thus, you can use it to test not just the performance of a network software, but also how robust it is when placed under significant stress.

The method in Figure 2(a) works well when the communication is reliable, but not necessarily when it is unreliable. With an unreliable communication, when a message sent within the loop gets dropped, the test will deadlock because the Receiver will then block on receive inside the loop and the Sender will block on receiving Received_All_Messages. (The sending of the parameter and status messages should not be a concern since they can be sent on a separate channel using reliable communication.) One way to overcome this is to have the Receiver task, rather than the Sender because only successful deliveries are being counted, count the number of messages received in addition to recording the time stamps. This is illustrated in Figure 2(b) where, for simplicity, the Sender task is assumed to be continuously sending messages in an infinite loop (whether this can actually be done depends on the communication scheme being used, so your mileage may vary). Variations on this basic method can be used based on the characteristics of the communication scheme being tested. Clearly, measuring throughput when the communication is unreliable will be more challenging and less accurate than when it is reliable, but you should remember that the goal is to try to obtain the best possible estimate.

Netperf's implementation of the Sender and Receiver sides of the UDP throughput test are available electronically; see "Resource Center," page 7). While send() and recv() inside the loops use UDP, send_request(), recv_response(), and send_response() use TCP since, as stated previously, the parameter and status messages need to be delivered reliably. Both Sender and Receiver use the same value for test_time. (The Receiver's time duration is lengthened by PAD_TIME to prevent errors in the Sender that are caused by the Receiver timer expiring too soon.) The throughput, in this case, is computed using the elapsed time measured at the Sender and the number of messages received by the Receiver.

In measuring throughput, an important consideration is how tight the sending loop in the Sender task should be. There may be concern among a few developers of network software that a tight send loop will result in an onslaught of messages that may not be representative of what really happens in real-world worst-case situations. Though the concern may have some validity, the problem is that it is virtually impossible to predict what the actual worst-case situation will be. On the other hand, the throughput measurement, with a tight send loop, is a means to assess how well a network software performs when stressed very heavily, especially with nonblocking send functions. Users should at least be wary of network software that crashes during such a test since it may indicate that the software may not be robust. Well-designed network software should not crash even under a very heavy load. In any case, no matter if the send loop is tight or paced at a certain rate, you should fully disclose how the throughput performance was measured.

Test Program Usability

Even if there is no immediate plan to use the programs for comprehensive and exhaustive testing, serious thought should be given to their usability because they can then greatly ease the burden of future regression tests.

A good place to begin is to borrow ideas from Netperf. For example, the Receiver task, as presented in the listings, can be a server that waits for a message from Sender that a test is about to begin and the message can contain the parameters for that particular test. When the test completes, the Receiver can send its measurement data to the Sender and then return to the original state to await the parameter message for the next test. Thus, only the Sender needs to print the test results, facilitating the collection of test data for the user. It also allows the programmer to start the Receiver task only once and then focus on repeatedly running only one task, the Sender, for new test cases (unless, of course, a test causes Receiver to crash). This leads to another good practice, which is that the test parameters should not be hardcoded into the test programs. Instead, the user should be allowed to specify them, either on the command line, as in Netperf, or in a parameter file that can be read by the Sender task. Not hardcoding parameters should be obvious, but programmers are often tempted to take the shortcut and not invest the additional time to follow good usability guidelines. Careful thought should also be given to the format of the output results, especially when planning for many test runs. Intelligently formatting output results will allow you to, for example, easily organize them into columns that can be efficiently read, stored, and printed in a compact form. In addition, with such a format, you can easily import the data into a data analysis and charting program such as a spreadsheet.

Test Parameters

To obtain a useful set of measurements, you should choose a range of relevant test parameters. Common parameters include message size, test duration (message count or timer duration, for example), and the quantity of repeated measurements. There may be other relevant parameters, such as quality-of-service and buffer sizes, that may be unique to a specific network software. In performing a series of measurements, only one parameter should be varied while the others should be kept unchanged to simplify the understanding of how a particular parameter affects the performance.

Consider the following example for a fictitious network software where throughput is the desired measurement and the parameter chosen for study is the message size. The range of message sizes are 1, 100, 500, 1000, 1500, 2000, and 2500 bytes, and five measurements, each obtained with Num (message count) of 10,000, are obtained for each message size. Thus, for a 1-byte message, you would obtain five separate throughput measurements, each with Num of 10,000. After determining that the variability among the five measurements is acceptable, you would average the measurements to obtain an estimate of the 1-byte throughput. You would repeat the same for each of the remaining message sizes. To facilitate the measurement process, you can incorporate the test runs into a script file, similar to that in Netperf, where the script automatically initiates the test for the next case after completion of each test run.

Obviously, if a test run crashes, the tests should be suspended and the problem should be investigated. After the problem has been fixed, a fresh round of tests will need to be run from the beginning. Often, after measurements have been made, it may be helpful to import the results into a data analysis software, such as a spreadsheet, and plot the results. In Figure 7, a sample throughput plot for the fictitious software, you can readily observe a peculiar pattern in throughput performance that would have been more difficult to perceive by simply looking at a series of measurement values. For a more comprehensive view of the performance, you should obtain measurements for a greater number of message sizes (or whatever the parameter of interest happens to be).

Test Configurations and Conditions

One of the main underlying ideas behind the performance tests is that they should be conducted under carefully controlled configurations and conditions. The test environment should be such that the tests are repeatable and also so that the results can be compared, not just between those of different network software versions but also between those of different network hardware configurations.

You should not confuse performance monitoring with performance testing. The purpose of performance monitoring is to periodically observe or record the status of the system performance under operational conditions that are often unpredictable. Problems that show up during performance monitoring can be difficult to debug. On the other hand, problems that are encountered during performance testing can be investigated and resolved with much greater ease. Note that the only real distinction between performance monitoring and performance testing, regardless of the techniques or tools used, is the extent to which the test environment is controlled.

There are two basic test configurations: loopback and interCPU. Loopback is where both the Sender and Receiver are collocated in the same CPU. Its purpose is to better characterize the performance of the network software by confining the test to the CPU system and eliminating factors such as the network interface card (NIC) and the rest of the network medium. InterCPU is where the Sender is on one CPU and Receiver is on another. Note that the CPUs do not have to be on different hosts. They can be in the same host, as in a multiprocessor system. One can use different variations of the basic test configurations that are more suitable to the unique requirements of a particular network software.

Through various test configurations, you can obtain measurements for different portions of a communication pipeline. To illustrate, consider a TCP/IP-based network consisting of a router and two hosts where one wishes to measure latency. First, you can obtain an estimate of the latency within the TCP/IP stack by performing the loopback test (Figure 3). Next, without the router in place, the interCPU test between hosts can provide an estimate of the additional latency (for instance, on top of the latency measured through loopback) that is due to the network medium, which includes components such as the network interface card (NIC) driver and hardware (Figure 4). With the router between the hosts, running the interCPU test again can provide an estimate of additional latency that is now due to the router (Figure 5). Going even further, if a TCP/IP-based middleware is added, you can obtain an estimate of the additional latency that is due to the middleware by running an interCPU test between two middleware applications (Figure 6). Through such a process of addition (or elimination), not only can you gain a valuable insight into the performance of a network, but you can also attempt to troubleshoot a problem such as an unusually long latency.

In measurements where the Sender and Receiver tasks are collocated in the same host, the measurements may be appreciably affected by certain additional factors. Such factors would include queue-management efficiency and overhead due to context switching between the Sender and the Receiver because the tasks may now be competing for the same resources. You should exercise care in evaluating results from such self-host tests because these additional factors would not exist in cases where the two tasks are on separate hosts. The same caution also applies to multiconfiguration tests if you use variations where multiple senders or receivers are executed in each host.

An example of collocated tasks occurs in a loopback test. On operating systems that have priority-based, preemptive scheduling, you have to be extra cautious when running the throughput loopback tests where the send function is nonblocking or where the interface uses connectionless protocols such as UDP. If the sender task has a tight send loop and has a priority equal to or greater than that of the receiver task, the sender may consume most, if not all, of the CPU execution time. This will significantly reduce the ability of the receiver to receive messages, resulting in misleadingly low throughputs. In such cases, you should ensure that the priority of the receiver is greater than that of the sender.

Thus far, the discussions on performance measurements assumed that no additional load was added to the network. Inevitably, among certain users of the network software, there will be thoughts on obtaining measurements with the network loaded at specific levels (for example, via carefully controlled transmission of dummy packets between dummy hosts). For the basic latency and throughput tests, injecting additional load into the network will present difficulties. Because the process of injecting the load into a network can itself be affected by the traffic generated by the performance tests, the result can be a load that is uneven and unpredictable. This can, in turn, result in measurements that are statistically unstable and unreliable. Hence, for basic latency and throughput measurements, adding additional load to the network can often produce data that have limited value.

However, there may be situations where additional network load may actually be useful. One is where a network software guarantees bounded latencies or certain quality-of-service, but even here, one can only attempt to produce a load at a certain level since the actual load will, again, depend on interactions with the measurement traffic. Another situation where injecting load may be helpful is where one simply wishes to make qualitative observations on the network software performance in the presence of additional load, especially if the software incorporates algorithms designed to minimize the disruptive effects of network congestion.

Conclusion

Thoroughly testing network software is a necessity. At minimum, you should conduct latency and throughput tests as part of the network software development process -- especially in the early stages because the tests can be used to shake down the software and help determine whether it needs to be redesigned. You should also be aware that they may have to conduct other tests, such as variations of the basic latency and throughput measurements, depending on the requirements that are unique to your software. The main goal of these measurements is to improve the network software quality. Thus, if you care about the quality of the software you create, you should incorporate the tests into your development process. Users of the network software can beneficially contribute to the process by becoming more aware of how network software performance can be measured and becoming better consumers by requesting that the programmers or vendors supply the relevant performance data.

Further Reading

Cooke, D., et al. Statistical Computing in Pascal. Edward Arnold Ltd., 1985, ISBN 0-7131-3545-X.

Jain, Raj, The Art of Computer Systems Performance Analysis, John. Wiley & Sons, 1991, ISBN 0-471-50336-3.

Papoulis, A. Probability, Random Variables, and Stochastic Processes, McGraw-Hill, 1984, ISBN 0-07-048468-6.

DDJ

Listing One

/* the sending (netperf) side of the TCP_RR test. */

void
send_tcp_rr(remote_host)
     char       remote_host[];
{
 . . .
    /* Set-up the test end conditions. For a request/response test, they */
    /* can be either time or transaction based. */

    if (test_time) {
      /* The user wanted to end the test after a period of time. */
      times_up = 0;
      trans_remaining = 0;
      start_timer(test_time);
    }
    else {
      /* The tester wanted to send a number of bytes. */
      trans_remaining = test_bytes;
      times_up = 1;
    }
    /* The cpu_start routine will grab the current time and possibly */
    /* value of the idle counter for later use in measuring cpu */
    /* utilization and/or service demand and thruput. */
    cpu_start(local_cpu_usage);
 . . .
    /* We use an "OR" to control test execution. When the test is */
    /* controlled by time, the byte count check will always return false. */
    /* When the test is controlled by byte count, the time test will */
    /* always return false. When the test is finished, the whole */
    /* expression will go false and we will stop sending data. */

    while ((!times_up) || (trans_remaining > 0)) {

      /* send the request. we assume that if we use a blocking socket, */
      /* the request will be sent at one shot. */
      if((len=send(send_socket, send_ring->buffer_ptr,
                                        req_size, 0)) != req_size) {
        if ((errno == EINTR) || (errno == 0)) {
          /* we hit the end of a timed test. */
          timed_out = 1;
          break;
        }
        perror("send_tcp_rr: data send error");
        exit(1);
      }
      send_ring = send_ring->next;

      /* receive the response */
      rsp_bytes_left = rsp_size;
      temp_message_ptr  = recv_ring->buffer_ptr;
      while(rsp_bytes_left > 0) {
        if((rsp_bytes_recvd=recv(send_socket, temp_message_ptr,
                                           rsp_bytes_left, 0)) < 0) {
          if (errno == EINTR) {
            /* We hit the end of a timed test. */
            timed_out = 1;
            break;
          }
          perror("send_tcp_rr: data recv error");
          exit(1);
        }
        rsp_bytes_left -= rsp_bytes_recvd;
        temp_message_ptr  += rsp_bytes_recvd;
      }
      recv_ring = recv_ring->next;

      if (timed_out) {
        /* we may have been in a nested while loop - we need */
        /* another call to break. */
        break;
      }
      nummessages++;
      if (trans_remaining) {
        trans_remaining--;
      }
  . . .
    }
    /* this call will always give us the elapsed time for the test, and */
    /* will also store-away the necessaries for cpu utilization */
    cpu_stop(local_cpu_usage,&elapsed_time);
  . . .
    /* We now calculate what our throughput was for the test. */
    thruput     = nummessages/elapsed_time;
  . . .
}

Back to Article

Listing Two

/* This routine implements the TCP unidirectional data transfer test */
/* (a.k.a. stream) for the sockets interface. It receives its */
/* parameters via global variables from the shell and writes its */
/* output to the standard output. */
void
send_tcp_stream(remote_host)
char    remote_host[];
{
 . . .
    /* Tell the remote end to do a listen.... */
    /* This call also sends a parameter message to Receiver (remote) side. */
    send_request();
 . . .
    /* receive message from Receiver (remote); contains data such as
     * port number but can be seen as a "ready" status.  -- yun
     */
    recv_response();
 . . .
    /*Connect up to the remote port on the data socket  */
    if (connect(send_socket, (struct sockaddr *)&server, sizeof(server)) <0){
      perror("netperf: send_tcp_stream: data socket connect failed");
      printf(" port: %d\n",ntohs(server.sin_port));
      exit(1);
    }
 . . .
    /* Set-up the test end conditions. For a stream test, they can be */
    /* either time or byte-count based. */

    if (test_time) {
      /* The user wanted to end the test after a period of time. */
      times_up = 0;
      bytes_remaining = 0;
      start_timer(test_time);
    }
    else {
      /* The tester wanted to send a number of bytes. */
      bytes_remaining = test_bytes;
      times_up = 1;
    }
    /* The cpu_start routine will grab the current time and possibly */
    /* value of the idle counter for later use in measuring cpu */
    /* utilization and/or service demand and thruput. */

    cpu_start(local_cpu_usage);
 . . .
    /* We use an "OR" to control test execution. When the test is */
    /* controlled by time, the byte count check will always return false. */
    /* When the test is controlled by byte count, the time test will */
    /* always return false. When the test is finished, the whole */
    /* expression will go false and we will stop sending data. */

    while ((!times_up) || (bytes_remaining > 0)) {

      if((len=send(send_socket, send_ring->buffer_ptr, send_size,
                   0)) != send_size) {
          /* the test was interrupted, must be the end of test */
          break;
        }
        perror("netperf: data send error");
        printf("len was %d\n",len);
        exit(1);
      }
      /* now we want to move our pointer to the next position in the */
      /* data buffer...we may also want to wrap back to the "beginning" */
      /* of the bufferspace, so we will mod the number of messages sent */
      /* by the send width, and use that to calculate the offset to add */
      /* to the base pointer. */
      nummessages++;
      send_ring = send_ring->next;
      if (bytes_remaining) {
        bytes_remaining -= send_size;
      }
    }
    /* The test is over. Flush buffers to the remote end. We do a graceful */
    /* release to insure all data has been taken by the remote. */
 . . .
    if (shutdown(send_socket,1) == -1) {
      perror("netperf: cannot shutdown tcp stream socket");
      exit(1);
    }
    /* hang a recv() off the socket to block until the remote has */
    /* brought all the data up into the application. it will do a */
    /* shutdown to cause a FIN to be sent our way. We will assume that */
    /* any exit from the recv() call is good... raj 4/93 */

    recv(send_socket, send_ring->buffer_ptr, send_size, 0);

    /* this call will always give us the elapsed time for the test, and */
    /* will also store-away the necessaries for cpu utilization */

    cpu_stop(local_cpu_usage,&elapsed_time);

    /* we are finished with the socket, so close it to prevent hitting */
    /* the limit on maximum open files. */

    close(send_socket);

    /* Get the statistics from the remote end. The remote will have */
    /* calculated service demand and all those interesting things. If it */
    /* wasn't supposed to care, it will return obvious values. */

    recv_response();
    if (!netperf_response.content.serv_errno) {
      if (debug)
        fprintf(where,"remote results obtained\n");
    }
    else {
      errno = netperf_response.content.serv_errno;
      fprintf(where,
              "netperf: remote error %d",
              netperf_response.content.serv_errno);
      perror("");
      fflush(where);
      exit(1);
    }
    /* We now calculate what our thruput was for the test. In the future, */
    /* we may want to include a calculation of the thruput measured by */
    /* the remote, but it should be the case that for a TCP stream test, */
    /* that the two numbers should be *very* close... We calculate */
    /* bytes_sent regardless of the way the test length was controlled. */
    /* If it was time, we needed to, and if it was by bytes, the user may */
    /* have specified a number of bytes that wasn't a multiple of the */
    /* send_size, so we really didn't send what he asked for ;-) */

    bytes_sent  = ntohd(tcp_stream_result->bytes_received);

    thruput     = (double) calc_thruput(bytes_sent);
 . . .
}

Back to Article

Listing Three

/* This is the server-side routine for the tcp stream test. It is */
/* implemented as one routine. I could break things-out somewhat, but */
/* didn't feel it was necessary. */

void
recv_tcp_stream()
{
 . . .
  /* The parameter message from the Sender side had already been received
   * by the parent of recv_tcp_stream().  -- yun */
 . . .
  /* Now, let's set-up the socket to listen for connections */
  if (listen(s_listen, 5) == -1) {
    netperf_response.content.serv_errno = errno;
    close(s_listen);
    send_response();

    exit(1);
  }
 . . .
  /* send data such as port number to Sender; can also be seen as
   * Receiver's "reeady" status  -- yun */
  send_response();
 . . .
  if ((s_data=accept(s_listen, (struct sockaddr *)&peeraddr_in,
                     &addrlen)) == -1) {
    /* Let's just punt. The remote will be given some information */
    close(s_listen);
    exit(1);
  }
 . . .
  cpu_start(tcp_stream_request->measure_cpu);
 . . .
  /* The loop will exit when the sender does a shutdown, which will */
  /* return a length of zero   */

  while ((len = recv(s_data, recv_ring->buffer_ptr, recv_size, 0)) != 0) {
    if (len < 0) {
      netperf_response.content.serv_errno = errno;

      netperf_response.content.serv_errno = errno;
      send_response();
      exit(1);
    }
    bytes_received += len;
    receive_calls++;

    /* more to the next buffer in the recv_ring */
    recv_ring = recv_ring->next;
  }
  /* perform a shutdown to signal the sender that */
  /* we have received all the data sent. raj 4/93 */

  if (shutdown(s_data,1) == -1) {
      netperf_response.content.serv_errno = errno;
      send_response();
      exit(1);
    }
  cpu_stop(tcp_stream_request->measure_cpu,&elapsed_time);
 . . .
}

Back to Article