Netperf
Netperf (http://www.netperf.org/) was created by Rick Jones at Hewlett-Packard and based on earlier HP software. Since its release by HP into public domain, Netperf has been extended and improved by a community of users. It is heavily used for testing UNIX network performance (TCP, UDP, and the like), though a version of Netperf is also available for Windows. To make Netperf more useful, there have been a number of attempts to port it to different platforms, with varying degrees of success. This underscores one disadvantage of Netperf, which is that, because of the number of additions made to Netperf, its source code has become increasingly unwieldy. Thus, a programmer who wishes to perform, for example, TCP/IP network tests on nonUNIX platforms for which Netperf is unavailable may discover that it is sometimes quicker to write new, but simpler, network performance measurement programs rather than to port Netperf to that platform, especially where the operating system support is significantly different. Nevertheless, for UNIX network tests, Netperf is still one of the most capable off-the-shelf tools available.
One often misunderstood feature of Netperf is its use of the confidence interval in measuring averages. The purpose of the feature is to obtain the desired statistical information with significantly less amount of test execution time than would have been possible otherwise. In statistics, there are what are known as the "population mean" and "estimated mean." A population is the set of all values that can possibly exist for a random variable, which can be a measurement. The population mean is then the mean computed from this complete set and hence, the population mean is constant. You can also think of the population mean as the true average. Estimated mean is an estimate of the population mean and it is computed from a subset of the population. If the subset is randomly selected, as is the case for the performance measurements, then the estimated mean is also random. As the size of the subset increases, the estimated mean approaches the population mean. Unfortunately for the performance tests, this also means that the test execution time will increase to infinity. Furthermore, there are other reasons why significantly increasing the measurement time length may not necessarily produce measurements that are more accurate, but they are outside the scope of this article.
A goal in the performance measurements is to obtain an estimated mean (that is, an average of multiple measurements such as latencies or throughputs) that is as close to the population mean as that which can be achieved without excessive execution time. Confidence interval is a means of accomplishing this. It works primarily because it assumes that the measurement averages have a certain probability distribution. This allows it to determine how close an estimated mean is to the population mean without requiring a very large number of measurements.
In Netperf, the measurement averages are assumed to have a Gaussian (or normal) distribution, and for most cases, this turns out to be a good approximation. Since the confidence interval is dependent on the number of measurements, Netperf will iteratively repeat the tests until either the desired confidence interval or the user-specified maximum number of iterations is reached. For well functioning network systems (both hardware and software), the number of iterations needed to reach the desired confidence interval will usually be small, resulting in relatively short execution times. (The converse, that if the number of needed iterations is small, then the network is well functioning, is not necessarily true because a poorly designed network software can perform poorly on a consistent basis.) When Netperf fails to achieve the desired confidence interval within a reasonable number of iterations, it is usually because of a wide variability in the underlying measurements. Whatever the cause of this failure, Netperf will print the actual confidence interval it was able to obtain, which you should examine to determine whether it is still acceptable.
When Netperf updates the confidence interval with each new iteration, it uses a lookup table for the t distribution. (You can think of the t distribution as what the Gaussian distribution would look like if you had only a limited number of measurements or observations. The number of measurements minus one is also known as "degrees of freedom.") Because of the finite size of the table, you can only specify a confidence level of either 95 or 99 percent. Furthermore, the maximum number of iterations is set to 30, again, due to the limited size of the table because the number of iterations (or measurements) minus one is really the degree of freedom in the t distribution. You can specify a greater number of iterations, but Netperf will truncate it to 30.
When a measured average is reported by Netperf to have a confidence interval of 10 percent with a confidence level of 95 percent, it means that the probability is 95 percent that the population average is within +/-5 percent of the measured average. It can also be interpreted to mean that there is a 5 percent chance that the population average is outside the confidence interval. The actual population average is never known. In general, specifying smaller confidence intervals or greater confidence levels may result in Netperf executing a greater number of iterations because more measurements may be needed to satisfy the tighter statistical conditions. (In fact, if you specify a confidence interval and level that are too tight, Netperf may never be able to obtain measurements that satisfy your conditions.) On the other hand, without a confidence interval, you would need far greater number of measurements, and thus far greater amount of execution time, to achieve a similar degree if certainty in the results.
By the way, the usage of the confidence interval is far more common than you might think. For instance, the margin of error that is often reported in polls by the media is actually the confidence interval of the poll results, typically at 95 percent confidence level. Because conducting polls and surveys can be expensive, only a small number of people are randomly sampled and the results are reported as reflecting those of the general population. Of course, you can't really compare polls with network performance measurements because, unlike machines, people can lie or change their minds, but both polls and Netperf have the same motivation of reducing costs without sacrificing the quality of results when they use the confidence interval methodology.
-- J.K.Y.
