Mike Bunnell and Mitch Bunnell

The acquisition and storage of large amounts of data is vital in many industrial, research, and aerospace/defense applications in which computer systems typically must collect analog data at very high rates (1,000 to 1,000,000 samples/sec.) for relatively long periods (a few seconds to a few days). Consequently, computers used in these environments must be able to assemble huge volumes of data in real time and store them in an organized way so that the information can be analyzed later. Until recently, systems that could do this cost more than $100,000. With current hardware and software technologies, however, equivalent PC-based systems are one-tenth that price.

A computer system -- large or small -- designed for real-time data acquisition must meet four basic requirements: It needs to have an interface that converts continuous real-world data into discrete, digitized samples at a rapid sampling rate; there must be sufficient throughput to mass storage to save the samples at the sampling rate; a data acquisition program must read the samples from the interface and write them to mass storage; and the software must have sufficient real-time response so data samples are never lost. In this article, we'll describe the elements of a common PC-based data acquisition system and show how it can be used in a typical application.

Although most real-world data is analog in nature, digital computers, by definition, can only process discrete digital data. To solve this dilemma, an analog-to-digital (A/D) interface is needed. Because analog data is continuous, the A/D interface must take a snapshot of the analog data at one point in time. This process, called sampling, is usually done at a particular sampling rate depending on how fast the analog data is expected to change. The interface must digitize each sample by converting it to a numerical value. The resolution, or number of bits, used to represent this value is dependent on the analog interface. Unless the sampling rate is slow (less than 100 samples/sec.) the interface must provide some way of buffering the digitized data until the CPU accesses it. This can be done either by providing a first-in, first-out (FIFO) buffer as part of the analog interface or direct-memory-access (DMA) capability so the interface can save the data directly to computer memory without using the CPU.

Handling the analog interface is left to a device-dependent code -- the device driver. The device driver sets the operating parameters of the interface, maintains the data buffers, and services requests to read converted data or set the sampling rate. In the same way as the analog-to-digital converter is the interface between the computer and the real world, the device driver is the interface between a data acquisition program and the converter hardware.

In a data acquisition system, sampled data is saved to mass storage (typically a hard disk) for later processing. The reason for saving the data to disk and not just computer memory is that there is usually more data that must be saved than there is memory available. For convenience sake, it is preferable that data be saved in named files on the disk so the data can be conveniently accessed simply by reading it from the appropriate files.

Writing data to a file must be fast enough to keep up with the sampling rate. If, for example, the sampling rate is 100,000 samples/sec. and each sample is 2 bytes, then the computer must be capable of writing 200,000 bytes/sec. to a file on a sustained basis. Although disk-drive manufacturers may quote burst transfer rate and seek rates, these do not reveal the sustained data rate. The best way to determine this rate is to do real-life testing.

This testing process must include writing data to a file, a process that is typically slower than writing directly to the disk. On most operating systems (including Unix and DOS), file access is optimized assuming random intermittent access and blocks are dynamically added to the file as it grows. The data blocks that make up a file may be scattered all over the disk, making access to some files faster than to others. To provide faster reads and writes to files used for high-speed data acquisition, some operating systems offer contiguous files that have data blocks that are preallocated on the media to provide maximum speed for sequential access. The file cannot grow dynamically beyond a preset maximum size, but it can be written to as fast as writing directly to the disk.

To acquire data at high speed, there must be a program to read the data from the analog input device and write it to the file on mass storage. This program must use a FIFO buffering scheme to simultaneously read data from the analog device and write it to disk. Buffering is important because it allows large buffers to be transferred to disk without losing input samples during the disk access. Although this can be done with a single task and asynchronous I/O, it is much easier to use two tasks -- one to store sampled data into the buffer and one to write data from the buffer to disk. The ability of a program to write to a file on disk presupposes that the program is running under some operating system. A multitasking operating system is needed to support the data acquisition program so it can read and write simultaneously (the CPU can't continually read from the analog input if it is in a busy wait loop waiting for the disk).

For a real-time data acquisition system to work, much care must be taken to ensure that samples are not lost because the input buffer of the analog device driver has overflowed. Because the input buffer is filled at the sampling rate, the buffer must be emptied within a certain period of time.

In a multitasking system, the problem of emptying the input buffer in time becomes one of CPU throughput and task response. The computer must run fast enough to be able to copy the data from the input buffer before any of it is overwritten. This is the easy part; 32-bit microprocessors can typically copy 2 to 8 Mbytes/sec. and the setup time is minimal. The more difficult part is task response. The input task waits for the input buffer to fill to a certain point by giving up use of the CPU. This gives other tasks in the system the chance to run (such as the task that writes the sampled data to disk). When the buffer has filled to the certain point, the device driver schedules the input task to run. The time to schedule a task, the amount of time for which task switches are disabled, and the time it takes to do a task switch all contribute to the worst-case task response.

Many operating system manufacturers quote only the best and typical task responses because they have not taken care to limit the time for which task switches are disabled. This is not useful because data samples can still be lost if the input task cannot respond in time occasionally. For this reason, programmers must construct their code with the worst-case task response times in mind rather than best, typical, or average response times.

A real-time operating system guarantees fast worst-case task response, making the operating system an important part of a high-speed data acquisition system. These requirements and how they can be met are best explained with an example of a data acquisition system.

For the purposes of this article, we'll describe a typical data acquisition system that can collect 12-bit analog data on up to 16 channels and save it to disk at rates from as low as once every two seconds up to 250,000/sec. (aggregate). The hardware in our system consists of an 80386 AT-compatible computer, an analog I/O board, and a high-capacity hard disk. The software consists of a real-time operating system, an analog I/O device driver, and a couple of utility programs.

Before we start acquiring data, we must make sure that we have the real-time response necessary to make our system work. The task reading from the DT2821 device has to read the data out of the circular buffer before the DMA controller overwrites the data. The task reading is awakened by the timer to read out of the 4K circular buffer after the DMA has time to fill half the buffer. This gives the task half of the time it takes to fill the buffer to get the data. At our maximum DMA transfer speed of 500K/sec., 2,048/500,000 seconds (approximately 4 milliseconds), are available to transfer the data.

The operating system guarantees 500 microseconds (0.50 milliseconds) worst-case response for the highest priority task on a 4-MIP 80386 computer. Therefore, if the task reading from the DT2821 is the highest priority task, we are assured of success. If there are other tasks at higher priority, such as another data acquisition task, we would have to measure its longest continuous CPU usage and add that to the 500 microseconds and make sure that that value was less than 4 milliseconds.

We can estimate the longest continuous CPU usage for our data acquisition tasks. The task accessing the DT2821 incurs a system call overhead of 25 microseconds when reading or writing. The longest stretch of time the DT2821 device driver uses the CPU is the 0.3 to 0.6 milliseconds it takes to copy 2K. Thanks to contiguous files, the task writing to the disk uses even less CPU time, only about 70 microseconds to set up the DMA controller and SCSI controller to transfer the data to or from disk, including the system call overhead. The continuous CPU usage for both data acquisition tasks is 0.02 + 0.60 + 0.07 = 0.69 milliseconds.

Thus the worst-case response of a task running at lower priority than both tasks doing the data acquisition is the guaranteed worst-case response of 500 microseconds plus the CPU usage of the data acquisition tasks, or 0.69 + 0.50 = 1.19 milliseconds.

As an application of a high-speed data acquisition system, we can record a song on the hard disk at compact disc speeds and then play it back, first on just one channel, then in stereo.

This particular application shows acquisition of analog data, which is the kind of data usually acquired. The rate of acquisition is on the same order as that of a typical application. The quantity of data acquired is also common to many high-speed data acquisition applications. Finally, recording and playing back music is a good test of the acquisition system because you can hear whether data is acquired properly when it is played back.

The computer system described in this article meets all requirements of a high-speed data acquisition system. The DT2821 analog I/O board serves to change the real-world data to discrete digital data. The SCSI disk system provides the high-speed mass storage capability. LynxOS, dbuff, and the device driver for the DT2821 are the software that performs the acquisition. The operating system guarantees the real-time response and provides the environment to analyze the data.

All source code for articles in this issue is available on a single disk. To order, send $14.95 (Calif. residents add sales tax) to Dr. Dobb's Journal, 501 Galveston Dr., Redwood City, CA 94063, or call 800-356-2002 (from inside Calif.) or 800-533-4372 (from outside Calif.). Please specify the issue number and format (MS-DOS, Macintosh, Kaypro).

Real-Time Data Acquisition by Mike and Mitch Bunnell

<a name="0111_000d">

     /* Reverb.c    IIR filter program to add reverberation */

     #include  <file.h>

     extern char *malloc();

     ewrite(s)
     char *s;
     {
          write(2, s, strlen(s));
     }

     /*
          Read the whole size read() under UNIX returns the amount it
          read.  Last buffer is (biased) zero-filled.
     */
     fullread(fd, buff, size)
     int fd;
     char *buff;
     int size;
     {
          int i, j;

          i = 0;
          do {
              j = read(fd, &buff[i], size - i);
              if (j <= 0) {
                  /* This must be the last buffer of the file */
                  while (i < size)
                      buff[i++] = 0x800;
                  return -1;
              }
              i += j;
         }  while (i < size);

         return size;
     }

     main(ac, av)
     int ac;
     char **av;
     {
          short *ibuff, *obuff;
          int delay;
          int i;
          int fd;
          int rundown;
          int rv;
          char *fn;
          register short *p, *q;

          if (ac > 2) {
              ewrite("usage: reverb [delay]\n    (delay expressed in samples)\n");
              exit(1);
          }
          if (ac == 2)
              delay = atoi(av[1]);
          else
              delay = 10240;

          /* make sure delay is multiple of 512 bytes */
          delay -= delay & 511;

          /* make delay >= 512 andd <= 128K           */
          if (delay < 512)
              delay = 512;
          if (delay > 128*1024)
              delay = 128*1024;

          fd = 0;

          ibuff = (short *) malloc(delay * sizeof(*ibuff));
          obuff = (short *) calloc(delay * sizeof(*obuff));

          do {
              /* Read a buffer, but don't check error status yet */
              rv = fullread(fd, ibuff, delay * sizeof(short));

              /*
                Add the fresh input samples to the old samples, after
                dividing the old samples by 2
              */
              for (p = ibuff, q = obuff, i = 0; i < delay; ++i, ++p, ++q)
                 *q = ((*q - 0x800) >> 1) + *p;

              /*
                 Write the output reverbed buffer
              */
              write(1, obuff, delay * sizeof(short));
          } while (rv != -1);

          /*
              Allow sound in output buffer to "die down"
          */
          for (rundown = 11; --rundown >= 0; ) {
              for (q = obuff, i = 0; i < delay; ++i)
                   *q = (*q - 0x800) >> 1;

              write(1, obuff, delay * sizeof(short));
          }
     }






<a name="0111_000e"><a name="0111_000e">
<a name="0111_000f">

<a name="0111_000f">

     /*  reverse.c   Write a file in reverse to standard output */

     #include  <file.h>
     #include  <types.h>
     #include  <time.h>
     #include  <stat.h>

     main(ac, av)

     int ac;
     char **av;
     {
          int fd;
          short buff[4096];
          int rc;
          int i, j, t;
          long pos;
          struct stat s;

          ++av;
          if ((fd = open(*av, O_RDONLY, 0)) == -1) {
              perror(*av);            /* exit if can't open file */
              exit(1);
          }

           fstat(fd, &s);             /* find the size of the file */
          pos = s.st_size &  1;

          while (pos > 0) {
              /* See how many bytes can be read now */
              if (pos < sizeof(buff))
                  rc = pos;
              else
                  rc = sizeof(buff);

              pos -= rc;
             /* Seek back a block and read */
             lseek(fd, pos, 0);
             read(fd, buff, rc);

             /* Reverse the samples in the block */
             for (i = 0, j = (rc / 2) - 1; i < j; ++i, --j) {
                 t = buff[i];
                 buff[i] = buff[j];
                 buff[j] = t;
             }

             /* Write the reversed block */
             write(1, buff, rc);
           }

          close(fd);
     }

Copyright © 1989, Dr. Dobb's Journal