Writing Serial Drivers for UNIX

More often than not, things don't go right with UNIX serial ports because the serial drivers are often kludges and hacks. Bill implements a reliable serial driver for FreeBSD from the ground up.


December 01, 1994
URL:http://www.drdobbs.com/architecture-and-design/writing-serial-drivers-for-unix/184409364

DEC94: Writing Serial Drivers for UNIX

Dialing in and out on the same serial line

Bill, a longtime UNIX programmer, can be contacted at [email protected].


If you've ever butted heads with UNIX serial ports, you've noticed that things often don't work just right. One reason is that most serial drivers are derived from a few ancestral sources, which have been hacked and kludged until they work--more or less. (Indeed, I've read two books on writing device drivers that start out by saying that the only practical way to write a terminal-like driver is by starting from someone else's driver.)

This is certainly the case with the serial driver that is part of FreeBSD, the UNIX-like operating system I run on my machine. That driver started out as the HP dca driver, was modified into the com.c driver, and was finally used to build the sio.c driver. It wouldn't surprise me at all to discover other transformations in there, too. The FreeBSD system is based on the 386BSD UNIX-like operating system developed by William Jolitz and described in a series of DDJ articles entitled "Porting UNIX to the 386" (January 1991--July 1992). FreeBSD is available from a number of sources, including Walnut Creek CD-ROM and the ftpmail server at gatekeeper.dec.com.

While the FreeBSD serial driver has several minor annoyances, its major defect is that your system can't dial in or out on the same serial line without manual intervention. After mulling over the options of further hacking the FreeBSD driver, waiting for developers to fix the problem, or porting an existing driver, I opted to write a new driver altogether. (This driver has since been ported to NetBSD, another 386BSD descendant.)

To successfully write a serial driver, you must have an understanding of concurrency and control flow in the device driver, the kernel interface, and the serial device itself. Good software-engineering practice dictates that the various parts of the driver be distinguished and separate; my driver has five sections: declarations, debugging and statistic facilities, hardware manipulation, state changes, and system-call interface.

Coding the Serial Driver

In my serial driver, the declarations section contains type and variable declarations specific to the driver itself. In particular, the LINE_STATE enumeration describes the overall state of a line. One essential design decision was to describe the line-state concept using a scalar rather than a host of variables scattered through various data structures. By categorizing and characterizing the four primary states of a line before I began coding, I avoided most of the problems that beset other drivers. I'll discuss this further when I talk about the driver's open routine.

If you've ever written code that involved asynchronous events, you'll know just how difficult it can be to debug--bugs tend to depend on timing, so they are often very difficult to reproduce, much less track down. Ideally, you want a complete trace of the events that occurred just before the bug bit. In reality, such a trace requires that expensive hardware or CPU time be spent executing otherwise useless debugging code.

Each function entry, function exit, and significant driver action gets recorded in a circular buffer. This is moderately expensive in time, but a judicious use of inline assembly and functions in the debugging code keeps cost down. An external program takes a snapshot of the circular buffer, then interrupts and prints it out so that you can make sense of it. If the driver locks up the machine, the debugging code can be configured to record events on the console screen where, by no coincidence, the event codes result in distinguishable, readable characters on the screen.

Another useful item in this section is a "status print" routine. FreeBSD's kernel debugger, ddb, can call an arbitrary kernel function. At any time, you can break into the debugger and call this routine for a dump of most of a line's variables.

The hardware-manipulation section is the only part of the driver that knows the details of the hardware. Separating this part of the code let me implement a "virtual UART," which separates the bit-twiddling code from the primary driver logic. This also makes it relatively easy to support different sorts of hardware.

One difficulty of writing a serial device driver is that UNIX has a long interrupt latency. Consequently, serial drivers coded the "standard" way lose characters. One solution, "pseudo-dma," available in FreeBSD and used in my driver, replaces the standard interrupt-handler code with a much simpler one. Instead of establishing a normal UNIX execution thread with all the overhead that entails, the serial device's interrupt goes directly to the interrupt handler. Instead of calling standard UNIX functions to transfer data to and from the device (which is rather slow), data is transferred to and from the driver's control structures. The handler signals the UNIX kernel when it needs data or when it has data available; this functions much like an interrupt except that it is software generated. The "soft-interrupt" handler does the same work as a standard interrupt handler, except that instead of reading and writing to a device, it reads and writes to the driver's control structures.

The state-transition section moves data from here to there and keeps track of the state of the driver. Most of the tricky logic goes here. System calls eventually result in calls into this section, and the soft-interrupt handler is here. This part of the driver knows that it is dealing with a serial device but it relies on the hardware routines to do the actual device manipulation.

In many drivers, an attempt is made to propagate changes to driver variables throughout all the other variables that might be affected, This usually results in extremely complex code, full of incomprehensible conditionals--that never quite work right. In my driver, wherever this might be a problem, I use a different approach. I centralize the computation of the variable in a single routine, and then whenever anything might require a change to the variable, I call that routine. Instead of a computation from cause to effect that (hopefully!) considers exactly the effect that the change causes, I fully recompute the variable. The variable always has the right value, and if this is less efficient than the alternative, it isn't measurably so.

The system-call interface primarily handles system calls from user processes. This is where your open/close/read/write calls end up in the driver. This is fairly straightforward code, except for the open routine in Listing Two .

Suppose you want to dial out on a modem. You need to communicate with the modem to get it to dial. When a connection is established, you then need to let the dial-out application gain control of the line. If the carrier goes away, you want the application to receive a hang-up signal.

Dialing in, on the other hand, requires that a front-end program (typically getty) monitor the line for a connection indicated by the presence of a carrier, then execute the application once the connection exists. If you were only dialing in on the line, this would be easy: Just wait for input and proceed. However, if you dial both in and out on the same line, things get trickier. Sure, you could make everything work with the help of application-level interlocking. This involves finicky code that every single application must get right. Rather than do this, it's better to get the driver to help out.

One approach is to prevent an open dial-in device from completing until a carrier is present. Then getty can simply attempt to open the line; once the open is successful, it knows there is a connection on the other side.

While you could have two varieties of opens--blocking for dial-in and nonblocking for dial-out--this isn't enough. What happens to an open that occurs while the line is open? There's a carrier, so both types of open will complete. If that open is a getty while a dialect is in progress, it could be made to work, but this, too, would involve ugly interlocking code in each dial-in and dial-out application.

A better approach is to have the driver distinguish between dial-out and dial-in opens; when either one has completed, the other is prevented from completing. This leaves open how the driver is to distinguish the two sorts of opens. There are a number of ways to do this; the one I chose is to distinguish the type of open by a bit in the minor device number.

Devices in the range 0--127 are dial-in devices; those in the range 128--255 are dial-out. A getty tries to open a dial-in device, but it cannot succeed until there is a carrier and no dial-out device is open. A dial-out program, such as uucico, tries to open a dial-out device. This succeeds unless the corresponding dial-in device is open, in which case the open fails immediately.

It would be nice if the UNIX kernel kept track of who is open and who is waiting to open; unfortunately, it doesn't. First, the kernel has no notion that the dial-in and dial-out devices are related. Second, the kernel does not try to keep track of opens and those waiting for opens; for most other types of drivers, it just isn't needed. Keeping track is thus left to the driver itself.

The primary parameter for line state is whether opens are waiting or active for the line. When there are none, no one is using or trying to use the driver. DTR is not sent by the UART, and the UART is ignored. Once an open is attempted, one of the open wait counts goes nonzero. In that state, DTR is turned on to let a modem know that a process is preparing to use the line. The modem-status lines are monitored for a carrier. When the open completes, the wait count is decremented but the appropriate active flag is set. Modem controls are used to manipulate the modem, modem status is used for flow control and for discovering loss of carrier, and data is transferred to and from the line.

These conditions are summarized in a single LINE_STATE variable. Each time one of the open wait counts or active flags changes, this variable is recomputed; see Listing One . If it changes, various bits in the driver are changed appropriately and the UART is programmed for that state.

(There is one more state: When the last close for a line happens, the driver is placed in a "shutdown state" while it does the things needed to clean up the line. During the shutdown state, no opens are allowed on the line, and, as the last open has been closed, the line is entirely under the control of the driver. That lets it do things such as manipulate the modem-control lines to hang up a modem without interference.)

The open Routine

Having a state variable makes coding the open a snap. The heart of the open is of a single loop. On entry to the loop, the open wait count is incremented. The loop is only exited when the open is to fail with an error or succeed. In either case, the open wait count is decremented, because the open is not waiting anymore. Then, the open either returns an error or sets the appropriate open active flag and actually opens the line.

The loop is, conceptually, very simple. At the top of the loop, the line status is tested and the driver set to the appropriate state. Then a series of tests are performed. A given test says either that this open may not succeed, in which case the loop is exited with an error, or that this open is to succeed right now, in which case the loop is exited without an error. If all the tests fall through, the open goes to sleep and stays asleep until something relevant changes. When it wakes up, it goes right back up to the top of the loop, where it does all of the tests again.

Coding this loop was, in fact, my main reason for writing a new driver. Most drivers simply get this wrong. Typically, they use two loops, which test for different open conditions, both of which have to be true before the open may succeed. However, if you have two loops, there is either a sequence of events that causes the open to succeed when it shouldn't or a sequence that causes an open to deadlock, thus preventing the open from succeeding when it should.

About Ring Buffers

One problem facing driver developers is efficiently moving the data between the interrupt handlers and the rest of the driver. For small amounts of data, this isn't difficult; but the characters sent and received are a large amount of data that must be moved quickly. Functionally, you need a queue: characters placed in one end, where they sit until extracted from the other. One of the most time- and space-efficient ways to implement a queue is the "ring buffer," or circular queue.

A ring buffer is an array of elements with two pointers: "write" and "read." A character is added to the ring buffer by storing it at the write pointer, then advancing the write pointer. A character is removed from the ring buffer by loading it from the read pointer, then advancing that pointer. When the pointer advances past the end of the ring buffer, it is made to point to the start of the buffer.

There is a "gotcha" when writing ring-buffer code: When the ring buffer is empty, the read and write pointers are equal; the same is true when it is full. A full ring buffer must be distinguished from an empty one, so the ring buffer must not be allowed to become full.The solution I use involves having the read pointer trail one behind the actual read position. Instead of read and advance, I advance and read. As the write pointer is not allowed to advance past the read pointer, it can never actually reach the read position, so the buffer cannot become full.

While writing the ring-buffer code in Listing Three , I discovered that if you are careful, it is not necessary to do any sort of interlocking between buffer readers and writers--you can interleave the execution of a read routine and a write routine, and things will still work. The trick is that at the front of the read routine, the read pointer is accessed once and stored in a temp. This value is then compared with the write pointer to check for an empty buffer. Once data is retrieved, the updated pointer value is stored back in the ring buffer's pointer. A similar procedure is followed in the write routine. To enforce this access/compare sequence, the pointers are declared volatile. This tells the compiler not to do things like optimizing those operations into something unexpected.

The end result is that the top half of the driver can add characters to the write buffer while the interrupt routine reads from it, without any special precautions being taken. Similarly, characters read by the interrupt routine can be placed in the read ring buffer, without worrying if the top half of the driver was reading from it.

The ring-buffer code in the driver is stand-alone, implemented as a C include file, and not restricted to character elements. It is intended to be portable and useful in applications other than the driver.

--B.W.

Listing One


STATIC void
sio_change_line_state(SIO_CTL *ctl)
{
    LINE_STATE new_state;
    LINE_STATE old_state;
    sio_record_call(EV_CHANGE_LINE_STATE, ctl->sc_unit, 0);
    /* What should the new state be? Return if no change. */
    if (ctl->sc_shutdown != CL_NONE) {
        new_state = ST_SHUTDOWN;
    } else if (ctl->sc_actin || ctl->sc_actout) {
        new_state = ST_ACTIVE;
    } else if (ctl->sc_winc || ctl->sc_woutc) {
        new_state = ST_WOPEN;
    } else {
        new_state = ST_INACT;
    }
    old_state = ctl->sc_lstate;
    if (old_state == new_state) {
        sio_set_modem_control(ctl);
        sio_record_return(EV_CHANGE_LINE_STATE, 0, 0);
        return;
    }
    sio_record_event(EV_LSTATE, new_state, 0);
    ctl->sc_lstate = new_state;
    if (new_state == ST_ACTIVE) {
        sio_flush_input(ctl);
    }
    if (new_state == ST_INACT) {
        ctl->sc_rtsline = ctl->sc_dtrline = 0;
        ctl->sc_carrier = 0;
    }
    sio_set_interrupt_state(ctl);
    if (old_state == ST_INACT) {
        ctl->sc_rtsline = ctl->sc_dtrline = 1;
    }
    sio_set_modem_control(ctl);
    sio_wake_open(ctl);
    sio_record_return(EV_CHANGE_LINE_STATE, 1, 0);
}


Listing Two


int
sioopen(dev_t dev, int flag, int mode, PROC *p)
{
    SIO_CTL *ctl;
    bool_t  callout;
    dev_t   unit;
    spl_t   x;
    TTY     *tp;
    const char *reason;
    error_t error;
    SIO_IF_DEBUG(static ucount_t onum;)
    sio_record_call(EV_SIOOPEN, minor(dev), flag);
    /* Extract the unit number and callout flag. */
    SIO_IF_DEBUG(++onum;)
    unit = UNIT(dev);
    if ((u_int)unit >= NSIO || !(ctl = sio_ptrs[unit])) {
        sio_record_return(EV_SIOOPEN, 0, ENXIO);
        return (ENXIO);
    }
    callout = CALLOUT(dev);
    dev = makedev(major(dev), UNIT(dev));
    tp = ctl->sc_tty;
    /* Record that we're waiting for an open. */
    if (callout) {
        ++ctl->sc_woutc;
    } else {
        ++ctl->sc_winc;
    }
    sio_set_wopen(ctl);
    error = 0;
    x = spltty();
    while (1) {
        /* Get the device set up as necessary. */
        sio_change_line_state(ctl);
        /* If the line is set to exclude opens, and if the line is
           actually open, forbid anyone but root from opening it. */
        if ((tp->t_state & TS_XCLUDE)
            && (ctl->sc_actout || ctl->sc_actin)
            && p->p_ucred->cr_uid != 0) {
            error = EBUSY;
            break;
        /* Shutdown temporarily prevents all opens. */
        } else if (ctl->sc_lstate==ST_SHUTDOWN) {
            reason = "sioocls";
        /* A dialout open succeeds unless there is an active
           dialin open, in which case it fails. */
        } else if (callout) {
            if (!ctl->sc_actin) {
                break;
            }
            if (!(tp->t_cflag & CLOCAL)) {
                error = EBUSY;
                break;
            }
            reason = "sioinw";
        /* A dialin open will not succeed while there are active or 
                pending dialout opens. It also requires a carrier or clocal. */
        } else {
            if (ctl->sc_actout || ctl->sc_woutc) {
                reason = "sioout";
            } else if (!(tp->t_cflag & CLOCAL)
                && !(tp->t_state & TS_CARR_ON)) {
                reason = "siocar";
            } else {
                break;
            }
        }
        /* If we're here, either the line was in shutdown or a dialin
           open is going to wait. If this is a nonblocking open,
           return. Otherwise, sleep. */
        if (flag & O_NONBLOCK) {
            error = EWOULDBLOCK;
            break;
        }
        sio_record_event(EV_SLEEP, onum, 0);
        error = tsleep((caddr_t)ctl, TTIPRI | PCATCH,
            reason, 0);
        sio_record_event(EV_WAKE, onum, error);
        if (error != 0) {
            break;
        }
    }
    /* The open has succeeded. We're no longer waiting for open.*/
    if (callout) {
        --ctl->sc_woutc;
    } else {
        --ctl->sc_winc;
    }
    sio_set_wopen(ctl);
    /* If the open errored, reset the device and return the error. */
    if (error != 0) {
        sio_change_line_state(ctl);
        splx(x);
        sio_record_return(EV_SIOOPEN, 1, error);
        return (error);
    }
    /* Next, set up the tty structure. */
    tp->t_oproc = siostart;
    tp->t_param = sioparam;
    if (!(tp->t_state & TS_ISOPEN)) {
        tp->t_dev = dev;
        ttychars(tp);
        if (!tp->t_ispeed) {
            tp->t_iflag = 0;
            tp->t_oflag = 0;
            tp->t_cflag = CREAD | CS8 | HUPCL;
            tp->t_lflag = 0;
            tp->t_ispeed = tp->t_ospeed = TTYDEF_SPEED;
        }
        ttsetwater(tp);
        (void)sioparam(tp, &tp->t_termios);
    }
    /* Do the line discipline open. This marks the line open. */
    error = (*linesw[tp->t_line].l_open)(dev, tp, 0);
    if (error != 0) {
        sio_change_line_state(ctl);
        splx(x);
        sio_record_return(EV_SIOOPEN, 2, error);
        return (error);
    }
    /* The line is now open. Let it rip. */
    if (callout) {
        ctl->sc_actout = 1;

        /* Dialout devices start by pretending they have
           carrier. */

        tp->t_state |= TS_CARR_ON;
    } else {
        ctl->sc_actin = 1;
    }
    sio_set_wopen(ctl);
    sio_change_line_state(ctl);
    splx(x);
    sio_record_return(EV_SIOOPEN, 3, error);
    return (error);
}


Listing Three


#include <stdlib.h>

#if !defined(RB_PREFIX)
#define RB_PREFIX rb_
#define RB_TYPE char
#define RB_CONTROL RBUF
#define RB_QUAL static
#endif

#define RB_GLUE1(x,y) x ## y
#define RB_GLUE(x,y) RB_GLUE1(x,y)
#define RB_NAME(x) RB_GLUE(RB_PREFIX, x)

#if !defined(RB_SET)
#define RB_SET(buf,rh,wh) (\
    (buf)->rb_rhold = (buf)->rb_start + (rh),\
    (buf)->rb_whold = (buf)->rb_start + (wh))
#define RB_GET(buf,rh,wh) (\
    (rh) = (buf)->rb_rhold - (buf)->rb_start,\
    (wh) = (buf)->rb_whold - (buf)->rb_start)
#endif
#if !defined(RB_OVERHEAD)
#define RB_OVERHEAD (1)
#endif

typedef struct {
    RB_TYPE *volatile rb_rhold;
    RB_TYPE *volatile rb_whold;
    RB_TYPE *rb_start;
    RB_TYPE *rb_end;
    size_t  rb_size;
} RB_CONTROL;
/* This initializes a ring buffer. */
RB_QUAL void
RB_NAME(init)(RB_CONTROL *p, RB_TYPE *d, size_t n)
{
    p->rb_start = d;
    p->rb_end = d + n;
    p->rb_size = n;
    p->rb_whold = p->rb_start;
    p->rb_rhold = p->rb_end - 1;
}
/* Returns the size of the ring buffer. Note that this is the maximum number
   of elements that may be placed in it, not the size of allocated area. */
RB_QUAL size_t
RB_NAME(size)(const RB_CONTROL *p)
{
    return (p->rb_size - 1);
}
/* This writes one datum to the ring buffer. The return value is the
   number of items written, 0 or 1. */
RB_QUAL size_t
RB_NAME(putc)(RB_CONTROL *p, const RB_TYPE *d)
{
    RB_TYPE *wp;
    wp = p->rb_whold;
    if (wp == p->rb_rhold) {
        return (0);
    }
    *wp++ = *d;
    if (wp == p->rb_end) {
        wp = p->rb_start;
    }
    p->rb_whold = wp;
    return (1);
}
/* This writes an arbitrary number of elements to the ring buffer. The
   return value is the number of items written. */
RB_QUAL size_t
RB_NAME(puts)(RB_CONTROL *p, const RB_TYPE *d, size_t n)
{
    RB_TYPE *rh = p->rb_rhold;
    RB_TYPE *wp;
    size_t  c;
    size_t  r;
    /* If the data in the buffer is wrapped, the hole into which data may
     be placed is not wrapped. This makes a wrapped buffer be the easy case. */
    wp = p->rb_whold;
    if (wp < rh) {
        c = rh - wp;
        if (n < c) {
            c = n;
        }
        if (!c) {
            return (0);
        }
        r = c;
        do {
            *wp++ = *d++;
        } while (--c);
        p->rb_whold = wp;
        return (r);
    }
    /* This next case handles an unwrapped buffer where the data
       fits before the end of the buffer. */
    c = p->rb_end - wp;
    if (c >= n) {
        c = n;
        if (!c) {
            return (0);
        }
        r = c;
        do {
            *wp++ = *d++;
        } while (--c);
        if (wp == p->rb_end) {
            wp = p->rb_start;
        }
        p->rb_whold = wp;
        return (r);
    }
    /* Finally, deal with the case where data will wrap. Since the write
       pointer is never at the end of the buffer, there is always one 
       element in the buffer. So, this copy doesn't require testing. */
    r = c;
    n -= r;
    do {
        *wp++ = *d++;
    } while (--c);
    /* Next, copy data to the start of the buffer. This might not
       copy any data if rhold is at the start of the buffer. */
    wp = p->rb_start;
    c = rh - wp;
    if (n < c) {
        c = n;
    }
    if (c) {
        r += c;
        do {
            *wp++ = *d++;
        } while (--c);
    }
    p->rb_whold = wp;
    return (r);
}
/* Returns the number of elements that may be put into the buffer. It is, in
   effect, a put routine, which means that you can't call it in a context where
   it might overlap with a put of the ring buffer. However, asynchronous gets 
   may occur, which would increase the number of available elements to above 
   what this routine returns. */
RB_QUAL size_t
RB_NAME(pcount)(const RB_CONTROL *p)
{
    RB_TYPE *rp = p->rb_rhold;
    RB_TYPE *wp = p->rb_whold;
    return (rp < wp ? p->rb_size - (wp - rp) : rp - wp);
}
/* This reads one datum from the ring buffer. The return value is the
   number of items returned, 0 or 1. */
RB_QUAL size_t
RB_NAME(getc)(RB_CONTROL *p, RB_TYPE *d)
{
    RB_TYPE *rp;

    rp = p->rb_rhold + 1;
    if (rp == p->rb_end) {
        rp = p->rb_start;
    }
    if (rp == p->rb_whold) {
        return (0);
    }
    *d = *rp;
    p->rb_rhold = rp;
    return (1);
}
/* This reads an arbitrary number of items from the ring buffer. The
   return value is the number of items returned. */
RB_QUAL size_t
RB_NAME(gets)(RB_CONTROL *p, RB_TYPE *d, size_t n)
{
    RB_TYPE *wh = p->rb_whold;
    RB_TYPE *rp;
    size_t  c;
    size_t  r;
    /* Handle the easy case, where the buffer is not wrapped. */
    rp = p->rb_rhold + 1;
    if (rp == p->rb_end) {
        rp = p->rb_start;
    }
    if (rp <= wh) {
        c = wh - rp;
        if (n < c) {
            c = n;
        }
        if (!c) {
            return (0);
        }
        r = c;
        do {
            *d++ = *rp++;
        } while (--c);
        p->rb_rhold = rp - 1;
        return (r);
    }
    /* The buffer is wrapped, which means that the data to be
       returned might span the end of the buffer. This case
       applies when the data wanted will not span the end of the buffer. */
    c = p->rb_end - rp;
    if (n <= c) {
        c = n;
        if (!c) {
            return (0);
        }
        r = c;
        do {
            *d++ = *rp++;
        } while (--c);
        p->rb_rhold = rp - 1;
        return (r);
    }
    /* The buffer is wrapped and so is the data that is to be
       returned. First, copy the data at the end of the buffer. */
    r = c;
    n -= r;
    do {
        *d++ = *rp++;
    } while (--c);
    /* There might be nothing left to copy if whold is at the
       start of the buffer. */
    rp = p->rb_start;
    c = wh - rp;
    if (n < c) {
        c = n;
    }
    if (c) {
        r += c;
        do {
            *d++ = *rp++;
        } while (--c);
        p->rb_rhold = rp - 1;
    } else {
        p->rb_rhold = p->rb_end - 1;
    }
    return (r);
}
/* This routine returns the number of data elements in the buffer. It
   is, in effect, a get routine, which means that you can't call it in
   a context where it might overlap with a get of the ring buffer.
   However, asynchronous puts may occur, which would increase the
   number of elements to above what this routine returns. */
RB_QUAL size_t
RB_NAME(gcount)(const RB_CONTROL *p)
{
    RB_TYPE *rp = p->rb_rhold + 1;
    RB_TYPE *wp = p->rb_whold;

    return (wp >= rp ? (wp - rp)
             : rp == p->rb_end ? wp - p->rb_start
                       : p->rb_size - (rp - wp));
}
/* This clears all data from a ring buffer. */
RB_QUAL void
RB_NAME(gclear)(RB_CONTROL *p)
{
    RB_TYPE *wp = p->rb_whold;
    p->rb_rhold = wp == p->rb_start ? p->rb_end - 1 : wp - 1;
}
#undef RB_PREFIX
#undef RB_TYPE
#undef RB_CONTROL
#undef RB_QUAL
#undef RB_GLUE1
#undef RB_GLUE
#undef RB_NAME

Copyright © 1994, Dr. Dobb's Journal

Terms of Service | Privacy Statement | Copyright © 2024 UBM Tech, All rights reserved.