Tape Devices & Java

Tape devices can support either fixed- or variable-length blocks and let an application change the block size before writing. Most modern devices also allow a combination of fixed- and variable-length blocks. The commonly available QIC-format devices, however, allow only fixed-length blocks of 512 bytes. The tape library I'll present in this article allows an application to manage the block size, but the application must do the right thing based on its needs. Usually, the various on-tape formats, such as the Microsoft Tape Format (MTF) or the System Independent Data Format (SIDF), handle the block size issue.

A single device usually supports different media types. In the case of a QIC-format device, only one type of media can be written, while many types of media can be read. In a more advanced device, such as a DAT or DLT, several media types can be written transparently.

Regardless of how devices handle different media types, almost all support different densities on a single media type. Density support varies greatly across operating systems and is often poorly implemented. Only compression can be modified on Windows NT. This limitation can make writing media that need to be read on another platform more difficult. On UNIX systems, different densities are usually available only by specifying a different path name when opening the tape device. Luckily, for most purposes, using the default density is the best choice.

Error handling is one of the more challenging issues in dealing with tape devices. Most UNIX platforms provide two different types of error handling: persistent and nonpersistent. Linux provides only nonpersistent error handling. Nonpersistent error handling is often referred to as "BSD behavior," while persistent error handling is referred to as "System V behavior." Windows NT systems provide only nonpersistent error handling.

Persistent error handling forces an application to clear error conditions, such as end-of-file, before further operations can continue. Nonpersistent error handling returns an error once, and then continues processing with the next I/O operation. Typically, this only matters in how the end-of-file condition is handled while reading. A nonpersistent error handling environment automatically skips the end-of-file mark whenever it is reached. Persistent error-handling environments will only skip the end-of-file mark after the error has been cleared or when explicitly told to space over it.

In this discussion, I focus on the basic operations of tape devices — reading, writing, rewinding, and skipping to the end of data. Additional support operations, such as manipulating the block size and allowing writes past the logical end-of-media, are provided to allow realistic use. All of the code has been developed with Java 2 compilers and run-time environments, but also works in Java 1 environments.

Designing the Interface

The java.net.Socket class provides an excellent example of creating an interface to a device that requires complex control mechanisms and simple I/O streams at the same time. I'll follow this lead for the BasicTapeDevice class, which provides all control mechanisms and two methods to retrieve an InputStream and an OutputStream for basic stream I/O. Persistent error handling is used in the interface regardless of the environment's choice for error handling.

The interface for the BasicTapeDevice contains the ability to construct, cleanup, configure, and perform stream I/O to the device. Example 1 shows the interface definition of this class. Construction of a BasicTapeDevice requires a single String argument specifying the path name of the tape device. I assume that the application or the user knows the correct device path name to use. The close method allows the application to cleanup and release the tape device resource on demand, rather than waiting for the Java garbage collector to do it using the finalize method.

The getInputStream and getOutputStream methods return an InputStream and OutputStream object, respectively. Since an application might retrieve I/O streams several times during its lifetime, both of the stream objects are kept internally in the BasicTapeDevice object. By keeping internal references to these objects, the garbage collector cannot clean these objects and cause an unexpected call to the close method.

The simplest tape movement commands are also provided in the interface. The abilities to rewind and to skip to the end of recorded media are critical for the basic operation of tape applications. I've left out a discussion of the more advanced movement commands, such as skipping between files or individual blocks on tape, because of their complexity.

The Implementation

Listing One shows the implementation of BasicTapeDevice. The object contains several properties for maintaining state, two inner classes to provide I/O streaming, and the declarations needed for native interfaces.

When the Java class loader loads the BasicTapeDevice class, a static block in the class is executed. The static block determines the appropriate JNI library to load based on the current operating-system name. The library name is in the form of Tape<osname>, such as TapeLinux, TapeSolaris, or TapeWinNT. Once the library has been loaded, the native method initFields is called. The initFields method determines the field identifiers of all of the member variables of the BasicTapeDevice object that the native library manipulates. The private fd member variable of the FileDescriptor object used is also extracted. Luckily, JNI methods can access private methods of objects if they explicitly specify the full class name of the object. Example 2 shows the initFields method that can be used by any native library.

The inner class TapeInputStream provides an object that calls the appropriate native method for reading on behalf of the device object. Because the class is an inner class, it has full access to the private member variables of BasicTapeDevice. The implementation of TapeInputStream is simplistic — it fulfills the contract specified by InputStream and calls the native method tapeRead when necessary to perform input. The tapeRead method must set the eof member variable whenever end-of-file is encountered. Once end-of-file is encountered, the stream will always return -1 for read operations until the end-of-file indicator has been cleared by the application by using the clearEOF method in BasicTapeDevice. Two consecutive end-of-file indicators signal that the end of recorded data has been reached.

Like TapeInputStream, TapeOutputStream provides writing functionality to the parent device object. TapeOutputStream delegates writes to the native method tapeWrite. All writes must be a multiple of the block size specified by the getBlockSize method of BasicTapeDevice. If the block size is zero, then writes of variable length are allowed. Some operating systems limit the maximum variable-length block to 64 KB, so you should determine if this will affect the tape format for your application. If a partial write to the device is encountered, TapeOutputStream ensures that all data is written to the tape media so the application never has to worry about it.

One caveat of writing to tape devices is the separation between logical and physical end-of-media. Most modern tape devices have a logical end-of-media indicator well before the physical end of tape. When a write detects this indicator, it informs the application to flush its buffers and write any end-of-media records required by the tape format. When logical end-of-media is found by the TapeOutputStream class, it will throw a LogicalEOMException to indicate this (see Listing Two). The application must then call the clearEOM method of BasicTapeDevice to allow further writes. Once physical end-of-media is reached, all writes will fail with an IOException.

Other than a few support methods, the remainder of the BasicTapeDevice class is the declaration of the following native methods: initFields, tapeOpen, tapeClose, tapeRead, tapeWrite, tapeGetBlockSize, tapeSetBlockSize, tapeRewind, and tapeSpaceEOD. All of these methods must be implemented in the native library for each platform supported.

Implementing the Linux Native Interface

The file TapeLinux.c (available electronically; see "Resource Center," page 5) is the native interface library for Linux. The implementation of this library is straightforward. The tapeOpen, tapeClose, tapeRead, and tapeWrite functions are thin wrappers around the Linux system calls. The tapeRead and tapeWrite methods set the eof and eom member variables, respectively, whenever they encounter end-of-file or logical end-of-media. Since Linux currently implements only nonpersistent error handling, I assume that file marks are skipped whenever end-of-file is encountered. Besides the common initFields method, the library also contains three utility functions: getFD, setFD, and throw. All of these methods could be ported to any of the UNIX variants.

The tapeGetBlockSize, tapeSetBlockSize, tapeRewind, and tapeSpaceEOD methods all use the Linux mtio API. In these methods, either the MTIOCGET or MTIOCTOP ioctl functions are used to call the appropriate behavior in the tape API. The mtio API is similar across all UNIX variants, yet can differ enough to make porting a chore.

Implementing the Windows NT Native Interface

TapeWinNT.c (also available electronically) shows the native interface library for Windows NT/2000. Windows 9x environments do not use the same tape API, and can't make use of the library.

The native interface library for NT is only slightly more complex than the Linux library. Although the function names and structures are all different, the Win32 tape API is remarkably similar to traditional UNIX implementations. Integer file descriptors are replaced with HANDLE variables, but these are still 32-bit quantities easily stored in the java.io.FileDescriptor class.

Windows NT has one annoyance that is not present on UNIX implementations. If a media change is detected at any time, the next tape API call will fail with the error code ERROR_MEDIA_CHANGED. While knowing that the media has been changed is useful, the Win32 API saves this information between applications. If an application is run for the first time and it just opened the tape device, it is possible to receive this error on the first tape operation performed. To solve this issue, I call the Win32 API function GetTapeStatus while in the native tapeOpen method. It is still possible to receive the ERROR_MEDIA_CHANGED status while performing other tape operations, but I consider that to be a genuine error that should be thrown as an exception to the application.

Performance

You can create efficient I/O applications in Java. However, unlike disk or network I/O, tape I/O is especially sensitive to a buffer underrun condition. If a streaming tape device is starved of data then the tape drive will stall. A stalled tape device must stop the write head, rewind slightly, and then continue forward until a safe write position is found once again--the infamous "shoe-shining" effect. Some operating systems and tape devices allow automatic padding of the buffer to prevent starvation. Adding support of this feature to your tape device code might be warranted in some applications.

Buffering is always critical for I/O applications, and especially for ones that write to tape devices. Without an adequate buffering mechanism in place, it's very easy to stall a tape device and ruin your performance. In my testing, I've found that a 1-MB buffer size provides the best throughput with the smallest memory usage. You may find that your application and hardware situation can live with as little as 64 KB of buffer, or require as much as 128 MB or more.

It is more complex to implement buffered writes to a tape device compared to other devices. The java.io.BufferedOutputStream class is useful, but does not guarantee any granularity on the actual output buffers written to the final output stream. If your tape device has a fixed buffer size, then you will immediately cause an error the first time BufferedOutputStream attempts to write a block that is not a multiple of that block size. To solve this issue, I've created a FixedBufferedOutputStream class (also available electronically). This class provides the same functionality as Java's BufferedOutputStream, but ensures that all writes to the tape device are a multiple of the specified block size.

Most applications do more with data than write it to tape. Filter streams are commonly used in Java applications to calculate data checksums. These additional filter streams can easily consume your CPU time if your Java run-time environment does not have a Just-In-Time (JIT) compiler installed. Without a JIT, a simple output stream chained to a DigestOutputStream can consume seven times the CPU time as an environment with a JIT. Figure 1 presents some comparisons of writing a 16-MB file to tape using a variety of JREs to demonstrate the importance of a good Java run-time environment.

Providing Full-Featured Tape Device Support

The BasicTapeDevice class presented here is only the beginning of a robust tape support library. Most applications that do more than dump data to tape will need the ability to space between the various files on tape. More advanced applications might require spacing between the records in a file and absolute logical and physical positioning. Controlling the various attributes of a tape device and its media, such as compression and density, are abilities most applications would find useful. Extending the BasicTapeDevice class described here is a straightforward task and can be customized based on your application's needs. The implementation described in this article, along with a more advanced library, is also available at http://jtape.sourceforge.net/.

DDJ

Listing One

/* BasicTapeDevice.java */
import java.io.*;
public class BasicTapeDevice {
    private FileDescriptor fd;
    private InputStream in;
    private OutputStream out;
    private boolean eof;
    private boolean eom;
    private boolean ignoreEOM;

    public BasicTapeDevice(String pathName) throws IOException {
        fd = new FileDescriptor();
        tapeOpen(pathName);
        in = new TapeInputStream();
        out = new TapeOutputStream();
        eof = false;
        eom = false;
        ignoreEOM = false;
    }
    public synchronized void close() throws IOException {
        if (fd != null) {
            try {
                if (fd.valid()) {
                    tapeClose();
                }
            } finally {
                fd = null;
            }
        }
    }
    public InputStream getInputStream() throws IOException {
        ensureOpen();
        return in;
    }
    public OutputStream getOutputStream() throws IOException {
        ensureOpen();
        return out;
    }
    public int getBlockSize() throws IOException {
        ensureOpen();
        return tapeGetBlockSize();
    }
    public void setBlockSize(int bs) throws IOException {
        ensureOpen();
        tapeSetBlockSize(bs);
    }
    public void rewind() throws IOException {
        ensureOpen();
        tapeRewind();
    }
    public void spaceEOD() throws IOException {
        ensureOpen();
        tapeSpaceEOD();
    }
    public void clearEOF() throws IOException {
        ensureOpen();
        if (eof) {
            eof = false;
            /* assume that the file mark has already been skipped */
        } else { 
            throw new IOException("not at end of file");
        }
    }
    public void clearEOM() throws IOException {
        ensureOpen();

        if (eom) {
            ignoreEOM = true;
        } else {
            throw new IOException("not at logical end of media");
        }
    }
    class TapeInputStream extends InputStream {
        private byte[] temp = new byte[1];
        public int read() throws IOException {
            int n = read(temp, 0, 1);
            if (n <= 0) {
                return -1;
            }
            return temp[0] & 0xff;
        }
        public int read(byte[] b, int off, int len) throws IOException {
            if (b == null) {
                throw new NullPointerException();
            }
            if (off < 0 || len < 0 || off+len > b.length) {
                throw new IndexOutOfBoundsException();
            }
            if (len == 0) {
                return 0;
            }
            if (eof) {
                return -1;
            }
            ensureOpen();
            int n = tapeRead(b, off, len);
            if (n <= 0) {
                return -1;
            }
            return n;
        }
        public long skip(long numbytes) throws IOException {
            return 0;
        }
        public void close() throws IOException {
            BasicTapeDevice.this.close();
        }
    }
    class TapeOutputStream extends OutputStream {
        private byte[] temp = new byte[1];

        public void write(int b) throws IOException {
            temp[0] = (byte) b;
            write(temp, 0, 1);
        }
        public void write(byte[] b) throws IOException {
            write(b, 0, b.length);
        }
        public void write(byte[] b, int off, int len) throws IOException {
            if (b == null) {
                throw new NullPointerException();

            }
            if (off < 0 || len < 0 || off+len > b.length) {
                throw new IndexOutOfBoundsException();
            }
            if (eom && !ignoreEOM) {
                throw new LogicalEOMException("logical end-of-media");
            }
            int n = tapeWrite(b, off, len);
            while (n < len) {
                n += tapeWrite(b, off + n, len - n);
            }
        }
        public void close() throws IOException {
            BasicTapeDevice.this.close();
        }
    }
    protected void finalize() {
        try {
            close();
        } catch (IOException ex) {
        }
    }
    private void ensureOpen() throws IOException {
        if (fd == null || !fd.valid()) {
            throw new IOException("tape device is not open");
        }
    }
    private static native void initFields();
    private native void tapeOpen(String pathName) throws IOException;
    private native void tapeClose() throws IOException;
    private native int tapeRead(byte[] b, int off, int len) throws IOException;
    private native int tapeWrite(byte[] b, int off, int len) throws IOException;
    private native int tapeGetBlockSize() throws IOException;
    private native void tapeSetBlockSize(int bs) throws IOException;
    private native void tapeRewind() throws IOException;
    private native void tapeSpaceEOD() throws IOException;

    /* load the JNI library specific for this platform */
    static {
        StringBuffer buf = new StringBuffer("Tape");
        String osName = System.getProperty("os.name");
        if (osName.equals("Windows NT") || osName.equals("Windows 2000")) {
            buf.append("WinNT");
        } else {
            buf.append(osName);
        }
        System.loadLibrary(buf.toString());
        initFields();
    }
}

Back to Article

Listing Two

/* LogicalEOMException.java */
import java.io.IOException;
public class LogicalEOMException extends IOException {
    public LogicalEOMException() {
        super();
    }
    public LogicalEOMException(String s) {
        super(s);
    }
}

Back to Article