Aaron, who is a staff engineer at Intel where he develops video teleconferencing systems, is also the coauthor of Win32 Multithreaded Programming (O'Reilly & Associates, 1997). He can be contacted at [email protected], or http://www.netcom.com/~alcohen/.
The Java API defines an abstract imaging model that can be used to display and manipulate both static images and sequences of images. The imaging model is defined in terms of the interactions between an abstract class, Image, and three interfaces: ImageProducer, ImageConsumer, and ImageObserver. These interfaces and their support classes are defined in the java.awt.image package.
The Image abstract class represents a platform-independent displayable image. Images can be created from image files loaded from the local file system or over a network using getImage(URL url) or getImage(URL url, String name). These functions will only succeed with files using one of the supported file formats. Currently, JPEG and GIF file formats are universally supported.
Images can also be created with a call to the createImage() function of an AWT Component or available Toolkit object. There are several forms of the createImage() function:
- Image createImage(ImageProducer source) creates an image from an object implementing the ImageProducer interface.
- Image createImage(int width, int height) creates a blank image of a given size that can be drawn on using a Graphics object.
- Image createImage(byte[] imagedata) creates an image from JPEG or GIF format data stored in an array. Only available in Java 1.1 or later.
- Image createImage(byte[] imagedata, int offset, int length) creates an image from JPEG or GIF format data stored in length bytes of an array starting at the given offset. Only available in Java 1.1 or later.
One of the createImage() methods lets you create an Image from an ImageProducer. You can retrieve the ImageProducer associated with an image by using the ImageProducer getSource() method of the Image class.
Drawing onto an image is accomplished by using the methods of the java.awt.Graphics class. Create a Graphics object associated with the image by using the getGraphics() method. Anyone who has programmed in Java even a little has written the code to draw an image into a existing Graphics context by calling drawImage() inside a Component's paint() handler, so I will not cover that in more detail here.
An image also has properties such as width and height, which can be retrieved with methods defined in the Image base class. Java has the built-in capability to download and display images in the background. Because of this it is possible to call methods on an Image object before the necessary data is available. These methods require an ImageObserver as a parameter. For example, the getWidth() and getHeight() methods each require an object implementing the ImageObserver interface as a parameter. When the requested information cannot be returned immediately, these functions return -1.
The ImageObserver interface defines one method, imageUpdate (see Example 1). When an operation on an Image object cannot be completed immediately because the data is not yet available, a thread is created, which loads the data in the background. As the data is loaded, imageUpdate() of all registered image observers is called notifying the observer of progress. The img parameter refers to the image for which there is new information. The infoflags parameter is a set of flags that define what type of information is now available. The meaning of the rest of the parameters is dependent upon the content of infoflags. Returning true from imageUpdate() requests further information on the image. The function should return false if the ImageObserver is not interested in any further callbacks for this image. All AWT components implement the ImageObserver interface. The default behavior repaints the image when additional pixel data arrives. You usually do not have to override the default component implementation.
So where does the image data come from? Each Image is associated with an ImageProducer, which can be retrieved by calling the getSource() method on the Image object. The ImageProducer is responsible for delivering the image data upon request to ImageConsumer objects. An ImageProducer object makes calls on the methods of the ImageConsumer interface to inform the ImageConsumer of the image type, size, and pixel data. Both the ImageProducer and ImageConsumer interfaces are defined in the java.awt.image package. The relationship between the classes in the Java imaging model looks like Figure 1.
All of the methods of the ImageProducer class have to do with ImageConsumer objects registering as consumers with the producer object and requesting data. These methods are fairly straightforward:
- void addConsumer(ImageConsumer ic) adds a consumer to the ImageProducer. The consumer will be delivered image data the next time new data is available.
- boolean isConsumer(ImageConsumer ic) determines whether a consumer is currently registered with the ImageProducer.
- void removeConsumer(ImageConsumer ic) removes the given consumer from the producer's list of consumers.
- void requestTopDownLeftRightResend(ImageConsumer ic) requests the producer to resend its image data in top-down, left-to-right order. The producer may choose not to honor this request.
- void startProduction(ImageConsumer ic) registers a consumer with the producer and deliver the current image data to the consumer as soon as possible.
More interesting than the methods of the ImageProducer interface are the methods of the ImageConsumer interface, which an ImageProducer calls to deliver the image type, size, and pixel data to the consumer object. These methods can be described as follows:
- void setDimensions( int width, int height) is called to notify the consumer of the image's size. This notification will be called before the first call to setPixels().
- void setColorModel(ColorModel cm) is called with a ColorModel parameter, which will be the color model of most, but not necessarily all, of the pixels delivered to the setPixels() method. This notification is optional. If called, it will be invoked before the first call to setPixels().
- void setHints(int hintflags) notifies the consumer of the order in which pixels will arrive, which the filter may use to optimize some operations. For example, the COMPLETESCANLINES flag will be set if each call to setPixels() will deliver entire unbroken scan lines (rows) of pixels, and the TOPDOWNLEFTRIGHT will be set if the pixels will be delivered in top to bottom, left to right order. This notification is optional. If called it will be before the first call to setPixels().
- void setProperties(Hashtable props) is used by the consumer to add some programmer-defined information to the image stream. This notification will be called before the first call to setPixels().
- void setPixels() passes the image pixels to the consumer by calling this method one or more times. Each call will deliver a rectangle of image pixels to the consumer. The layout of the pixels delivered will correspond to any information sent in an earlier call to setHints(). There are two versions of this method. One which receives 32-bit pixel data and another which receives eight-bit pixel data.
- void imageComplete(int status) is called with a status parameter to inform the consumer that a complete static image has been delivered, one frame of a multiframe image has been delivered, or an error has occurred. This notification will be called after the last call to setPixels().
The setPixels() method requires more explanation. The full function prototypes of the two setPixels() functions look like Example 2. A portion of the image is delivered to an ImageConsumer with each call to setPixels(), which may be as large as the entire image, or as small as one pixel. The portion delivered in each call to setPixels() is a rectangle whose upper-left corner is at x,y and has a width of w and height of h. The data is passed in the array object pixels, which has valid data starting at position offset, and scansize elements between each row. What all this boils down to is that the sample in the pixels buffer which corresponds to the image pixel(m,n) is at array index offset + (n-y)*scansize + (m-x). The only valid pixel array sample values are those that lie in the rectangle described by w, h, offset, and scansize. The producer will continue to call setPixels() until all of the image pixels have been delivered, or an error occurs.
ColorModels
The delivered image data may consist of 32- or 8-bit samples. A ColorModel is passed along with the image data in each call to setPixels() so that consumers can retrieve and manipulate the color information in the samples. The data in the pixels buffer is defined in terms of the ColorModel passed with the pixels. The ColorModel passed in the setColorModel call is only a hint to the consumer that the majority of pixels will use that model; you still need to check the ColorModel passed in each call to setPixels().
The ColorModel abstract class defines a set of functions that an ImageConsumer uses to map a pixel value to red, green, blue, and alpha values. The functions int getRed(int pixel), int getGreen(int pixel), int getBlue( int pixel), and int getAlpha(int pixel) return the components for the given pixel value. Each color component can take on a value between 0 and 255, inclusive, with 0 being the minimum of a component and 255 being the maximum. The alpha component quantifies the transparency of the pixel and ranges from 0, which lets an underlying image show through completely, to 255, which is opaque.
All ColorModel objects also define a function, int getRGB(int pixel), which returns the color of a given pixel value in terms of the default color model, which is known as the RGBDefault color model. An instance of the RGBDefault color model is returned by calling the ColorModel static function, getRGBdefault(). The RGBDefault color model uses 32-bit data samples and allocates eight bits each for red, green, blue, and alpha with a bit layout of 0xAARRGGBB.
A ColorModel object that allocates a portion of the total bits in a sample to each color component is known as a DirectColorModel. A color model which uses the pixel value as an index into a palette of colors is called an IndexedColorModel. DirectColorModel objects implement functions that allow you to determine which bits of the pixel are allocated to each component. These are int getAlphaMask(), int getRedMask(), int getGreenMask(), and int getBlueMask(). The RGBDefault color model is a DirectColorModel.
In general, eight-bit samples correspond to palette-based color models and 32-bit samples correspond to direct color models such as 32-bit RGB. However, a ColorModel is a flexible representation that can be used to convert between pixel values and red, green, blue, and alpha components in any way that may be appropriate. For example, a topological map image could be stored as an array of ints with each sample representing the elevation of its location. A color model could then be created, which mapped the elevation to some standard topological map colors. This color model would be an indexed color model, however since the samples are stored as integers and not bytes, it could have many more than 256 entries and thus very fine shading of color to represent changes in geography.
Image Processing
Image processing with Java is accomplished using image filters, which are objects derived from the ImageFilter base class. The ImageFilter class implements the ImageConsumer interface to receive the pixel samples for the image to be processed. An ImageFilter has a protected member variable, consumer, which refers to the ImageConsumer downstream of the image filter. Essentially, an ImageFilter is an ImageConsumer that manipulates the incoming pixel data and forwards the results to another ImageConsumer object.
As the image filter processes the samples received in its setPixels() function, the filter sends the processed samples downstream by passing the samples to the setPixels() function of its consumer. The image filter can pass the pixels to its consumer as it receives and processes the samples, or the filter can save the raw or processed pixels in a buffer and send them all at once to the consumer when the filter's imageComplete() method is called. The default implementation in the ImageFilter base class is a null filter. For each call to setPixels(), the base class simply calls the corresponding function in the consumer without modifying the samples. The rest of the methods inherited from the ImageConsumer method are similarly forwarded to the downstream consumer.
The Java API includes a few built-in filters to handle common tasks. The ReplicateScaleFilter will stretch or shrink an image to a given size by dropping or duplicating samples. The AreaAveragingScale filter resizes an image using a bilinear interpolation algorithm. The CropImageFilter extracts a rectangular subimage from the original image.
The FilteredImageSource class
Given that you have an instance of an ImageFilter object, how do you apply it to an image? This is where the FilteredImageSource class comes in. An instance of the FilteredImageSource class is an ImageProducer that creates a new image source from a given ImageProducer and an ImageFilter. The resulting image source produces an image which has been processed by the image filter.
Example 3 should make this clear. First, an image is loaded from the local file system using the default toolkit. Then an AreaAveragingScaleFilter is created which will scale images to half the size of the original loaded image. Next, a FilteredImageSource object is created from the scaling filter and the ImageProducer of the original image. Finally, an Image object is created from the new FilteredImageSource. Basically the FilteredImageSource constructor adds an image consumer/source pair in between the original ImageProducer and any future consumers. This process can be repeated any number of times to create an image filter chain or "pipeline;" see Figure 2.
It is important to realize that the image data is not actually filtered until it is requested by a consumer. Typically this is done implicitly when the image is displayed by calling drawImage(), although any ImageConsumer can start the pipeline flowing by calling startProduction() on the ImageProducer at the end of the pipeline. Notice that downstream FilteredImageSource objects forward delivery requests upstream. Eventually the request arrives at an image source that has image data ready to deliver and this ImageProducer object invokes methods on its registered consumers to push image data through the pipeline.
Image Filters
There are an infinite variety of filtering operations that can be applied to images in order to enhance or modify them. For the purposes of image processing with Java, filters can be classified according to the information required in order to process each sample. Here, we will classify image filters as point operations, geometric operations, or neighborhood filters.
Point Filters
The simplest filters process each sample independent of the surrounding samples. These filters can be simple color modifications, or can be dependent upon the pixel's location in the image. In any case, a point operation filter only has knowledge of a single pixel at a time. Contrast enhancement, color inversion, and dither filters, among others, can be written this way. This type of filter is so common that the Java API includes a special base class, RGBImageFilter, which makes implementing point operation filters very easy.
The RGBImageFilter class implements all the necessary ImageFilter machinery to filter an image pixel by pixel. One abstract function, int filterRGB(int x, int y, int rgb), needs to be implemented. This function is called for each pixel in the image and passed the pixel location as well as the 32-bit default RGB color model sample value. The filterRGB() function in the derived class should return the processed pixel value using the 32-bit default RGB color model.
Listing One is the implementation of the GreyOutImageFilter, which demonstrates the use of the RGBImageFilter class. This filter simply replaces half of the original image pixels with grey using a checkerboard pattern. This is an effect which you may want to use to make an icon or button appear inactive. Notice how short the implementation is. All the hard work has been done for us in the base class.
GreyOutImageFilter also demonstrates the proper way to handle the canFilterIndexColorModel protected member variable. For indexed color images, RGBImageFilter attempts to be more efficient by filtering only the color table, not the entire image. If the results of the filter are only dependent upon the color of each filter, the derived class should set this member to true. If the results depend upon the pixel location, as they do in the GreyOutImageFilter, then canFilterIndexColorModel must be set to false.
Geometric Filters
Another kind of filter moves pixels from one location in an image to another. These are known as geometric transformations. This is a very broad category and includes many types of filters including horizontal and vertical flipping, mirroring, and rotations. The Java API does not provide a specialized base class for implementing geometric operations like it does for point operations. We need to derive geometric transform filters from ImageFilter.
Listing Two implements a filter that rotates an image 90 degrees clockwise. An m×n pixel image is transformed into an n×m pixel image, with the pixel value occupying the original upper-left now occupying the upper-right corner.
The code is longer than that of the previous filter because we need to override several functions of the ImageFilter base class. To properly rotate the image, we need to save the width and height passed into setDimensions(), and pass appropriate hints to the base class setHints(). Notice that after processing these values, you invoke the superclass implementation. For all of the functions in the ImageFilter class except setPixels(), you should invoke the superclass implementation to properly initialize the base class, passing either the original parameters or values modified appropriately for your implementation. Calling the superclass also ensures that the information is passed along to downstream consumers.
Image rotation is accomplished by taking each row of the rectangle of pixels received in setPixels() and passing it to the consumer as a column. We do some coordinate transformation to ensure that the pixels wind up in the correct location. Since the filter typically passes a column of pixels to the consumer, we clear the COMPLETESCANLINES and TOPDOWNLEFTRIGHT flags before we pass the hints along.
Neighborhood Filters
The most complex kind of filter that we will deal with here is the neighborhood filter. Each output pixel is a function of several nearby input pixels. Image sharpeners, smoothers, high-pass filters, and general convolution all fit into this category. Usually the pixel neighborhood is a small window of pixels centered on the pixel being processed. Common window sizes are 3×3 and 5×5. There are also "separable" image filters which process an image, a row, and then a column at a time. In this case, the neighborhood is composed of nearby pixels in the same row or column as the target pixel.
The implementor of a neighborhood filter must decide how to handle the image boundaries where there is not a full window of pixels to process. Common strategies are to leave the boundary pixels unprocessed, or to fill in the missing window samples by reflecting the image over the edge. The primary reason why implementing neighborhood filters with the ImageFilter class becomes complicated is that each call to setPixels() may pass only a small subrectangle of the image. The pixels at the edges of the subrectangles will not be delivered along with the neighborhood pixels required to process them. The easiest strategy to work around this problem is simply to save the image pixels in a private buffer as they are delivered to setPixels and process the image all at once when imageComplete() is called. The fully processed image can then be delivered with a single call to the consumer's setPixels() method.
The SharpenImageFilter (available electronically; see "Resource Center," page 3) uses this method to implement a 3×3 sharpening filter. Notice that the image to be processed is separated into color components and stored in four arrays named red, green, blue, and alpha. This allows the filter to sharpen both direct and indexed color model images.
Chaining Filters Together
Applying an ImageFilter to a given Image with the FilteredImageSource creates an image pipeline. If another filter is then applied to the resulting image, the new FilteredImageSource is added to the front of the pipeline. Each time the image is processed it is pushed all the way through the pipeline. If an image is filtered, and then displayed, applying another filter to the image results in repeating processing that has already been done as the original image source is sent all the way from the beginning of the pipe, through the previous filters and finally through the newest filter. Therefore when applying n filters in sequence to an image and displaying each intermediate result, the image is actually processed n(n + 1)/2 times.
The Java image processing API is flexible and written this way to handle changing source images. For example, if the original image source were video and not a still image, sending each frame though the entire pipeline would be the proper thing to do. However, it can be very inefficient when applying a series of filters to a still image in an interactive application.
To overcome this unnecessary overhead, the BufferingFilteredImageSource class (available electronically) can be used just like the FilteredImageSource. For still images, it buffers the result of applying its ImageFilter to the original image. When a downstream filter requests image data the BufferingFilteredImageSource delivers the samples from its internal buffer instead of forwarding the request upstream. For animated GIFs and other image sequences, it functions identically to the FilteredImageSource. Using this class makes applying a sequence of filters, one at a time, much more efficient since each filter is applied to an image only once.
To keep the implementation as simple as possible, the BufferingFilteredImageSource internally uses the FilteredImageSource class to actually perform the image filtering. An instance of MemoryImageSource is created and stored in the source member and functions as the ImageProducer for the buffered, filtered image. If the image is animated, then no buffering is done and the FilteredImageSource instance is used as the source instead. BufferingFilteredImageSource implements the ImageProducer interface simply by forwarding the calls to the source member object.
The only complexity in the implementation is the BfisInternalConsumer class that the BufferingFilteredImageSource uses to save the processed image in a buffer. The buffer is then used to create the MemoryImageSource. Since the FilteredImageSource object may deliver the filtered image synchronously or asynchronously, we need to allow for both cases. This is done by synchronizing the BufferingFilteredImageSource constructor with the imageComplete() function of the BfisInternalConsumer using the object and the done flag.
The JIPTestApplet
To demonstrate the filters and classes presented in this article, use the test applet, JIPTestApplet (also available electronically), along with JIPTestApplet.html, which is an HTML page used to demonstrate the applet. The applet can be configured to display and process up to ten images. The filters given in this article can be applied repeatedly using either the FilteredImageSource or BufferingFilteredImageSource. Applying several filters in succession to a still image will demonstrate the efficiency gained by using the BufferingFilteredImageSource. Using the FilteredImageSource, successive filter applications take longer and longer. Using the BufferingFilteredImageSource, each invocation takes the same amount of time. You can also use the test applet to explore the results of applying several filters to an image, for example, the GreyOutFilter followed by the SharpenImageFilter.
Conclusion
The Java image-processing model is powerful, flexible, and expandable, and can be used to create complex image-processing applications. Image processing is a computationally intensive activity. With a JIT compiler, Java performance is more than adequate for basic processing of moderate sized images. For interpreted Java virtual machines or complex multipass computations, the processing should be kept to small images.
DDJ
Listing One
import java.awt.Color;import java.awt.image.*;
</p>
public class GreyOutImageFilter extends RGBImageFilter {
// save the int value of the grey color...
protected int greyOutValue = Color.gray.getRGB();
public GreyOutImageFilter() {
// this filter is position dependent, so we can't filter
// by just changing the color table...
canFilterIndexColorModel = false;
}
public int filterRGB( int x, int y, int rgb) {
// set every other pixel to grey using a checkerboard pattern...
if (((x ^ y) & 1) == 0) {
return greyOutValue;
}
else {
return rgb;
}
}
}
</p>
Listing Two
import java.awt.image.*;
</p>
public class RotateClockwiseImageFilter extends ImageFilter {
protected int srcwidth, srcheight;
protected int destwidth, destheight;
public void setDimensions( int width, int height) {
// source height becomes destination width and vice-versa...
this.destwidth = height;
this.destheight = width;
// tell the consumer the size of the image that we will be sending...
super.setDimensions( this.destwidth, this.destheight);
}
public void setHints( int hints) {
// because this filter delivers pixels a scan column at a time,
// we need to clear COMPLETESCANLINES and TOPDOWNLEFTRIGHT hint
// bits and set the RANDOMPIXELORDER bit...
hints = (hints & (~COMPLETESCANLINES) &
(~TOPDOWNLEFTRIGHT)) | RANDOMPIXELORDER;
super.setHints(hints);
}
public void setPixels( int x, int y, int w, int h, ColorModel cm,
int[] pixels, int offset, int scansize) {
// start is the offset into buffer of first pixel of current source
// row, which will become the top pixel in destination column...
int start = offset;
// destx is zero-indexed destination column of source pixel row...
int destx = destwidth - y - 1;
// send the pixels on to the consumer, each row of the source image
// becomes a 1-pixel wide column of pixels in destination image...
for (int j = 0; j < h; j++) {
consumer.setPixels( destx, x, 1, w, cm, pixels, start, 1);
start += scansize;
destx--;
}
}
public void setPixels( int x, int y, int w, int h, ColorModel cm,
byte[] pixels, int offset, int scansize) {
// start is the offset into the buffer of the first pixel of current
// source row, which will become top pixel in destination column...
int start = offset;
// destx is zero-indexed destination column of source pixel row...
int destx = destwidth - y - 1;
// send the pixels on to the consumer, each row of the source image
// becomes a 1-pixel wide column of pixels in destination image...
for (int j = 0; j < h; j++) {
consumer.setPixels( destx, x, 1, w, cm, pixels, start, 1);
start += scansize;
destx--;
}
}
}
Copyright © 1998, Dr. Dobb's Journal
