Jeff Perry is a computer programmer at the Center for Perceptual Systems at the University of Texas. He can be contacted at jsp@mail .utexas.edu.
Matlab is a flexible computing environment for creating, exploring, and visualizing data and algorithms. It is available for several platforms including Windows, Linux, Solaris, Macintosh, AIX, and more. The Matlab language, on the other hand, is a high-level language that lets you perform complex operations with a few lines of code. Of course, the trade-off for this power is that some operations can suffer from poor performance. Consequently, Matlab lets you extend its functionality by calling externally compiled executables through an external interface API. In this article, I show how to create compiled C++ Matlab external interface routines—called "MEX-files"—using this API, and compare the performance of the Matlab C++ external interface routines to functionally equivalent Matlab routines. To do so, I use MEX-files to enhance digital images.
The lab I work in uses Matlab to conduct psychophysics experiments, and we use Matlab extensions in several development projects. For instance, we often use MEX-files to meet real-time requirements simulating visual fields with varying degrees of vision loss to determine the affect on visual search performance. It would be impossible to do this in real time purely in Matlab, without MEX-files.
The Matlab Environment and Language
The Matlab environment is a graphical desktop that contains help, a directory browser, command history, command interpreter, and other tools designed to manage data and files associated with Matlab. The Matlab language is an interpreted language and provides mathematical expressions that consist of variables, numbers, operators, and functions. Variables in Matlab do not require type declarations nor do they require user memory management, so Matlab language scripts and functions are typically lean and powerful.
I won't go into details about the Matlab language and environment here, but instead, I'll focus on the API and C++ programming aspects of Matlab extensions. For more information on Matlab, check with The MathWorks (http://www.mathworks.com/). I should point out that variables in Matlab typically do not contain single values, but instead contain several values stored as arrays. Furthermore, Matlab functions typically do not operate on single values, but usually will operate on whole arrays at once.
Color-Space Conversion
Suppose you want to convert an RGB triplet to YUV color-space. (Think of YUV color-space as a way to specify a particular color using three values in the same way that you might specify hue, saturation, and brightness, or red, green, and blue.) Image-processing experts might argue that the color-space used here is not really YUV but YPbPr (or YCbCr, YIQ, EBU, and so on). Technically, the transform I use is the one specified by SMPTE-240M Y'PbPr. But whatever you call it, the important thing about this color-space is that it separates the brightness information (the Y value) from the color information (the U and V values), thus letting you manipulate color information independently of brightness information. Converting between different color-spaces is therefore a common task in digital image processing in general, and in my field of vision research in particular.
Listing 1 lets you convert from RGB to YUV in Matlab. Matlab functions are saved as so-called "M-files" containing interpreted Matlab code. Since the filename of the Matlab M-file must have the same name as the function that it contains, I call the function mrgb2yuv (the prepended "m" distinguishes it from C++ versions of the function), and I name the file mrgb2yuv.m. You can convert back to RGB with Listing 2, myuv2rgb, stored in the M-file myuv2rgb.m.
Making It Faster
Listings 1 and 2 work, but what if you were doing real-time processing on large images and wanted to ensure that your conversions were as fast as possible? Say you were generating real-time virtual worlds that interact with users, and were testing their ability to navigate through the world based on the amount of color information in the world. In this case, it might be impossible to generate the video sequences prior to the experiment and save them to disk. You may, therefore, want to speed up the color-space conversion by writing it in C++.
MEX-Files
To write C++ MEX-files, you use the Matlab external interface API that lets you access and manipulate the data passed to the external routine from Matlab. MEX-files are compiled and linked as DLLs under Windows, and as shared libraries under Linux. The name of the MEX-file needs to have the same name as the function that it implements, so, in this case, you need one file called "rgb2yuv.cpp" and another called "yuv2rgb.cpp". After the MEX-file is compiled and linked, it has the same name as the C++ file except with a .dll, .so, or some other filename extension, depending on the operating system.
All C++ MEX-files contain an entry point that looks like this:
void mexFunction (int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
prhs contains the data passed to the MEX-file, and plhs contains the data returned from the MEX-file. These variables are C arrays of pointers to Matlab array structures. To access the data contained within the prhs and plhs arrays, you must call Matlab external interface API routines, indexing the C arrays and using them as parameters.
The variables nlhs and nrhs specify the number of Matlab arrays passed to/from the routine. For example, if a function accepts three arrays as parameters, nrhs is set equal to 3. Don't confuse this value with the number of elements within the Matlab arrays passed to the function. To get the number of elements contained within each Matlab array, you have to use a Matlab external interface API function.
Checking Inputs/Outputs
The rgb2yuv routine contains one input, an N×3 or N×M×3 array of RGB values, and returns one array output, an N×3 or N×M×3 array of YUV values. Listing 3 presents the C++ rgb2yuv function.
The first thing the rgb2yuv routine does is to check the number of inputs and outputs by examining the variables nlhs and nrhs.
// Check inputs
if (nrhs != 1)
mexErrMsgTxt ("This function requires 1 input.");
// Check outputs
if (nlhs > 1)
mexErrMsgTxt ("Too many return values were specified.");
The variable nrhs should be equal to 1 on entry to indicate that one array is being converted, and the variable nlhs should be equal to 0 or 1. Zero outputs are specified if users do not assign the output to a variable. However, when this occurs, the output is automatically stored in the special Matlab desktop variable called ans. So, whether nlhs is equal to either 0 or 1, you allocate storage and perform the conversion in the same way.
The reason for the absence of return statements in the if clauses of this routine is that the mexErrMsgTxt function does not return to the caller, but instead, terminates execution upon exit.
After checking the number of inputs and outputs, the routine checks the type of the input passed to it. This is done by calling the mxIsDouble function to ensure that the inputs are all doubles.
// Check input types
if (!mxIsDouble (prhs[0]) || mxIsComplex (prhs[0]))
mexErrMsgTxt ("The input must be a noncomplex double.\n");
Floating-point variables in Matlab are almost always double precision; single-precision variables are rarely used. The doubles passed to the routine specify red, green, and blue color values on a scale of 0.0 to 1.0 or 0 to 255. The routine must also check to make sure that the values passed are not complex numbers. A complex number is a double, but a double is not necessarily a complex, so mxIsComplex must be called to make this check.
The mxGetNumberOfDimensions function is used to get the number of dimensions in the input, then mxGetDimensions gets the actual dimensions of the input. A check is then made to make sure that the input is either two- or three-dimensional and that the last dimension is three—one dimension for each RGB plane.
// Get input dimensions
int ndims = mxGetNumberOfDimensions (prhs[0]);
const int *dims = mxGetDimensions (prhs[0]);
if ((ndims != 2 && ndims != 3) || dims[ndims-1] != 3)
mexErrMsgTxt ("The input must be either N X 3 or M X N X 3");
When the input is two-dimensional, it contains N triplets of RGB values. When the input is three-dimensional, it contains M×N triplets of RGB values.
There are two more important things to note about the input to the rgb2yuv routine. One is that the input dimensions in Matlab specify the number of rows first and the number of columns second. This is what you are used to when indexing multidimensional arrays in C/C++. However, the values in a Matlab array are stored in columns rather than in rows. This is the opposite of what you are used to in C/C++, where memory addresses advance across the rows of a two-dimensional image.
The other thing to note about the input to the routine is that Matlab typically stores color images such that each RGB triplet is stored in a separate color plane; whereas many color graphics file formats, such as Windows BMP or Linux PNM, for example, store RGB triplets as interlaced triplets. This doesn't affect the code much—you just need to be aware of it when you index the RGB values. However, for large images it (unfortunately) affects the processing speed because it does not allow the memory cache to work as effectively as when the RGB values are interlaced.
Allocating Outputs
Once you have checked the inputs and outputs and made sure that they are the correct type and dimension, you need to allocate space to store the output values. This is done with a call to mxCreateNumericArray in which you pass the array dimensions and type.
// Allocate the outputs plhs[0] = mxCreateNumericArray (ndims, dims, mxDOUBLE_CLASS, mxREAL);
mxDOUBLE_CLASS specifies the type, and mxREAL distinguishes the double real from the double complex.
RGB/YUV Conversion
To do the actual conversion, use the same formulas used in the mrgb2yuv function. First, however, you must get pointers to the input and output data:
// Get pointers to inputs and outputs double *yuv = mxGetPr (plhs[0]); double *rgb = mxGetPr (prhs[0]);
and calculate how many triplets are contained in the input.
// Determine how many iterations we need int total = mxGetNumberOfElements (prhs[0]) / 3;
You can then iterate through the input, converting to YUV as you go.
// Do the conversion
for (int n = 0; n < total; ++n)
{
yuv[n+0*total] = 0.2122*rgb[n] +
0.7013*rgb[n+total] + 0.0865*rgb[n+2*total];
yuv[n+1*total] = -0.1162*rgb[n] -
0.3838*rgb[n+total] + 0.5000*rgb[n+2*total];
yuv[n+2*total] = 0.5000*rgb[n] -
0.4451*rgb[n+total] - 0.0549*rgb[n+2*total];
}
Compiling and Testing
Compiling Matlab extensions such as rgb2yuv.cpp is done from the Matlab command line. Before compiling, however, you must setup mex, the Matlab external interface compiler by invoking the Matlab external interface compiler with a setup parameter.
>> mex -setup
When you do this, Matlab automatically searches the system for installed compilers and lets you choose which one it will use to compile MEX-files. For example, under Windows the choices may look something like:
Would you like mex to locate installed compilers [y]/n? Select a compiler: [1] Compaq Visual Fortran version 6.6 in C:\Program Files\ Microsoft Visual Studio [2] Lcc C version 2.4 in C:\MATLAB_SV13\sys\lcc [3] Microsoft Visual C/C++ version 7.0 in C:\Program Files\ Microsoft Visual Studio .NET [4] Microsoft Visual C/C++ version 6.0 in C:\Program Files\ Microsoft Visual Studio [0] None
After setting up the compiler, the Matlab extension may be compiled using the filename as the argument.
>> mex rgb2yuv.cpp
If the compiler doesn't encounter problems, the extension is stored using the same base name as the source file. On Windows, the output will be rgb2yuv.dll. When you then call rgb2yuv from the Matlab command line, it looks in the current directory and path for either a Matlab rgb2yuv.m function or a Matlab rgb2yuv extension, which it finds in this particular case at rgb2yuv.dll.
For example, you can create 100 random RGB triplets in Matlab with a call to the rand function.
>> rgb=rand(100,3);
When you call rgb2yuv, the mexFunction routine will be called in rgb2yuv.dll with nrhs set to 1 and prhs[0] set to point to an mxArray structure containing 100 random triplets.
>> yuv=rgb2yuv(rgb);
Listing 4, yuv2rgb.cpp, is almost identical to rgb2yuv.cpp in that it checks inputs/outputs, allocates storage for the output, and does a conversion. The only difference between yuv2rgb.cpp and rgb2yuv.cpp is that yuv2rgb.cpp converts YUV values to RGB values using the formulas from myuv2rgb.m.
Also like rgb2yuv.cpp, the yuv2rgb.cpp Matlab external routine is compiled with a simple call to the Matlab external interface routine compiler, which creates the Matlab callable external routine; with Windows, this is yuv2rgb.dll.
>> mex yuv2rgb.cpp
The extension may now be called just as before:
>> newrgb=yuv2rgb(yuv);
Here, newrgb contains the same random RGB triplets stored in rgb, with perhaps some rounding errors.
Timing and Comparison
Along with the conversion routines, mrgb2yuv.m and myuv2rgb.m, and their Matlab external counterparts, rgb2yuv.cpp and yuv2rgb.cpp, three test/timing/examples functions are available at http:// www.cuj.com/code/. These files are yuvtest.m, yuvtime.m, and yuvexample.m.
The first file, yuvtest.m, is used to verify that the routines work. It tests both the Matlab and Matlab C++ external versions of the routines. The second file, yuvtime.m, times and compares the Matlab versions to the Matlab C++ external interface versions. The following output shows the performance difference between the two versions when run on a 2.2-GHz Pentium 4 running under Windows 2000.
>> yuvtime Timing rgb/yuv/rgb conversions... Elapsed time: 4.600000e-002 seconds, 21.739130 Hz Elapsed time: 3.910000e-001 seconds, 2.557545 Hz
As you can see, the C++ conversion is over eight times faster than the Matlab conversion. Of course, these numbers do not include the actual image processing, but if a good portion of your runtime is spent performing RGB/YUV color-space conversions, then this performance gain might mean the difference between being able to perform in real time and not being able to perform in real time.
For example, this timing routine is using a 640×480, 24-bit image as input. In this case, 22 Hz might be fast enough to create smooth video sequences, whereas 2 Hz would just not cut it. And, of course, further performance tweaks could be made to the C++ module with inline assembly, loop unwinding, a call to the Intel MMX-optimized YUV/RGB conversion code, and so on.
Image Processing
The third file, yuvexample.m, illustrates how to use the two-color space-conversion routines. In this example, I take an image and enhance it by making it brighter and increasing its color contrast; Figure 1 is the original image.
To make the image brighter, you first need to convert it to YUV color-space to separate its color information from its brightness information. This is done using the rgb2yuv MEX-file. Figure 2 shows respective portions of the separate Y, U, and V channels combined into a single figure in order to conserve print space. I have used different color palettes on the U and V channels to suggest red and blue color components and to make the image more visually appealing; however, the color in these images doesn't correspond to the images' actual colors.
Once you have the image separated into Y, U, and V components, increase the brightness by multiplying the Y channels by a scalar greater than 1. The Y channel has a range of 0 to 255, so multiplying by a value greater than 1 simply increases the overall brightness.
The U and V channels in the image range from -128 to 127. Therefore, to increase the color contrast of the image, you can also just multiply these two channels by a scalar greater than one. This increases the difference between two colors whose values lie on opposite sides of the mean value, 0.
After making these modifications to the image, you convert it back to RGB color-space with the yuv2rgb routine. Figure 3 shows the resulting processed image. As you can see, the processed image is indeed brighter and has more vibrant colors.
Conclusion
Implementing code in a high-level language such as Matlab, then identifying "hot-spots" and rewriting those parts in lower level languages such as C++, makes a lot of sense. It lets you quickly generate prototypes, but it also lets you modify the prototype such that it runs at reasonable speeds. In this way, the prototype has the potential to evolve into a final, usable product. External interface routines are what make this strategy possible in the Matlab environment.
