Lint Early, Lint Often
"Lint is your software conscience. It tells you when you are doing bad things. Always use Lint. Listen to your conscience." [1]Lint is a tool for finding problems in your source code that the compiler may have missed before run time. Linting saves development time by squashing bugs before the effort-intensive unit and integration testing cycles. This article describes a set of shell scripts for UNIX systems that makes running Lint against applications and libraries as simple as running Make.
I will present a wrapper that lets you reuse makefiles written for a C compiler to drive Lint. Although Lint uses a subset of the compiler switches (e.g., -D, -I, -U), it is ordinarily not possible to just drop in Lint instead of cc to produce a report and generate Lint libraries.
The wrapper piggybacks Lint onto the C compiler, system linker, archiver, and installer. As Make runs, it calls the wrapper script. The wrapper first runs the original tool that Make expects and then runs Lint. Any project (that uses a Make variable for the C compiler name) can, therefore, instantly make use of Lint:
CC=lintwrapper make
As Make runs, the Lint report and Lint libraries are generated.
The benefit of this technique is that no additional configuration management is required to run Lint. In other words, you don't have to write extra makefile rules or create shell scripts to run Lint on the project. The Lint libraries are generated and installed along with the build:
CC=lintcc LD=lintld \
AR=lintar \
INSTALL=lintinstall \
configure
make
make install
Any libraries you build or install with the wrapper get a Lint library, and all other dependent packages get access to the extra checks in a Lint library that are not provided in the header files.
To summarize, using the wrapper script lets you run Lint without the overhead of modifying the build configuration.
Why a Lint Wrapper?
Lint can spot errors that the compiler does not see. The original purpose of Lint was to add extra semantic checking so that the C compiler could stay lean and specialize in code generation. Modern compilers are much better about reporting suspect code then when Lint was originally created. That being said, there is still a place for Lint. Many compilers will not spot the two errors in the following function.
#include <stdio.h>
void printsqr(unsigned int tosqr) {
int square;
/* !!! double equals !!! */
square == tosqr;
square *= square;
/* !!! format string error !!! */
(void)printf("%f^2 = %u\n",
tosqr, square);
}
Even the verbose gcc -Wall does not find the errors. You need to run gcc -Wall -Wunitialized -O to find both problems. (Surprisingly, the optimized flag is required to find the unset variable error.)
For those using less particular compilers, Lint can spot these errors and other errors the compiler/linker cannot find--especially problems between source files. The object code the C compiler generates does not record the data types of global symbols, function arguments, or function returns. Lint can retain this information and use it to check for interfile errors that the linker would miss.
When my company changed from an in-house build system to using the GNU Autotools (autoconf/automake/libtool) [2], we lost support for Lint in our builds. Unsurprisingly bugs that used to be caught by Lint began to appear in my code.
I needed my conscience back.
Solutions
To run Lint on an application, you need to know the source files, libraries, and preprocessor defines used to build it. On UNIX this information is almost always contained in makefiles. There were two ways I could retrieve it:1. Parse the makefile directly and extract the build information.
2. Pretend to be the compilation tool chain (cc/ld/ar) and run Lint at the same time.
Recreating the logic in Make seemed like overkill, so I chose the second option for simplicity and generality. The checking done by Lint should never have been separated from the C compiler, so why not put them back together?
Requirements of Lint
Replacing the compiler, linker, and archiver with wrapper scripts requires Lint to support all of the same concepts as the normal tool chain:
- Compiling source files into intermediate object files.
- Maintaining libraries of object files.
- Combining libraries and intermediate code all together to link (or in Lint's case, check) a program.
Lint's equivalent to a library is the Lint library, usually called llib-lname.ln for libname.a/so. Lint can take in many source and intermediate files and combine them to create the Lint library with the -o or -C flag, depending on the variant.
Wrapper Scripts
When pretending to be the C compiler, the lint command line is frustratingly close. You can almost replace calls to cc with lint and run Make. Lint supports some of the same arguments as the C compiler -I, -D, -U, -L, and -l. Unfortunately, that's where the similarity ends. Some Lints do not support renaming intermediate output files. The Lint I use on the SCO OpenServer even requires Lint libraries to be specified before the source files.If you've ever built an Autotools-based package with
configure && make && make install
then you'll know the weird and wonderful commands produced for the compiler and just how much output is produced. Before writing the wrapper, I wanted to find out just what it would need to do. A simple shell script proved that a wrapper script would work and captured how Make called the tool chain. Creating scripts for cc, ld, ar, and install, I collected all of the information I needed.
#! /bin/sh
# Example wrapper script for cc
echo $PWD:cc:"$@" >> /tmp/buildlog
exec cc "$@"
Being the Compiler or the Linker
The compiler generates object code from source files and sometimes it calls the linker on its behalf to link binaries or shared libraries. This process breaks into three tasks for the compiler wrapper:
Compiling .ln files.
Creating a Lint report for a
program.
Generating Lint libraries.
All the wrapper needs to do to create .ln files is extract the pertinent arguments and source file names from the compiler and then pass them on to Lint with the intermediate compile flag. Pertinent arguments give include file locations (-I) and preprocessor defines and undefines -D/-U, making sure to preserve the order the arguments are passed to the C compiler. Extracting these arguments and suppressing all of the other arguments translates this call to the compiler:
gcc -c -DHAVE_CONFIG_H -O3 -Wall -I.. ../mypackage/main.cinto a very similar call to Lint.
lint -c -DHAVE_CONFIG_H -I.. ../mypackage/main.cGenerating the Lint report for a program, the equivalent to the final link, requires discovering the object files and libraries from the command line and converting them to their Lint equivalents. The object files are the easy part; just change the file extension from the object file (e.g., .o) to the intermediate Lint file (.ln). Libraries are a little harder.
Libraries can be given as arguments using -lname with the library search path or by giving an actual path to the library file. Libraries also come in two forms, static and dynamic. Lint only supports one type of library so there is a namespace problem. libname.so and libname.a could have different functions in them. Additionally Lint libraries are not present for all libraries on the system--particularly third party libraries (unless, of course, you used this wrapper when building them).
The wrapper appends -so to the name of any dynamic libraries to solve the static/dynamic namespace problem. It solves the location problem by transforming any -lxxx to the Lint library name llib-lxxx.ln and searching the library path for it. If the wrapper is unable to find the library, the library is dropped to prevent Lint from warning about it--not many third party libraries provide a Lint library.
So after passing through the wrapper:
ld -G -o libdbl2.so libsrc1.o libsrc2.o -lc
cc -L/usr/local/lib -omyprog main.o app1.o \
app2.o libdbl2.so -lm
becomes
lint /usr/ccs/bin/llib-lc.ln -odbl2-so libsrc1.ln libsrc2.ln
lint -L/usr/local/lib llib-lsbl2-so.ln \
/usr/ccs/lib/llib-lm.ln main.ln app1.ln app2.ln -lc
Being the Static Library Archiver
Static libraries can be created by the archiver from several object files and updated incrementally by adding new files to the library as they are compiled. Some Makes support this directly by supporting automatic dependency rules for static libraries.
libxyz.a : libxyz.a(src1.o src2.o)
If any of the source files are newer than the object code stored in the library, they will be compiled and archived into the library.
[make] cc -c -o src1.o src1.c
[make] ar rv libxyz.a src1.o
a - src1.o
Creating libxyz.a
[make] cc -c -o src2.o src2.c
[make] ar rv libxyz.a src2.o
a - src2.o
This is not how Lint normally creates its libraries. Lint needs all of the intermediate files at creation time like a dynamic library. Fortunately there is a way to make Lint create libraries in the way I just described. Although the archiver is normally used only for object files, it will happily archive any file. At the same time, the static archive is created, a parallel Lint archive is built from the Lint intermediate files with an .aln extension. If the static library is updated with new object files, the Lint archive can be updated with new Lint object files. After each update, all of the Lint object files can be extracted to a temporary directory and the Lint library rebuilt from all of the files contained in the archive.
[make] lint -c -src1.c
[make] ar rv libxyz.aln src1.ln
a - src1.ln
Creating libxyz.aln
[make] lint -oxyz src1.ln
[make] lint -c src2.c
[make] ar rv libxyz.aln src2.o
a - src2.o
[wrapper] mkdir tmplibdir
[wrapper] ar x ../libxyz.aln
[wrapper] lint -oxyz *.ln
[wrapper] mv llib-lxyz.ln ..
[wrapper] cd .. && rm -rf tmplibdir
An unfortunate side effect of this technique is the repeating of Lint errors each time the library is updated, but enduring repeated messages is better than missing the messages entirely.
Installing Lint Libraries
It would be a great shame if the newly created Lint libraries never saw the light of day. The install wrapper makes sure they are installed at the same time as the real libraries by intercepting each call to the install command.If the install command arguments match the pattern libname.a or libname.so*, the wrapper assumes a library is being installed. It uses the same mapping as the linker wrapper to transform each of the library names and then reissues the modified install command for the Lint library. The Lint library is copied to the same directory as the real library for future Linting and will be installed correctly with the same user name and permissions.
Living with libtool
libtool is a GNU package that simplifies building dynamic libraries on many different platforms. libtool takes care of the esoteric compiler/linker options necessary to create and version dynamic libraries. Automake packages often use Libtool to build dynamic libraries. The libtool build process is complicated by the fact that libtool is designed to work on many systems.libtool arranges for a second compilation of every source file to make sure that dynamic libraries are compiled correctly. libtool passes the compiler the correct flags for creating the position-independent code necessary for dynamic libraries. The extra object file is given an extension .lo to differentiate it from the static version and is first written into a libtool-generated subdirectory .libs, then moved back into the build directory by a move command run after the compiler.
Creating two versions of the object files causes a problem for Lint because many Lint variants do not allow you to set the output filename. To support other output file names, the Lint wrapper just needs to rename the output file. The Lint wrapper renames .lo files to -lo.ln files. So that existing Lint object files with the default .ln name are not overwritten, the wrapper temporarily renames the existing .ln file while Lint is run and puts it back afterwards.
The solution for the .libs problem is to simply remove it from the path whenever it is encountered. This solution seems to work in practice.
The Lint Wrappers
There is a wrapper for the compiler (lintcc), linker (lintld), archiver (lintar) and install command (lintinstall). (The source to the wrappers is available for download at <www.cuj.com/code>.) Each wrapper perfoms the steps described above for each of the tools with a little extra housekeeping. The wrappers themselves are written as shell scripts. They all run the real tool, record the real exit code, do their Lint processing, and return the real exit code.The Lint wrappers need to know what to do with the generated output. Sometimes it is good to have Lint output separate from the compiler. Linting, especially for the first time, can generate a lot of output. When the wrapper poses as the compiler, there is no way to pass options to it, so environment variables are used instead. If the environment variable LINTLOG is set, all output, (including the output to stderr) is redirected there, otherwise, the output is written to the same place as the compiler messages.
In one situation, regardless of the output mode, the Lint wrappers must not produce any output at all. Inside configure scripts, the output from the C compiler is tested to detect which compiler/version it is and what features it supports by compiling small test files. If Lint finds errors in any of the source files and outputs messages, it stops configure from working correctly.
Using a little inside knowledge, it is possible to determine whether the wrapper is running inside an autoconf-generated configure script. The configure script uses file descriptor 6 for logging, so if it is open then it is very likely that configure is calling the wrapper. This fragment of shell can detect it on Korn/Bash shells. The echo command will fail because the file descriptor is not open.
INCONFIGURE=$( \
( if echo -n '' >&6 ; \
then echo "Y" ; \
else echo "N" ; fi ) \
2>/dev/null )
When automatic detection doesn't work, the environment variable LINTOFF=Y can be set during the configuration stage and unset when running Make.
Installing, Configuring, and Running the Wrappers
The Lint wrappers build and install as a normal package does. Uncompress the source, then run configure, make, and make install. The configuration script detects the normal names for the real tools and the variant of Lint on the current machine.Make builds a suitable set of wrapper scripts from the configured parameters. The wrappers get installed into the bin directory as lintcc, lintld, lintar, and lintinstall.
Different Lints can be supported concurrently if required by rerunning the configuration script with a different Lint name and giving a different program prefix. SCO provides a Lint designed to help porting applications to 64-bit. It can be wrapped using
configure --program-prefix=g64 \
LINT=/g64/usr/ccs/bin/g64lint \
LINTLIBPATH="/g64/usr/ccs/lib:/g64/usr/lib"
The wrapper will be installed as g64lintcc, g64lintld...
There are several different ways to run the wrapper. If the package uses configure, setting environment variables to use the wrappers and running the build system is enough. Usually, the variable CC is used to define the C compiler, LD the linker, AR the library archiver, and INSTALL the installer.
CC=lintcc LD=lintld
AR=lintar INSTALL=lintinstall
export CC LD AR INSTALL
configure && make && makeinstall
Without configure, if the Make system follows the common Make idiom of using variables to the define compilation tools, the tools can be overridden on the command line.
make CC=lintcc \
LD=lintld AR=lintar \
INSTALL=lintinstall
Once completed, the application and/or libraries will be built and installed with the Lint messages interspersed with compiler messages. If you would prefer to separate the sometimes lengthy Lint output from the already lengthy Make output, just define LINTLOG.
LINTLOG=$PWD/lintlog
export LINTLOG
configure && make && make install
For build systems that do not support redefining the compiler, you can place the wrappers, renamed as the real tools, at the beginning of the PATH
PATH=/usr/local/lib/lintwrapper:$PATH make
Caveats
The wrappers can only work with the information they are given. They will fail if the build system manipulates files in the build area, e.g., copying them, symbolically linking them to new names, or moving files to other directories.One other process the linker performs that the linker wrapper does not support is combining multiple relocatable object files together into a single file (ld -r). You may be able to combine files by building a Lint library and renaming it, but I haven't explored this option because it hasn't been necessary for any packages I have built with the wrapper.
Future Work
The wrapper could be extended to support Lints that do not have support for Lint object files, notably the secure programming Lint (splint, also known as "lclint"). Instead of running Lint as the build progresses, you could capture the information and afterwards run Lint on the entire source tree in one command. The hard part is taking care of subdirectories and making sure there are no conflicting command line preprocessor defines or configuration header files.
Acknowledgments
Many thanks to the R&D team at Alchemetrics who kindly reviewed this article.
References
[1] Peter Van Der Linden. Expert C Programming (Prentice Hall, 1994).[2] Vaughan, Elliston, Tromey, and Taylor. GNU Autoconf, Automake, and Libtool (New Riders, 2000).
[2] Is also available as an open-source book at http://sources.redhat.com/autobook/.
About the Author
Jon Meredith works for Alchemetrics Ltd, developing high performance data processing systems for the marketing industry. He holds a bachelors degree in Electronic Engineering from the University of East Anglia, UK and has been developing in C for the past 15 years. He can be reached at [email protected].
