John is vice president of engineering at Electric Cloud, which focuses on reducing software build times. He can be contacted at [email protected].
Make giveth and Make taketh away. It's ironic that the tool Make, designed to solve the "I've changed some source now what do I need to recompile?" problem, creates dependency nightmares.
Any project larger than a "Hello World" example faces dependency management problems. Dependencies must be generated and kept up to date as you add to, modify, and delete from the project. And Make itself provides no tools for dealing with this problem—all Make provides is a mechanism for expressing the relationships between files with its familiar target : prerequisite1 prerequisite2... syntax.
The target of the rule (the file that will be built) appears before the colon (:) and the files that the target depends upon (called either the "dependencies" or "prerequisites") appears after the colon (:). For example:
foo.o : foo.c header.h
has target foo.o and prerequisites foo.c and header.h.
Even Make's dependency syntax is flawed because it incorporates both "foo.o must be updated if header.h or foo.c are changed" and "foo.o is the result of compiling foo.c". Thus, anything to the right of the ":" is a prerequisite, but the first prerequisite where there's a rule body (that is, commands) is special—it's the prerequisite that's passed to the compiler (or other command) to actually generate the target.
Look at this GNU Make example (I use GNU Make throughout this article because of its wide platform coverage and large set of features):
foo.o: foo.c header.h system.h
@echo Compiling $@ from $<...
which outputs:
Compiling foo.o from foo.c...
Here, foo.o is built if foo.c, header.h, or system.h change, but the rule also states that foo.o is made from foo.c (in GNU Make terms, the target of the rule to the left side of the ":" is written $@ and the first prerequisite is $<). If the example were written like this:
foo.o: foo.c
foo.o: header.h system.h
@echo Compiling $@ from $<...
the output would be:
Compiling foo.o from header.h...
which is clearly wrong. If you want to continue exploring Make's idiosyncrasies with $<, play around with this Makefile:
foo.o: header.h
foo.o: system.h
foo.o: foo.c
foo.o:
@echo Compiling $@ from $<...
and try permuting the first three lines to see how Make's interpretation of the Makefile changes.
The biggest problem of all is generating these rules for a large project. In the rest of this article, I use this contrived example Makefile as a starting point:
.PHONY: all
all: foo.o bar.o baz.o
foo.o: foo.c foo.h common.h header.h
bar.o: bar.c bar.h common.h header.h ba.h
baz.o: baz.c baz.h common.h header.h ba.h
Three object files (foo.o, bar.o, and baz.o) are built from corresponding C files (foo.c, bar.c, and baz.c). Each .o file has dependencies on various header files as expressed in the Makefile. The Makefile uses GNU Make's built-in rules to perform compilation using the system's compiler.
There's no mention here of the final executable being built because I concentrate on dealing with dependencies between sources and objects. Relationships between objects are usually easier to maintain by hand as there are fewer of them and the relationships are part of the product design.
Because maintaining any real Makefile by hand is impossible, many projects use the widely available makedepend program (makedepend is usually installed on UNIX and CYGWIN systems). makedepend reads C and C++ files looking at the #include statements, follows the #includes, and builds the dependency lines for you. A basic way of incorporating makedepend in a project is a special depend target:
.PHONY: all
all: foo.o bar.o baz.o
SRCS = foo.c bar.c baz.c
DEPENDS = dependencies.mk
.PHONY: depend
depend:
@makedepend -f - $(SRCS) > $(DEPENDS)
-include $(DEPENDS)
Doing makedepend with this Makefile causes the depend rule to execute, which runs makedepend on the sources (defined in the SRCS variable) and outputs the dependency lines to dependencies.mk (defined by the DEPENDS variable).
The Makefile includes the dependencies lines in its final line. dependencies.mk looks like this:
# DO NOT DELETE
foo.o: foo.h header.h common.h
bar.o: bar.h header.h common.h ba.h
baz.o: baz.h header.h common.h ba.h
Notice that makedepend doesn't try to define the relationship between an object file (for example, foo.o) and the source file it is made from (foo.c). In this case, GNU Make's standard rules automatically find the related .c file. There are two problems with the makedepend style:
- Running makedepend can be slow, as every source file has to be searched even if there are no changes.
- It's a manual step. Before every make, users have to do makedepend to ensure that the dependencies are correct.
The answer to these problems is automation. Here's a version of the Makefile that still uses makedepend to generate dependencies, but automates the process, and only runs makedepend for sources that have changed:
.PHONY: all
all: foo.o bar.o baz.o
SRCS = foo.c bar.c baz.c
%.d : %.c
@makedepend -f - $< | sed 's,\($*\.o\)[ :]*,\1
$@ : ,g' > $@
-include $(SRCS:.c=.d)
It works by associating a .d file with each .c; for example, foo.o has a foo.d file that only contains the dependency line for foo.o. Here are foo.d's contents:
# DO NOT DELETE
foo.o foo.d : foo.h header.h common.h
This line specifies when to rebuild foo.o, but also that foo.d should be rebuilt under the same conditions—if any of the sources associated with foo.o change, then foo.d gets rebuilt. foo.c isn't mentioned in this list because it's mentioned as part of the pattern rule for rebuilding a .d file (%.d : %.c means that foo.d gets rebuilt if foo.c itself changes). foo.d got added to the dependency line created by makedepend using the aforementioned sed magic.
The final line of the Makefile includes all the .d files: The $(SRCS:.c=.d) macro transforms the list of sources in the SRCS variable by changing the extension from .c to .d. The include also tells GNU Make to check to see if the .d files need rebuilding. GNU Make looks to see if there are rules to rebuild included Makefiles (in this case, the .d files), rebuilds them if necessary (following the dependencies specified in the Makefile), then restarts. This "Makefile remaking" feature (see section 3.7 "How Makefiles Are Remade" in the FSF's GNU Make manual; http://www.gnu.org/software/make/manual/html_mono/make.html#SEC20) means that simply typing "make" will rebuild any dependency files that need rebuilding—but only if the sources have changed. Then, GNU Make will perform the build, taking into account the new dependencies.
Unfortunately, this Makefile breaks with a fatal error if a header file is removed. If header.h is no longer needed and all references to it are removed from the .c files and the file is removed from disk, this error occurs when Make is run:
No rule to make target 'header.h',
needed by 'foo.d'.
This happens because header.h is still mentioned in foo.d as being a prerequisite of foo.d; hence, foo.d cannot be rebuilt. This Catch-22 can be fixed by making the generation of foo.d smarter. The new foo.d includes the dependencies for foo.o and foo.d separately. foo.d's dependencies are wrapped in a call to GNU Make's $(wildcard) function (see section 4.4.3 "The Function Wildcard" in the FSF's GNU Make manual; http://www.gnu.org/software/make/manual/html_mono/make .html#SEC33). Here's the new foo.d:
# DO NOT DELETE
foo.d :
$(wildcard foo.h header.h common.h)
foo.o : foo.h header.h common.h
And here's the updated Makefile with a new invocation of makedepend, followed by a sed line that creates the modified .d file:
.PHONY: all
all: foo.o bar.o baz.o
SRCS = foo.c bar.c baz.c
%.d : %.c
@makedepend -f - $< | sed 's,
\($*\.o\)[ :]*\(.*\),
$@ : $$\(wildcard \2\)\n\1 : \2,g' > $@
-include $(SRCS:.c=.d)
Now removing a header file doesn't break the Make: When foo.d is parsed, the dependency line for foo.d is passed through $(wildcard). When there are no globbing symbols such as "*" or "?" in the filename, $(wildcard) acts as an existence filter, removing those files that don't exist from the list. So, if header.h had been removed, the first line of foo.d would be equivalent to:
foo.d : foo.h common.h
and the Make would work correctly. This example Makefile now works when .c files are added (users just update SRCS and the new .d file is created automatically), when .c files are removed (users update SRCS and the old .d file is ignored), and when headers are added (because that requires touching an existing .c or .h, the .d file is regenerated). And when they are removed, the $(wildcard) hides the deletion and the .d file is regenerated.
An optimization is to remove the need for GNU Make to restart by merging the rule that makes the .d file into the rule that makes the .o file. Because the .d file is updated if (and only if) the .o file needs to be updated (both are updated when any of the sources for the .o change), it's possible to have the makedepend occur at the same time as the compilation:
.PHONY: all
all: foo.o bar.o baz.o
SRCS = foo.c bar.c baz.c
%.o : %.c
@makedepend -f - $< |
sed 's,\($*\.o\)[ :]*\(.*\),
$@ : $$\(wildcard \2\)\n\1 : \2,g' > $*.d
@$(COMPILE.c) -o $@ $<
-include $(SRCS:.c=.d)
This rule makes use of $*, another GNU Make variable. $* is the part of the pattern %.c that matches the %. If this rule is building foo.o from foo.c, then $* is just foo. $* is used to create the name of the .d file that makedepend writes to. This final version does not use GNU Make's "Makefile remaking" system. There are no rules for making .d files (they are made as a side effect of making the .o); hence, GNU Make does not have to restart, providing the best combination of accuracy and speed possible.
In general, it's a bad idea to have a rule that makes multiple files because it's impossible for GNU Make to find the rule that makes a file if it's created as a side effect of something else. In this case, that behavior is desired: You want to hide the creation of .d files from GNU Make so that it doesn't try to make them and then have to restart. A similar idea was proposed by Tom Tromey, without the $(wildcard) trick, and more information about building dependency files can be found on the GNU Make maintainer Paul Smith's web site at http://make.paulandlesley.org/autodep.html. Another good resource for any GNU Make developer—once you've purchased the FSF's GNU Make manual—is Robert Mecklenburg's Managing Projects with GNU Make (O'Reilly & Associates, 2004).
Finally, it's possible to omit makedepend altogether if you are using the GNU GCC compiler. It has a -MD option that does the work of makedepend at the same time as the compilation:
.PHONY: all
all: foo.o bar.o baz.o
SRCS = foo.c bar.c baz.c
%.o : %.c
@$(COMPILE.c) -MD -o $@ $<
@sed -i 's,\($*\.o\)[ :]*\(.*\),
$@ : $$\(wildcard
\2\)\n\1 : \2,g' $*.d
-include $(SRCS:.c=.d)
For example, the compilation step for foo.o will create foo.d from foo.c and then sed is run on the foo.d to add the extra line for foo.d containing the $(wildcard).
The use of GCC -MD is an example of creating dependencies without using makedepend. There are a number of other possibilities. For example, GCC has -M and -MM options that just output dependency information (-M outputs all dependencies and -MM omits the system headers because they are unlikely to change).
Another option is the program fastdep (http://www.irule.be/bvh/c++/fastdep/), which aims to be a fast replacement for makedepend. Finally, Windows programmers can use Microsoft CL's /showincludes option to get include information and build dependency information.
All of these solutions share some common problems. They only work well for C and C++ code and need to be modified to handle other languages (although luckily, not all languages have the same dependency forests as C and C++). In addition, the files included might change as the result of preprocessor defines; hence, makedepend (or an equivalent program) needs to be told about any command line -D defines so that it builds the right dependency information.
Despite the problems outlined here, these Makefile snippets do provide a reliable way of keeping C and C++ dependencies up to date automatically and quickly.
DDJ