Channels ▼
RSS

Tools

A Build System for Complex Projects: Part 3


Gigi Sayfan specializes in cross-platform object-oriented programming in C/C++/ C#/Python/Java with emphasis on large-scale distributed systems. He is currently trying to build intelligent machines inspired by the brain at Numenta (www.numenta.com).


A Build System for Complex Projects: Part 1
A Build System for Complex Projects: Part 2
A Build System for Complex Projects: Part 3
A Build System for Complex Projects: Part 4
A Build System for Complex Projects: Part 5


This is the third article in a series of articles that explore an innovative build system for complicated projects. Part 1 and Part 2 discussed build systems in general and the internals of the ideal build system that can integrate with existing build systems. This article will explain in detail how the ibs can generate build files for NetBeans 6.x, which is a fantastic free cross-platform IDE that supports multiple programming languages.

To recap: ibs is an invisible build system that doesn't require any build files. It relies on a regular directory and conventions to infer build rules and it detects dependencies automatically. It generates build files for other IDEs or build systems like Makefiles, Visual Studio solutions, or NetBeans projects. It is focused on C/C++ projects, but can be extended to additional languages and other projects types.

Generating a NetBeans Build System

Back in the Hello world project (Enterprise Platinum Edition!!!) trenches Bob and the other developers picked NetBeans as their primary Mac and Linux IDE. NetBeans started as a Java IDE, but grew in leaps and bounds to become a component-based platform for application development in addition to an IDE that supports all the common programming languages in wide use today. The latest version is NetBeans 6.7.1 and it supports C/C++ development very well and even Python via a special early access.

The task facing Bob is to figure out the structure of the NetBeans build files for C/C++ and implement the build system specific parts (the helper and the project templates) to embrace NetBeans into the ibs.

Bob did some reading and poking around and discovered that NetBeans itself is a Makefile-based (build system generator. The way it works is that each project's directory has a common Makefile that includes a bunch of auto-generated sub-makefiles. The NetBeans user interface allows you to add files to every project and set dependencies between projects. All this information is stored in several files that NetBeans uses to generate the proper sub-makefiles. Figure 1 shows the NetBeans IDE with the various Hello World projects.

Figure 1

The NetBeans Build System Anatomy

Let's take a look and see what all these files are about. As a running example Bob suggests the the 'hello' project. This is a static library that contains two files: hello.cpp and hello.hpp. If you are unfamiliar with make-based build systems, you may want to take a small detour and read about it here.

The project Makefile

In the 'hello' directory there is the common Makefile. This file is the same for every project (on a given platform). It sets some environment variables and contains a bunch of make targets and most importantly includes the project implementation makefile (generated based on the actual content of the project). Every target (a step in the build process) has an empty pre and post targets that allow you to customize the build process by executing actions before and/or after each target. The original file has elaborate comments that explain exactly what targets are available and how you can override them. Here is an edited version without the comments and some of the targets of the Mac OS X Makefile:


# Environment 
MKDIR=mkdir
CP=cp
CCADMIN=CCadmin
RANLIB=ranlib

# clean
clean: .clean-pre .clean-impl .clean-post

.clean-pre:
# Add your pre 'clean' code here...

.clean-post:
# Add your post 'clean' code here...


# all
all: .all-pre .all-impl .all-post

.all-pre:
# Add your pre 'all' code here...

.all-post:
# Add your post 'all' code here...


# include project implementation makefile
include nbproject/Makefile-impl.mk

nbproject

The Makefile resides in the project's directory. All the other build files reside in a sub directory called nbproject. There is nothing special about it. It is just convenient to have all the build files in their own directory and not cluttering the project directory. The Makefile is the exception due to limitations of the make program.

Generated make files

NetBeans generates three "sub" make files that are included by the main project Makefile: Makefile-impl.mk, Makefile-Debug.mk and Makefile-Release.mk. Makefile-impl.mk is included directly by the main Makefile and it invokes either Makefile-Debug.mk or Makefile-Release.mk depending on the current active configuration, which is controlled by the $CONF environment variable. In the NetBeans IDE you may select what configuration is active.You may also create your own configurations and they will be available for activation just like the built-in Debug and Release configurations with their own Makefile-your configuration.mk file. Here is the Makefile-impl.mk file of the testPunctuator project:


# 
# Generated Makefile - do not edit! 
# 
# Edit the Makefile in the project folder instead (../Makefile). Each target
# has a pre- and a post- target defined where you can add customization code.
#
# This makefile implements macros and targets common to all configurations.
#
# NOCDDL

# Building and Cleaning subprojects are done by default, but can be controlled with the SUB
# macro. If SUB=no, subprojects will not be built or cleaned. The following macro
# statements set BUILD_SUB-CONF and CLEAN_SUB-CONF to .build-reqprojects-conf
# and .clean-reqprojects-conf unless SUB has the value 'no'
SUB_no=NO
SUBPROJECTS=${SUB_${SUB}}
BUILD_SUBPROJECTS_=.build-subprojects
BUILD_SUBPROJECTS_NO=
BUILD_SUBPROJECTS=${BUILD_SUBPROJECTS_${SUBPROJECTS}}
CLEAN_SUBPROJECTS_=.clean-subprojects
CLEAN_SUBPROJECTS_NO=
CLEAN_SUBPROJECTS=${CLEAN_SUBPROJECTS_${SUBPROJECTS}}

# Project Name
PROJECTNAME=testPunctuator

# Active Configuration
DEFAULTCONF=Debug
CONF=${DEFAULTCONF}

# All Configurations
ALLCONFS=Debug Release 

# build
.build-impl: .validate-impl 
  @#echo "=> Running $@... Configuration=$(CONF)"
  ${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf


# clean
.clean-impl: .validate-impl
  @#echo "=> Running $@... Configuration=$(CONF)"
  ${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .clean-conf


# clobber 
.clobber-impl:
  @#echo "=> Running $@..."
  for CONF in ${ALLCONFS}; \
  do \
      ${MAKE} -f nbproject/Makefile-$${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .clean-conf; \
  done

# all 
.all-impl:
  @#echo "=> Running $@..."
  for CONF in ${ALLCONFS}; \
  do \
      ${MAKE} -f nbproject/Makefile-$${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf; \
  done


# configuration validation
.validate-impl:
  @if [ ! -f nbproject/Makefile-${CONF}.mk ]; \
  then \
      echo ""; \
      echo "Error: can not find the makefile for configuration '${CONF}' in project ${PROJECTNAME}"; \
      echo "See 'make help' for details."; \
      echo "Current directory: " `pwd`; \
      echo ""; \
  fi
  @if [ ! -f nbproject/Makefile-${CONF}.mk ]; \
  then \
      exit 1; \
  fi

# help
.help-impl:
  @echo "This makefile supports the following configurations:"
  @echo "    ${ALLCONFS}"
  @echo ""
  @echo "and the following targets:"
  @echo "    build  (default target)"
  @echo "    clean"
  @echo "    clobber"
  @echo "    all"
  @echo "    help"
  @echo ""
  @echo "Makefile Usage:"
  @echo "    make [CONF=<CONFIGURATION>] [SUB=no] build"
  @echo "    make [CONF=&gltCONFIGURATION>] [SUB=no] clean"
  @echo "    make [SUB=no] clobber"
  @echo "    make [SUB=no] all"
  @echo "    make help"
  @echo ""
  @echo "Target 'build' will build a specific configuration and, unless 'SUB=no',"
  @echo "    also build subprojects."
  @echo "Target 'clean' will clean a specific configuration and, unless 'SUB=no',"
  @echo "    also clean subprojects."
  @echo "Target 'clobber' will remove all built files from all configurations and,"
  @echo "    unless 'SUB=no', also from subprojects."
  @echo "Target 'all' will will build all configurations and, unless 'SUB=no',"
  @echo "    also build subprojects."
  @echo "Target 'help' prints this message."
  @echo ""

The structure of this file is very uniform. Every command is implemented in the same way (except .help that just echos the help text to the screen). It always invokes eventually the configuration specific make file. Commands may operate on all configurations and on sub-projects too (on by default). For example the default .build command (if you just type 'make') is:


# build
.build-impl: .validate-impl 
  @#echo "=> Running $@... Configuration=$(CONF)"
  ${MAKE} -f nbproject/Makefile-${CONF}.mk SUBPROJECTS=${SUBPROJECTS} .build-conf

Let me decipher this line-noise that makes sense only to make-savvy people. The name of the command is .build-impl. It will execute the .validate-impl command, skip the commented out echo command (if you remove the # it will print the text between the double quotes) and run 'make' again on the file nbproject/Makefile-${CONF}.mk (CONF is the active configuration, which is Debug in this case, unless you specified CONF=Release when you run 'make'). Finally it will execute the .build-conf command that is defined in the Makefile-${CONF}.mk. This command builds all the sub projects (if necessary) and finally build the project itself by invoking the C++ compiler and linker.

It sounds complicated and it is complicated. This is the cleanest make-based system I have seen with good separation of concerns, very uniform structure and great extensibility. Most make-based build systems are simply a mess. The nice thing about NetBeans is that it takes care of all the messy parts and lets you work entirely at the IDE level, but still allows you to extend the build process at the makefile-level if you need to do something special.

Let's take a look at the Makefile-Debug.mk file:


#
# Generated Makefile - do not edit!
#
# Edit the Makefile in the project folder instead (../Makefile). Each target
# has a -pre and a -post target defined where you can add customized code.
#
# This makefile implements configuration specific macros and targets.

# Environment
MKDIR=mkdir
CP=cpj
CCADMIN=CCadmin
RANLIB=ranlib
CC=gcc
CCC=g++
CXX=g++
FC=

# Include project Makefile
include Makefile

# Object Directory
OBJECTDIR=build/Debug/GNU-MacOSX

# Object Files
OBJECTFILES= \
  ${OBJECTDIR}/main.o

# C Compiler Flags
CFLAGS=

# CC Compiler Flags
CCFLAGS=
CXXFLAGS=

# Fortran Compiler Flags
FFLAGS=

# Link Libraries and Options
LDLIBSOPTIONS=../../hw/utils/dist/Debug/GNU-MacOSX/libutils.a

# Build Targets
.build-conf: ${BUILD_SUBPROJECTS} dist/Debug/GNU-MacOSX/testpunctuator

dist/Debug/GNU-MacOSX/testpunctuator: ${BUILD_SUBPROJECTS}

dist/Debug/GNU-MacOSX/testpunctuator: ${OBJECTFILES}
  ${MKDIR} -p dist/Debug/GNU-MacOSX
  ${LINK.cc} -o dist/Debug/GNU-MacOSX/testpunctuator ${OBJECTFILES} ${LDLIBSOPTIONS}

${OBJECTDIR}/main.o: main.cpp 
  ${MKDIR} -p ${OBJECTDIR}
  $(COMPILE.cc) -g -I../.. -o ${OBJECTDIR}/main.o main.cpp

# Subprojects
.build-subprojects:
  cd ../../hw/utils && ${MAKE}  -f Makefile CONF=Debug

# Clean Targets
.clean-conf: ${CLEAN_SUBPROJECTS}
  ${RM} -r build/Debug
  ${RM} dist/Debug/GNU-MacOSX/testpunctuator

# Subprojects
.clean-subprojects:
  cd ../../hw/utils && ${MAKE}  -f Makefile CONF=Debug clean

This file includes the project's Makefile (which includes the Makefile-impl.mk), defines a bunch of variables that point to different tools like C and C++ compilers and it also defines the dependencies of the project (in this case just the hw/utils sub-project). Note the .build-conf command that I mentioned earlier when I discussed the .build-impl command from Makefile-impl.mk. So, there is a lot of interplay between the various make files. This is done in the interest of separating fixed logic like command invocation from very dynamic parts like the files that are contained in a project and its dependencies and also providing clear extension points (the.pre and .post commands in the main Makefile). The bottom line is that most developers don't even need to know that there is a make-based build system underneath and can just stay at the IDE level. Build administrators can automate the build process using this clean and standard make interface and people with special needs can customize the build process very elegantly using the extension points provided by .pre and .post targets, as well as add new targets.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video