The TMS Development Platform

Alexander is a software engineer at FAST TV SERVER AG (http://www.tv-server.de/) in Munich. He can be contacted at [email protected].

When developing platform-independent software, you quickly reach the point where you need a development environment that reaches beyond the familiar IDEs and makefiles, offering a more universal approach. In this article, I present a simple but powerful solution called the "TV Server Makefile System" (TMS) that we use at Fast TV Server AG (where I work) to deal with development and target platforms, ranging from Windows and Linux to various embedded processors.

TMS provides development teams with a universal way of building software components, integrates a common test approach, and addresses problems that arise when supporting different hardware and platforms. Together with coding rules and component templates, we use it as a common framework for developing object-oriented software in C—still the main language for embedded programming.

I used Make as the build engine because it offers three advantages: It's free, available for every platform, and all developers are at least somewhat familiar with it.

While the solution I present here is designed for C/C++, it can be used for other languages as well. One of the main goals was to keep the whole system as simple and readable as possible (which means to not use every trick Make offers and limit the necessary preparations to a minimum) so that developers can understand or modify it.

A Project

In TMS, a "project" is the basic element for creating software. A project contains everything that you need for building an executable or library. Usually a project is a component in a larger system that offers a certain functionality through an API. To support a test-driven software development process, a project usually produces a library to contain the API implementation but also an executable to test the library.

A project can stand on its own, as it contains all files, output-directories, testscripts, documentation, and the like required for building, testing, and using the component. In short, each project consists of the subdirectories in Table 1.

The output directories (bin, extlib, lib, and obj) also contain subdirectories for each target platform (such as GCC or Visual C 6), which contain directories for debug and release versions. Although this increases the total number of directories, it offers the advantage of allowing two source/object files (or in a limited way even components) to use the same name, and it also lets you copy or backup a complete project by simply copying the project's main directory, including its subdirectories.

First Steps

To be set up on your system, TMS only requires:

• The GNU Make tool. On Windows you can to download and install it from the MinGW open-source project (http:// sourceforge.net/project/showfiles.php? group_id=2435). On UNIX platforms, it's usually available in the development packages.
• A system variable called "TMS_BASE_ DIR," which contains the path to your working directory but with UNIX slashes (for example, TMS_BASE_DIR=c:/ work/). On Windows you can do this in the system control panel; on Linux, using bash, you can add an export statement in your .bash_profile file inside your home directory.

You can easily develop on Windows for Linux. You only need a Linux server that runs Samba. You can then mount your home directory on the server (create a network drive with a fixed drive letter, for example) and put all of your working directories and files there. To build and test projects on Linux, log in to the server using an ssh or telnet client. Experienced users may prefer an X server, like that offered by Hummingbird Exceed (http://www .hummingbird.com/products/nc/exceed/) or Cygwin (http://www.cygwin.com/).

The makefiles

The most important thing when writing makefiles is to keep them as simple as possible. In the TMS build system, the functionality is separated in a bunch of makefiles to simplify their use and avoid duplication by separating the common from the special:

• The project makefiles (makefile.mak, for example; see Listing One).
• Common settings for specific target platforms (gcc_makefile_settings.mak; see Listing Two).
• Common actions for specific target platforms (gcc_makefile_create.mak; see Listing Three).
• User settings for specific target platforms (gcc_makefile_user_settings.mak; see Listing Four).

Each project has its own project makefile, which consists only of a few lines. Depending on the user's target platform, it includes the so-called settings and create makefiles, which exist only once inside the $TMS_BASE_DIR/makefiles directory. They define the platform-specific compiler and linker settings, as well as the compiling and linking process.

Consequently, a project makefile only needs to specify what is really unique for the project (see Listing One):

• Special compiler and linker flags (CINCS, CFLAGS, LDFLAGS).
• The source files that should be compiled for the library (LIBOBJECTS) and executable (APPOBJECTS).
• Tests (TESTS) that should be performed when Make is called with the test target.
• The name of the library (ARCHIVE) and executable (APPLICATION).
• The projects/components that are used by this project (dependencies) using their include makefiles (include...).

Keeping the project makefiles this simple has two important advantages: They are easy to read and modify, and they do not contain any platform- or target-specific code.

The system makefile (Listing Two) defines the compiler, linker calls, default flags, libraries, paths, and so on. The create makefile (Listing Three) provides actions such as compiling and linking files that are performed when a certain target is defined on the console. Usually you only need to create them once when adding support for new compilers.

But not all settings are user independent—some are left to be defined in the so-called user makefile (see Listing Four), which is included by the settings makefile. Here you can specify, for example, build debug or release versions, or the target hardware platform. All of these settings can also be overwritten by using system variables or command-line options.

Building a Project

To build a project on the console in TMS, you only need to call Make for the projects makefile. The parameters you need to specify are the target compiler (called TMS_TARGET, which can be also defined by a system variable if you only work with a specific compiler) and the build targets, which are performed in the given order. Possible targets are:

• lib, which compiles the objects in the LIBOBJECTS list and combines them in a static library.
• dll, which compiles the objects in the LIBOBJECTS list and combines them in a shared library.
• exe, which compiles the objects in the APPOBJECTS list and links them together with system and user libraries to create an executable.
• clean, which removes all objects, libraries, and executable files.
• test, which invokes the executable with the parameters in the TESTS list once for every list item (more details will follow).
• dep, which builds all items in the DEPENDENCIES list with the same settings (dependencies are not built recursively).

For example, to rebuild the library and executable using the GCC compiler:

make -f makefile.mak TMS_TARGET=
gcc clean lib exe

Building Larger Projects By Components

Larger software projects are usually built out of many different libraries, classes, and APIs. The TMS environment makes it easy to build components that can be reused by higher level projects using so-called inc-makefiles (Listing Five) located in the directory $TMS_BASE_DIR/makefiles/inc. This type of makefile acts like a header file for a specific component and can be included by other projects. It adds the path to the source (the header, for instance) and library files, as well as the library itself to the higher level projects compiler and linker flags.

For example, imagine the project MyProject, which uses a component named MyTimer. The only thing you need to do is add an include statement for the inc_MyTimer makefile in the MyProject makefile; for example:

include $(MAKE_DIR)/inc_MyTimer.mak

When you build MyProject, it automatically links the MyTimer library. The inc-makefiles also add an entry in the DEPENDENCIES list, which is used when you call the makefile with the target dep. Then for each project in this list, Make is called again, as you can see at the end of the inc-makefile. For example, the command:

make -f makefile.mak TMS_TARGET=
gcc clean lib dep

rebuilds the project's library and all libraries in the DEPENDENCIES list. Remember that all components are built in place—no header or library file needs to be copied to another location.

You can order and visualize the components hierarchy by using meaningful subdirectories for your projects. Therefore, you can also build layers of software that can be reused in other projects.

Platform-Specific Implementations

When working on multiple platforms or when you need to support different hardware, you always encounter a situation where you need to do specific implementations of a certain functionality. For example, imagine the MyTimer component needed to be implemented differently on Windows and Linux.

The first solution that comes to mind is to use #ifdefs in the code, but although this is written within a second, the consequences usually are much harder to come by. The first problem with #ifdefs is that the code gets harder to read. Especially when the number of platforms or hardware versions rise, you find it more and more difficult to tell which part is used by just taking a quick look at the source.

When you need to support different versions based on the same platform/compiler, the second problem is that you need to recompile the code, which usually results in recompiling all projects every time you switch between different targets and platforms—just to be sure you are not linking some older objects.

And you'll really start to hate the #ifdefs when you need to add support for a new platform or hardware version later on. Then you will need to go through all of your code looking for #ifdefs and add support for the new target. And because you modified the source code, you should compile and test the older targets as well.

The only solution to these problems is to abandon #ifdefs and instead write abstraction layers for operating systems, hardware boards, and so on. In the MyTimer example, you would create two separate projects—MyTimer_win and MyTimer_linux—using the same header file MyTimer.h, but having different implementations (see Figure 1). If you are using a versioning system such as SourceSafe or CVS, you can simply do this by sharing the header file so it's always unique. Then you only need a small including makefile inc_MyTimer.mak that works as a switch to choose the right library at link time (Listing Six). A higher level project will now only need to include the MyTimer project and won't have to care about the implementation anymore. Using this approach, you get rid of the previously mentioned #ifdef problems and improve the quality and reusability of your code.

Testing

Again, TMS enables a test-driven software development process. The test code is separated from the library code and is only compiled to build the executable. This executable can use command-line parameters to perform different test cases. You should also think of using a generic test framework like CUnit; we use a simple, platform-independent parser to run test scripts.

Test cases can then be added to the TESTS list in the project makefile. When calling the makefile with the test target, the executable is invoked once for every item in the list. For example, if the MyTimer project had a TESTS list like the following:

TESTS = 123\ ABC\
testscript1.txt

A call to:

make -f makefile.mak TMS_TARGET=gcc test

results in two shell calls:

../bin/gcc/debug/MyTimer.out 123 ABC
../bin/gcc/debug/MyTimer.out testscript1.txt

Make quits if the executable does not return 0 as a result. (This is the desired behavior to stop if a test error occurs. You can disable this by using the -I option when calling make.) An advantage is that you can compile, link, and test your project in one step; you can even test all depending projects using dep test at once.

Remote Systems

You can also benefit from this integrated test approach when developing for remote systems that offer some kind of remote execution. You only have to put the command for running the executable on the remote machine below the test target in the create makefile.

For example, consider a remote Linux system. First, you have to make sure that the executable (and other test files or shared libraries) on the host are available for the remote machine. The easiest way to do this is to mount your host working directory using NFS. Then, inside the user makefile, create two variables:

• REMOTE_IP, to contain the remote IP address (for example, REMOTE_IP = 192.168.2.1).
• NFS_HOME, for the mounted working directory (for example, NFS_HOME = /var/tmp/nfs).

For remote execution, we use the remote shell daemon (rshd), a background process that listens on a well-known port for remote command requests (security isn't an issue for our development purposes). Finally, inside the create makefile, the test target can look like this:

$(TESTS):
@ echo "Executing on $(REMOTE_IP)
$(APPLICATION) with parameter(s): $@"
@ rsh $(REMOTE_IP)
"exportTMS_BASE_DIR=
$(NFS_HOME)$(TMS_BASE_DIR); \
cd $(NFS_HOME)$(PRJDIR);$
(NFS_HOME)$(BINDIR)/
$(APPLICATION).out $@"

Now the test target executes the tests on the remote machine, where the stdout messages are shown on the host system. It offers a transparent and fast way to do remote and embedded development.

Conclusion

The TMS build system offers a simple, powerful way to do multiplatform development. New compilers can easily be introduced by adding new settings and create makefiles. Using abstraction layers, new hardware targets can be added without problems and have no or only minimal impact on the existing software. Tests are an integrated part of the system, improving the software quality at the component level. Finally, it lets you choose your favorite IDE and development system.

DDJ

Listing One

# a project makefile

# check input
ifndef TMS_BASE_DIR
$(error "Error: TMS_BASE_DIR not defined!")
endif
ifndef TMS_TARGET
$(error "Error: TMS_TARGET not defined!")
endif

# the directory containing this file
CURRENT_DIR = $(TMS_BASE_DIR)/MyProject/prj

# include common compiler flags, definitions and user settings
include $(TMS_BASE_DIR)/makefiles/$(TMS_TARGET)_makefile_settings.mak

# additional include directories, compiler and linker flags 
CINCS +=
CFLAGS +=
LDFLAGS +=

# objects
LIBOBJECTS= $(OBJDIR)/MyProject.o

APPOBJECTS= $(LIBOBJECTS) \ 
                $(OBJDIR)/testMyProject.o
# tests
TESTS = std

# targets
ARCHIVE     = MyProject
APPLICATION = MyProject

# include extern projects (dependencies)
include $(MAKE_DIR)/inc_MyTimer.mak

# create targets
include $(TMS_BASE_DIR)/makefiles/$(TMS_TARGET)_makefile_create.mak

# settings makefile for gcc

# check input
ifndef TMS_BASE_DIR
$(error "TMS_BASE_DIR not defined!")
endif
ifndef CURRENT_DIR
$(error "CURRENT_DIR not defined!")
endif
ifndef TMS_TARGET
$(error "TMS_TARGET not defined!")
endif
ifneq "$(TMS_TARGET)" "gcc"
$(error "TMS_TARGET not gcc!")
endif

# include user settings
#######################
include $(TMS_BASE_DIR)/makefiles/
                     user_settings/$(TMS_TARGET)_makefile_user_settings.mak
# check user settings
ifndef DEBUG
$(error "DEBUG not defined!")
endif
ifndef HWPLATFORM
$(error "HWPLATFORM not defined!")
endif
ifndef OS_TYPE
$(error "OS_TYPE not defined!")
endif
ifndef GCC
$(error "GCC not defined!")
endif

# You normally should not need to change the following; add 
# additional include-directories and compiler/linker flags in your makefile
#############################################################################
CC          = $(GCC)/gcc
AS          = $(GCC)/as
LD          = $(GCC)/gcc
AR          = $(GCC)/ar
CINCS       = -I. -I/usr/local/include -I/usr/include
ASMFLAGS    = 
CFLAGS      =  -DLINUX -DHW_X86
LDFLAGS     = 
LDPREFIX    = -l
LDSUFFFIX   = 
LDPATHPREFIX= -L

# input/output directories
##########################
MAKE_DIR = $(TMS_BASE_DIR)/makefiles/inc

REL_BINDIR    = ../bin/$(TMS_TARGET)
REL_LIBDIR    = ../lib/$(TMS_TARGET)
REL_OBJDIR    = ../obj/$(TMS_TARGET)
REL_SRCDIR    = ../src
REL_PRJDIR    = ../prj
REL_TESTDIR   = ../test
REL_EXTINCDIR = ../extinc
REL_EXTLIBDIR = ../extlib/$(TMS_TARGET)

ifeq "$(DEBUG)" "YES"
REL_BINDIR_EXP    = $(REL_BINDIR)/debug
REL_LIBDIR_EXP    = $(REL_LIBDIR)/debug
REL_OBJDIR_EXP    = $(REL_OBJDIR)/debug
REL_EXTLIBDIR_EXP = $(REL_EXTLIBDIR)/debug
else
REL_BINDIR_EXP    = $(REL_BINDIR)/release
REL_LIBDIR_EXP    = $(REL_LIBDIR)/release
REL_OBJDIR_EXP    = $(REL_OBJDIR)/release
REL_EXTLIBDIR_EXP = $(REL_EXTLIBDIR)/release
endif

BINDIR    = $(CURRENT_DIR)/$(REL_BINDIR_EXP)
LIBDIR    = $(CURRENT_DIR)/$(REL_LIBDIR_EXP)
SRCDIR    = $(CURRENT_DIR)/$(REL_SRCDIR)
OBJDIR    = $(CURRENT_DIR)/$(REL_OBJDIR_EXP)
PRJDIR    = $(CURRENT_DIR)/$(REL_PRJDIR)
TESTDIR   = $(CURRENT_DIR)/$(REL_TESTDIR)
EXTINCDIR = $(CURRENT_DIR)/$(REL_EXTINCDIR)
EXTLIBDIR = $(CURRENT_DIR)/$(REL_EXTLIBDIR_EXP)

CINCS   += -I$(SRCDIR)
LDFLAGS += $(LDPATHPREFIX)$(LIBDIR)

# shared libraries dir
ifeq "$(DEBUG)" "YES"
SHAREDLIB_DIR = $(TMS_BASE_DIR)/sharedlibs/$(TMS_TARGET)/debug
else
SHAREDLIB_DIR = $(TMS_BASE_DIR)/sharedlibs/$(TMS_TARGET)/release
endif
LDFLAGS += $(LDPATHPREFIX)$(SHAREDLIB_DIR)

# Debug/Release versions differ by compiler/linker flags
########################################################
ifeq "$(DEBUG)" "YES"
CFLAGS  += -Wall -g -O0 -D_DEBUG
LDFLAGS += -g -lc
else
CFLAGS  += -Wall -O3 -DNDEBUG
LDFLAGS += -lc
endif

# Tools
#######
ifeq "$(OS_TYPE)" "Linux"
CMD_COPY        = cp
CMD_COPY_FLAGS  = 
CMD_DEL         = rm -f
CMD_COPY_FLAGS  = 
else
$(error "OS_TYPE not Linux!")
endif

# create makefile for gcc
# additional include directories
CINCS += -I$(EXTINCDIR)
# additional linker flags 
LDFLAGS += $(LDPATHPREFIX)$(EXTLIBDIR)

# compile objects
###################
$(OBJDIR)/%.o: $(SRCDIR)/%.c
    @ echo "Compiling C file $<" 
    @ $(CC) -fpic -c $(CFLAGS) $(CINCS) -o $@ $<
$(OBJDIR)/%.o: $(SRCDIR)/%.cpp
    @ echo "Compiling C++ file $<" 
    @ $(CC) -fpic -c $(CFLAGS) $(CINCS) -o $@ $<

# targets
###################
# Target 1: create library
lib:    lib$(ARCHIVE).a

# Target 1b: create shared library
dll:    lib$(SHAREDLIB).so.1.0.1

# Target 2: create executable
exe:    $(APPLICATION) 

# Target 3:   all dependencies and this component
dep:    $(DEPENDENCIES) 

# Target 4: clean all output directories
clean:
    @ echo "Performing clean-up for $(ARCHIVE)"
    @ -$(CMD_DEL) $(OBJDIR)/*.o $(CMD_DEL_FLAGS)
    @ -$(CMD_DEL) $(LIBDIR)/*.a $(CMD_DEL_FLAGS)
    @ -$(CMD_DEL) $(LIBDIR)/*.so.* $(CMD_DEL_FLAGS)
    @ -$(CMD_DEL) $(BINDIR)/*.out $(BINDIR)/core $(CMD_DEL_FLAGS)
    @ -$(CMD_DEL) $(SHAREDLIB_DIR)/lib$(SHAREDLIB).so

# Target 5: execute tests
 .PHONY: test $(TESTS)
 test: $(TESTS) 
 $(TESTS): 
    @ echo ""
    @ echo "Executing $(APPLICATION) with parameter(s): $@"
    @ cd $(PRJDIR); /lib/ld-linux.so.2 --library-path $(SHAREDLIB_DIR) 
                                             $(BINDIR)/$(APPLICATION).out $@
#   simple version without special shared library path:
#   @ cd $(PRJDIR); $(BINDIR)/$(APPLICATION).out $@

# creation
###################
# Target 1: create library
ifneq "$(ARCHIVE)" "notavailable"
lib$(ARCHIVE).a: $(LIBOBJECTS) 
    @ echo "Creating Archive $@" 
    @ -$(CMD_DEL) $(LIBDIR)/$@ $(CMD_DEL_FLAGS)
    @ $(AR) rc $(LIBDIR)/$@ $(EXT_ARCHIVE) $(LIBOBJECTS)
else
lib$(ARCHIVE).a:
    @ echo "Library not available"
endif

# Target 1b: create shared library
ifneq "$(SHAREDLIB)" ""
lib$(SHAREDLIB).so.1.0.1: $(LIBOBJECTS) 
    @ echo "Creating Shared Library $@" 
    @ -$(CMD_DEL) $(LIBDIR)/$@ $(CMD_DEL_FLAGS)
    @ $(LD) -shared $(LIBOBJECTS) $(EXT_ARCHIVE) $(LDFLAGS) -o $(LIBDIR)/$@
    @ ln -f $(LIBDIR)/$@ $(SHAREDLIB_DIR)/lib$(SHAREDLIB).so
else
lib$(SHAREDLIB).so.1.0.1: $(LIBOBJECTS) 
    @ echo "Shared library not available"
endif

# Target 2: create executable
ifneq "$(APPLICATION)" "notavailable"
$(APPLICATION): $(APPOBJECTS) 
   @ echo "Linking $(APPLICATION)" 
   @ -$(CMD_DEL) $(BINDIR)/$(APPLICATION).out $(CMD_DEL_FLAGS)
   @ $(LD) $(APPOBJECTS) $(EXT_ARCHIVE) $(LDFLAGS) 
                            -o $(BINDIR)/$(APPLICATION).out
else
$(APPLICATION):
    @ echo "Executable not available"
endif

# Target 3: external dependecies are defined in the 'inc'-makefiles

# user makefile for gcc

# define Debug or Release version
ifndef DEBUG
DEBUG = YES
endif

# select hardware platform
ifndef HWPLATFORM
HWPLATFORM = Linux
endif

# os version of build system
ifndef OS_TYPE
OS_TYPE = Linux
endif

# location of compliler etc.
ifndef GCC
GCC = /usr/bin
endif

# inc-makefile for MyTimer

# the name of the component
MYTIMER_NAME = MyTimer

# the directory containing the project makefile
MYTIMER_DIR  = $(TMS_BASE_DIR)/shared/MyTimer/prj

# add component to dependencies list
DEPENDENCIES += $(MYTIMER_NAME)

# additional include directories
CINCS += -I$(MYTIMER_DIR)/$(REL_SRCDIR) 

# include library in linker flags
LDFLAGS += $(LDPREFIX)$(MYTIMER_NAME)$(LDSUFFFIX) 
LDFLAGS += $(LDPATHPREFIX)$(MYTIMER_DIR)/$(REL_LIBDIR_EXP) 

# if target 'dep' specified, execute the projects makefile.
# Note: the dependencies are not built recursively
$(MYTIMER_NAME):
    @ echo "Building Dependency $@" 
    @ $(MAKE) -s -f $(MYTIMER_DIR)/makefile.mak \
      TMS_TARGET=$(TMS_TARGET) $(subst dep, , $(MAKECMDGOALS))

# inc-makefile for MyTimer (switch)

ifeq "$(TMS_TARGET)" "msvc60"
MYTIMER_IMP  = MyTimer_win32
endif
ifeq "$(TMS_TARGET)" "gcc"
MYTIMER_IMP  = MyTimer_linux
endif

include $(MAKE_DIR)/inc_$(MYTIMER_IMP).mak