Distributed Compilation

Vadim works as chief systems architect for AVISTAR Corp. He can be contacted at [email protected].

If you are working on a sizable C, C++, Objective-C, or Objective-C++ project and compilation time is an issue, you should consider building a compile farm and running a distributed compile on several machines simultaneously. distcc (http://distcc.samba.org/) is a tool that lets you do just this. With distcc, you can set up a cluster of machines to compile a single source-code tree, dramatically reducing compilation times. Your particular speed improvement depends, of course, on the number of available machines on the network. Essentially, all you do is install the distcc daemon on each of these machines, thereby building a "compile farm." When you compile from a client machine, it distributes the work (per source file) to multiple machines in the farm.

distcc works as a wrapper around the compiler. It passes source files through the preprocessor (thus eliminating all local dependencies) and sends the resulting preprocessed source to one of the build farm machines over the network via TCP. The build machine compiles the received files and sends back resulting object files to the client. Since the files are already preprocessed, the build farm compiler does not need to have any header files or know any macro definitions that you might need to compile the original source. Consequently, you don't need to install third-part libraries and header files on each compile farm machine—you only need them on your client machine.

All linking is done on the client machine. To parallelize the build process, it is generally recommended that you use the GNU make -j option. When -j is specified, GNU make tries to compile several files in parallel (assuming there are no dependencies among them). With distcc, these files are sent to different compile farm machines and compiled in parallel.

Automating Build Farms

Unfortunately, simple distcc deployments require that you know the names of all machines in the compile farm. Usually, you need to list them in the DISTCC_HOSTS environment variable. If some hosts become unavailable or new hosts are added, you need to update your local configuration accordingly. In larger organizations, this little problem quickly turns into a big one.

As it turns out, Apple provides a distributed build feature with its development tools. In particular, Apple automatically populates the list of machines in a compile farm via its OpenTalk (formerly known as "Rendezvous") technology, which provides automatic discovery of computers, devices, and services on IP networks. In this article, I show how to implement an Apple-like distributed build in Linux, giving you all the benefits of a distributed build farm with automatic discovery but using inexpensive hardware and free software. In doing so, I use Red Hat Linux 9, although with minimal changes, the setup could be used with other Linux and UNIX platforms as well.

Setting Up Compilation Servers

First, you must enable OpenTalk on your Linux boxes. There are several implementations of OpenTalk for Linux, including one from Apple (http://developer.apple.com/macosx/). OpenTalk is, in fact, Apple's name for zero-configuration networking Zeroconf (http://www.zeroconf.org/). For this project, I'm using an open-source, cross-platform implementation of Zeroconf for Linux called "Howl" from Porchdog Software (http://www.porchdogsoft.com/ products/howl/).

Start by installing Howl; I use the Red Hat Package Manager (RPM) provided by Porchdog. The Howl distribution includes several daemons, but only one—mDNSResponder (dynamic publication and service discovery daemon)—is of interest here. After it is installed, make sure it automatically starts on boot:

chkconfig —add mDNSResponder
chkconfig —level=345 mDNSResponder on

Note that howl-0.9.5-1.i386.rpm has a bug in the file /etc/init.d/mDNSResponder that specifies the wrong path to the mDNSResponder binary. Also, it does not specify the configuration file's location. Consequently, you should apply the patch in Listing One to /etc/init.d/mDNSResponder after installation.

Next, you need to create a configuration file with a list of the services on this machine, which you want to advertise via OpenTalk. For example, /etc/mDNSResponder.conf might look like Example 1, which advertises SSH and DISTCC services on this machine. Also in Example 1, rover is a local machine name; replace it with the actual name of your host. Since all machines in a distcc compile farm must have the same architecture and name for the compiler, I called my service _distcc-RH9 to show that it is a Red Hat 9 I386 machine. You can run several farms in the same organization by using different names, for example "_distcc-Solaris8," "_distcc-FreeBSD5," and so on.

Start mDNSResponder with:

service mDNSResponder start

If you have a Mac OS X machine on the local network, you can test how this works by using a Terminal application. (Go to "File->Connect To Server" menu and you should see the name of your Linux machine in the list of SSH hosts; see Figure 1.)

At this point, you need to install distcc-server. (Again, I used Red Hat Linux 9 RPMs.) Make sure it starts on boot:

chkconfig —add distcc
chkconfig —level=345 distcc on

and start it with:

service distcc start

Repeat these steps for each compilation farm machine.

Setting Up Compilation Clients

To use OpenTalk on clients, you need to install and run the mDNSResponder daemon. Follow the same steps as in the previous section; but this time, do not include the configuration line for distcc in mDNSResponder.conf unless your client machine also acts as a compilation server.

Next, compile a small program that looks up distcc servers using OpenTalk. To compile it, you need to install the Howl development libraries and headers on one machine (you can do this on one machine and copy the resulting binary to all others). I used the howl-devel RPM package to install the Howl libraries. Listing Two is the program you need to compile. You then compile mDNSlookup.c with the command:

cc -I/usr/include/howl mDNSlookup.c -o mDNSlookup -lhowl -lpthread -lrt

and install it in your executable path (for example, to /usr/local/bin). Finally, you can test to see what machines you can discover:

[root@rover root]# mDNSlookup _ssh._tcp 500
rover _distcc-RH9._tcp. local. 192.0.2.140 3632
zembla _distcc-RH9._tcp. local. 192.0.2.139 3632

In this case, the system found two machines: zembla and rover with IP addresses 192.0.2.139 and 192.0.2.140, respectively. Both machines are offering compilation services using distcc on Red Hat 9, listening on TCP port 3632.

Setting Up The Compiler

The final step involves making use of discovered hosts. While there are several ways to do this, I've chosen the compiler shell wrapper approach because, on multiuser systems, users can control individually when distcc should be used. There is no need to modify project Makefiles to take advantage of distributed builds. Users can easily switch distributed builds off and on (some tools, imake, for example, do not work well when distcc use is enabled).

You need to install the scripts in Listing Three in some directory, which is in the binary executable path of all users (for example, /usr/local/bin). You may want to tweak the scripts in Listing Three(a) and Three(b) (distcc_nhosts and distcc_wrapper), adjusting the variables at the beginning. MDNS_TIMEOUT defines how long to wait (in milliseconds) for OpenTalk responses. ADD_HOSTS_BEFORE and ADD_HOSTS_AFTER let you add some nonOpenTalk-enabled compile farm hosts to list hosts that will be used. The value of SERVICE_NAME should match the one you set earlier in /etc/mDNSResponder.conf on the compile farm hosts. To verify that it works, you may temporarily uncomment the line, setting the DISTCC_VERBOSE variable. This produces compilation-time debug output from distcc, showing what hosts are used. CCACHE (http://ccache.samba.org/) is another tool that could be used together with distcc to speed up compilation even more. It works well together with distcc and provides a noticeable compilation speed up, even when using a compilation farm. If you have it installed (you need to install it just on the client machine), replace this line in Listing Three(e) (distcc_wrapper):

exec distcc /usr/bin/c++ $*

with

exec ccache distcc /usr/bin/c++ $*

Now you're set. To start your first distributed compile, just enter:

make -j `1distcc_nhosts`

Conclusion

Once set up, no changes need to be made on the client side when new machines are added to or removed from a farm. If some machine becomes temporarily unavailable (for instance due to network connectivity issues), this will be immediately discovered and clients will make no attempt to use it.

If a number of machines in the compile farm—as well as a number of client machines—are noticeable, it might make sense to prepare RPM packages for both the client and server, which eliminates any manual configuration except RPM installation. mDNSResponder.conf could be generated automatically by the RPM post-install script (with the name of the machine taken from the hostname command and the service name built from the machine architecture name and gcc version).

A minor limitation to this approach is that distcc service is advertised whenever a machine is up, regardless of whether the distcc daemon is running or not. This can be corrected by a simple modification of the distcc daemon to register mDNSResponder on startup and unregister on exit. Howl provides an example of such code in the source code for the mDSNPublish example.

DDJ

Listing One

--- mDNSResponder.old 2004-06-02 17:16:03.000000000 -0700
+++ /etc/init.d/mDNSResponder 2004-06-02 17:45:47.000000000 
                                                    -0700@@ -12,7 +12,7 @@
  # processname: mDNSResponder
  # config:
  -OTHER_MDNSRD_OPTS=""
  +OTHER_MDNSRD_OPTS="-f /etc/mDNSResponder.conf"

  # Source function library.
 . /etc/init.d/functions
@@ -24,7 +24,7 @@
  start() {
         echo -n $"Starting mDNSResponder... "
-        /usr/local/bin/mDNSResponder $OTHER_MDNSRD_OPTS
+        /usr/bin/mDNSResponder $OTHER_MDNSRD_OPTS
         RETVAL=$?
         echo      return $RETVAL

/*
* Copyright 2003, 2004, 2004 Porchdog Software. All rights reserved.
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above
* copyright notice, this list of conditions and the following
* disclaimer.
* 2. Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following
* disclaimer in the documentation and/or other materials
* provided with the distribution.
* THIS SOFTWARE IS PROVIDED BY PORCHDOG SOFTWARE ``AS IS'' AND ANY
* EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE HOWL PROJECT OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
* USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
* ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
* POSSIBILITY OF SUCH DAMAGE.
* The views and conclusions contained in the software and
* documentation are those of the authors and should not be
* interpreted as representing official policies, either expressed or
* implied, of Porchdog Software.
* -------------------------------------------------------------------
* mDNSlookup is quick hack, based on mDNSBrowse example from HOWL package.
* Main changes:
* 1. introduced timeout parameter
* 2. less-verbose output (do not report some packets, do not print TXT records)
* To compile on Linux use following command:
* cc -I/usr/include/howl mDNSlookup.c -o mDNSlookup -lhowl -lpthread -lrt
* Vadim Zaliva
*/
#include <howl.h>
#include <stdio.h>
#include <time.h>

static sw_result HOWL_API lookup_resolver(
   sw_discovery_resolve_handler handler,
   sw_discovery discovery,
   sw_discovery_resolve_id id,
   sw_const_string name,
   sw_const_string type,
   sw_const_string domain,
   sw_ipv4_address address,
   sw_port port,
   sw_const_string text_record_string,
   sw_octets text_record,
   sw_ulong text_record_len,
   sw_opaque extra)
{
   char name_buf[16];
   sw_discovery_stop_resolve(discovery, id);
   printf("%s %s %s %s %d\n", name, type, domain,
   sw_ipv4_address_name(address, name_buf, 16), port
   );
   return SW_OKAY;
}
static sw_result HOWL_API lookup_browser(
   sw_discovery_browse_handler handler,
   sw_discovery discovery,
   sw_discovery_browse_id id,
   sw_discovery_browse_status status,
   sw_const_string name,
   sw_const_string type,
   sw_const_string domain,
   sw_opaque extra)
{
   sw_discovery_resolve_id rid;
   if(status==SW_DISCOVERY_BROWSE_ADD_SERVICE)
   {
   if(sw_discovery_resolve(discovery, name, type, domain, NULL, 
                                lookup_resolver, NULL, &rid) != SW_OKAY)
      {
         fprintf(stderr, "resolve failed\n");
         return -1;
      }
   }
   return SW_OKAY;
}
main(int argc, char **argv)
{
   sw_discovery discovery ;
   sw_result result ;
   sw_discovery_browse_id id ;
   sw_salt salt ;
   sw_ulong timeout ;
   if(argc == 3)
   {
      char *e;
      timeout=strtol(argv[2], &e, 10);
      if(*e)
      {
         fprintf(stderr, "Invalid timeout value. 
                                     Should be number in milliseconds!\n");
         fprintf(stderr, "Usage: mDNSlookup <type> [timeout]\n");
         return -1;
      }
   } else if(argc != 2)
   {
      fprintf(stderr, "Usage: mDNSlookup <type> [timeout]\n");
      return -1;
   } else  
      timeout = 0L;
   if((result = sw_discovery_init(&discovery)) != SW_OKAY)
   {
      fprintf(stderr, "sw_discovery_init failed: %d\n", result);
      return -1;
   }
   if(sw_discovery_browse(discovery, argv[1], NULL, NULL, 
                              lookup_browser, NULL, &id) != SW_OKAY)
   {
      fprintf(stderr, "sw_discovery_browse_services failed: %d\n", result);
      return -1;
   }
   if(sw_discovery_salt(discovery, &salt)!= SW_OKAY)
   {
      fprintf(stderr, "sw_discovery_salt failed: %d\n", result);
      return -1;
   }
   if(timeout == 0L)
   {
      /* No timeout - wait forever. */
      sw_discovery_run(discovery);
   } else
   {
   /*
      Correct code should be:

      while(timeout != 0L)
      sw_salt_step(salt, &timeout);

      but sw_salt_step() does not modify 
      timeout value (this bug was reported to authors).
   */
   /*
   So, we have to do a workaround, using
   POSIX 1003.1b Section 14 (Clocks and Timers) API:
   */
   struct timespec start;
   clock_gettime(CLOCK_REALTIME, &start);
   while(1)
   {
         struct timespec now;
         clock_gettime(CLOCK_REALTIME, &now);
         long waited = (1000L*now.tv_sec+now.tv_nsec/1000000L) -
                (1000L*start.tv_sec+start.tv_nsec/1000000L);
         if(waited>=timeout)
            break;
         sw_ulong remain=timeout-waited;
         sw_salt_step(salt, &remain);
      }
   }
   return 0;
}

 distcc_on

#!/bin/sh
# Enables distributed compilation using DistCC
ln -s /usr/local/bin/distcc_wrapper ~/bin/c++
ln -s /usr/local/bin/distcc_wrapper ~/bin/cc
ln -s /usr/local/bin/distcc_wrapper ~/bin/g++
ln -s /usr/local/bin/distcc_wrapper ~/bin/gcc

 distcc_off

#!/bin/sh
# Disables distributed compilation using DistCC
rm -f ~/bin/c++ ~/bin/cc ~/bin/g++ ~/bin/gcc

 distcc_status

#!/bin/sh
# Shows if distributed compilation using DistCC is enabled
if [ -f ~/bin/c++ ] ; then 
   echo "distcc is ON"
else
   echo "discc is OFF"
fi

distcc_nhosts

#!/bin/sh
# Prints number of hosts availiabe in compile farm
# Output of this script intended to be used with -j option of GNU make.
# By default we return number twice as much as hosts found in compile
# farm, plus one (to handle empty compile farm).
MDNS_TIMEOUT=200
SERVICE_NAME=_distcc-RH9._tcp
ADD_NHOSTS=1
MULTIPLY_NHOSTS=2
echo $(($ADD_NHOSTS+$((`mDNSlookup "$SERVICE_NAME" $MDNS_TIMEOUT | wc -l`+$MULTIPLY_NHOSTS))))

 distcc_wrapper

#!/bin/sh
#
# DistCC wrapper to use OpenTalk to discover compile
# farm machines. Should not be run directly by user.
MDNS_TIMEOUT=200
ADD_HOSTS_BEFORE=
ADD_HOSTS_AFTER=
SERVICE_NAME=_distcc-RH9._tcp
DYNAMIC_HOSTS=`mDNSlookup "$SERVICE_NAME" $MDNS_TIMEOUT | 
                                        awk '{printf "%s:%s ",$4,$5;}'`
export DISTCC_HOSTS="$ADD_HOSTS_BEFORE $DYNAMIC_HOSTS $ADD_HOSTS_AFTER"
export DISTCC_NHOSTS=`echo "$DISTCC_HOSTS" | wc -w`
echo $DISTCC_HOSTS
#export DISTCC_VERBOSE=1
exec distcc /usr/bin/c++ $*