Channels ▼
RSS

Tools

A Build System for Complex Projects: Part 3


The NetBeans Helper

The NetBeans Helper class is responsible for implementing all the code that is NetBeans-specific. The generic build_system_generator.py script is using this helper to generate all the NetBeans project files (inside the nbproject directory) for each project and a project group that includes all the generated project. Let's take a closer look at this class. The first thing it does is import some useful system modules and then import the BaseHelper and Template classes from the build_system_generator module (as well as the 'title' function for debugging purposes):


#!/usr/bin/env python
import os, sys, string
from pprint import pprint as pp

sys.path.insert(0, os.path.join(os.path.abspath(os.path.dirname(__file__)), '../'))
from build_system_generator import (BaseHelper,
                                    Template,
                                    title)

Then the Helperclass is defined. This is the class that the build system generator module is using to customize the build system generation for each specific target (NetBeans 6.x in this case). The __init__() method accepts the templates_dir, which is the path to the root of the templates directory used to generate all the build files. It also initializes the separator ('/') and line separator ('\n') to Unix values to make the generated files fit well in their intended environment. The skip_dir is used to tell the recursive drill-down code that looks for projects in sub-directories to ignore directories called 'nbproject' (which is the special sub-directory used by NetBeans to store the build files). The other methods this class implements are: get_templates(), prepare_substitution_dict(), and generate_workspace_files().


class Helper(BaseHelper):
  """NetBeans6 helper
  """
  def __init__(self, templates_dir):
    BaseHelper.__init__(self, templates_dir)
    self.sep = '/'
    self.linesep = '\n'
    self.skip_dir = 'nbproject'

  def get_templates(self, template_type):
    ...
    
  def prepare_substitution_dict(self, ...):
    ...
    def generate_workspace_files(self, name, root_path, projects):
    ...

get_templates()

The get_templates()method is pretty simple. For each build file there is a corresponding template file. These template files are just skeleton of real build files, with some place holders. You will see all the template files soon enough. The get_templates() method just iterates over all the template files (located in the nbproject) and adds a template for the Makefile in the project directory itself. For each such build a file a Template object is generated. Finally the list of Template objects is returned.


  def get_templates(self, template_type):
    result = [Template(os.path.join(self.templates_dir,
                                    template_type,
                                    'Makefile'),
                                    'Makefile',
                                    template_type)]

    nb_project = os.path.join(self.templates_dir, template_type, 'nbproject')
    assert os.path.isdir(nb_project)
    for f in os.listdir(nb_project):
      project_file_template = os.path.join(nb_project, f)
      if not os.path.isfile(project_file_template):
        continue

      filename = os.path.join(nb_project, f)
      relative_path = '/'.join(['nbproject', f])
      result.append(Template(filename, relative_path, template_type))
    return result

prepare_substitution_dict()

This method is the heart of NetBeans6_Helper class. It is responsible for creating a substitution dictionary that contains all the values to be substituted into the templates of each build file. This is not so trivial because some place holders are supposed to be replaced by dynamic content that is generated on the fly. In addition, as you saw earlier NetBeans has quite a few build files. The prepare_substitution_dict() method has several nested function to assist in prepare the substitution dictionary for each one of them. The nested functions are:

  • prepare_makefile() for generating the Makefile-Debug.mk and Makefile-Release.mk files
  • prepare_configurations_xml() for generating configurations.xml
  • prepare_project_properties() for generating project.properties
  • prepare_project_xml() for generating project.xml

The substitution dict for the project's main Makefile is empty because it is a generic file that doesn't have any place holder and the substitution dict for the Makefile-impl.mk file contains only the name of the project so no helper function is necessary. Here is the code of the method (without the nested functions). It accepts a long list of arguments that the various nested functions use to generate the proper values. The operating system and the dynamic library extension are also determined here. This method is called multiple times with different template names (each template_name corresponds to a build file) and prepare_substitution_dict() calls the proper nested function or generates the dict directly (for Makefile and MakeFile-Impl.mk).


def prepare_substitution_dict(self,
						project_name,
					project_type,
						project_file_template,
						project_dir,
						libs_dir,
						dependencies,
						source_files,
						header_files,
						platform):
if platform.startswith('darwin'):
  operating_system = 'MacOSX'
  ext = 'dylib'
elif platform.startswith('linux'):
  operating_system = 'Linux'
  ext = 'so'

temaplate_name = os.path.basename(project_file_template)
if temaplate_name ==  'Makefile':
  return {}

if temaplate_name ==  'Makefile-Debug.mk':
  return prepare_makefile('Debug', operating_system)

if temaplate_name ==  'Makefile-Release.mk':
  return prepare_makefile('Release', operating_system)

if temaplate_name == 'Makefile-impl.mk':
  return dict(Name=os.path.basename(project_dir))

if temaplate_name == 'configurations.xml':
  return prepare_configurations_xml(operating_system)

if temaplate_name == 'project.properties':
  return prepare_project_properties()

if temaplate_name == 'project.xml':
  return prepare_project_xml(dependencies)

assert False, 'Invalid project file template: ' + temaplate_name
return {}

Now, let's examine one of nested functions. I chose the prepare_makefile() function because it is not trivial. The keys in its substitution dictionary are: 'ObjectFiles', 'CompileFiles', 'LinkCommand', 'LDLIBSOPTIONS', 'BuildSubprojects', 'CleanSubprojects', 'OperatingSystem' and 'DynamicLibExtension'. Some of these are simple strings like 'OperatingSystem' and 'DynamicLibExtension'. Others are much more complicated like 'CompileFiles', which is a list of compile commands where each command itself requires a template with substitution values such as 'File', 'CompileFlag', 'Platform' and 'FPIC'. The link command depends on the project type and ldliboptions depends on the platform. Here is the code:


    def prepare_makefile(conf, operating_system):
      compile_flag = '-g' if conf == 'Debug' else '-O2'

      d = dict(Name=project_name)
      object_file_template = '	${OBJECTDIR}/%s.o \\\n'
      object_files = ''
      for f in source_files:
        f = os.path.splitext(os.path.basename(f))[0]
        object_files += object_file_template % f

      # Flag for dynamic libraries
      fpic = '-fPIC  ' if project_type == 'dynamic_lib' else ''

      # Get rid of last forward slash
      if len(object_files) > 2:
        object_files = object_files[:-3]
      d['ObjectFiles'] = object_files

      compile_file_template = \
        '$${OBJECTDIR}/${File}.o: ${File}.cpp \n' + \
        '\t$${MKDIR} -p $${OBJECTDIR}\n' + \
        '\t$$(COMPILE.cc) ${CompileFlag} -I../.. ${FPIC}-o $${OBJECTDIR}/${File}.o ${File}.cpp\n\n'

      t = string.Template(compile_file_template)
      compile_files = ''
      for f in source_files:
        f = os.path.splitext(os.path.basename(f))[0]
        text = t.substitute(dict(File=f,
                                 CompileFlag=compile_flag,
                                 Platform=platform,
                                 FPIC=fpic))
        compile_files += text

      # Get rid of the last two \n\n.
      compile_files = compile_files[:-2]
      d['CompileFiles'] = compile_files

      link_command = ''
      if project_type == 'dynamic_lib':
        if platform.startswith('darwin'):
          link_command = '${LINK.cc} -dynamiclib -install_name lib%s.dylib' % project_name
        else:
          assert platform.startswith('linux')
          link_command = '${LINK.c} -shared'
        d['LinkCommand'] = link_command

      ldlibsoptions = ''
      if dependencies != []:
        ldliboption_template = '../../hw/%s/dist/%s/GNU-%s/lib%s.a'

        ldlibsoptions = ' '.join([ldliboption_template % \
                        (dep.name, conf, operating_system, dep.name)
                        for dep in dependencies])
        if operating_system == 'Linux':
          ldlibsoptions += ' -ldl'
      d['LDLIBSOPTIONS'] = ldlibsoptions

      build_subproject_template = '\tcd ../../hw/%s && ${MAKE}  -f Makefile CONF=%s'
      clean_subproject_template = build_subproject_template + ' clean'

      build_list = [build_subproject_template % (dep.name, conf) for dep in dependencies]
      clean_list = [clean_subproject_template % (dep.name, conf) for dep in dependencies]
      d['BuildSubprojects'] = '\n'.join(build_list)
      d['CleanSubprojects'] = '\n'.join(clean_list)
      d['OperatingSystem'] = operating_system
      d['DynamicLibExtension'] = ext

      return d

Note, that there are better ways to accomplish this task. There are several third-party template languages like Genshi, Mako, Tempita and Jinja. These template engines can handle the nested templates that prepare_makefile() generates manually in a much more natural way. The code could have been much shorter and concise. I made a deliberate decision to use only standard Python libraries in the interest of keeping the scope of this project limited. Choosing a particular template language/engine would have made the code shorter, but required the reader to understand an additional language and might antagonize fans of other template languages.

The other prepare_XXX() nested functions are all very similar to make_makefile() although some of them generate XML files and another one generate a properties file (INI file like).

generate_workspace_files()

This method is responsible for generating the project groups in the user account. The reason the method is called generate_workspace_files() is that the method is defined in the generic Helper base class and NetBeans6_Helper is just overriding it. So, the NetBeans-specific term "Project Group" is not used here. The code itself is pretty simple. It either creates or updates the proper .properties files that dictate the contents of the project groups as explained earlier:


  def generate_workspace_files(self, name, root_path, projects):
    """Generate a NetBeans project group for all the generated projects

    """
    base_path = \
      '~/.netbeans/6.7/config/Preferences/org/netbeans/modules/projectui'
    base_path = os.path.expanduser(base_path)

    if not os.path.exists(base_path):
      os.makedirs(base_path)

    # Create a project group
    groups_path = os.path.join(base_path, 'groups')
    if not os.path.exists(groups_path):
      os.makedirs(groups_path)

    text = """\
name=%s
kind=directory
path=file\:%s"""
    group_filename = os.path.join(groups_path, name + '.properties')
    open(group_filename, 'w').write(text % (name, root_path))

    # Make it the active project
    text = 'active=' + name
    open(os.path.join(base_path, 'groups.properties'), 'w').write(text)

The NetBeans Project Templates

The substitution dictionaries are very important of course, but they can't do much by themselves. Each build file is generated by substituting the values from the proper dictionary into the proper template file.

As you recall NetBeans can build three types of projects: static library, dynamic library and a program. for each one of them there are templates of all the build files. A few templates are the same for some or all project types, so an identical copy is kept for each one. The templates are organized in the following file system structure:


project_templates
  NetBeans_6
    dynamic_lib
      Makefile
      nbproject
        configurations.xml
        Makefile-Debug.mk
        Makefile-Impl.mk
        Makefile-Release.mk
        project.properties
        project.xml
    program
      Makefile
      nbproject
        ...
    static_lib
      Makefile
      nbproject
        ...

This regular structure mimics the structure of the build files inside a project directory and allows the generic part of ibs to apply the substitution dicts to the templates blindly and end up with the correct build file in the correct place. Note, the project.properties file that wasn't mentioned earlier. This is an empty file that doesn't seem to have a role in C++ projects, but I keep it there to be consistent with NetBeans.

To create the template files I simply took the various NetBeans build files and replaced anything that was project-specific (like the source files or list of dependencies) with a place holder.Let's examine a couple of template files. Here is the main part of the Makefile-Debug.mk of the 'program' project type:


# Link Libraries and Options
LDLIBSOPTIONS=${LDLIBSOPTIONS}

# Build Targets
.build-conf: $${BUILD_SUBPROJECTS} dist/Debug/GNU-${OperatingSystem}/${name}

dist/Debug/GNU-${OperatingSystem}/${name}: $${BUILD_SUBPROJECTS}

dist/Debug/GNU-${OperatingSystem}/${name}: $${OBJECTFILES}
	$${MKDIR} -p dist/Debug/GNU-${OperatingSystem}
	$${LINK.cc} -o dist/Debug/GNU-${OperatingSystem}/${name} $${OBJECTFILES} $${LDLIBSOPTIONS}

${CompileFiles}

# Subprojects
.build-subprojects:
${BuildSubprojects}

# Clean Targets
.clean-conf: $${CLEAN_SUBPROJECTS}
	$${RM} -r build/Debug
	$${RM} dist/Debug/GNU-${OperatingSystem}/${name}

# Subprojects
.clean-subprojects:
${CleanSubprojects}

The placeholder are expressions of the form ${Place holder}. This is the format used by the string.Template class. Unfortunately, this convention is used by make files a lot too for environment variable, defined symbols and make variables. So, when a $ sign is part of the make file it is escaped by additional $ sign. For example, $${MKDIR} will not be treated as a place holder by the ibs.

Here is a simpler template of the project.xml file. It is just a bunch of XML with two place holders for the name of the project and its dependencies:

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://www.netbeans.org/ns/project/1"> <type>org.netbeans.modules.cnd.makeproject</type> <configuration> <data xmlns="http://www.netbeans.org/ns/make-project/1"> <name>${Name}</name> <make-project-type>0</make-project-type> ${MakeDepProjects} </data> </configuration> </project>


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video