Channels ▼
RSS

Design

Program Configuration in Python


ConMan

ConMan (short for "configuration manager") is an open-source Python package I wrote that manages and provides an easy-to-use interface for complex hierarchical configurations. The source is available on GitHub. It is designed to help when you have many configuration files in various formats or if you want to use etcd as a distributed configuration store. In both cases, it exposes all the configuration information as a Python dictionary.

ConMan doesn't deal with command-line arguments or environment variables because:

  • I wanted to focus on the more complex case of hierarchical configuration data shared by multiple programs. Command-line arguments in particular are designed for a single run of a program, and environment variables are typically used for flat key-value pairs (although the values could be something like an XML or JSON string with internal structure).
  • There are excellent tools to deal with command-line arguments (argparse), and environment variables are already exposed as a dictionary (os.environ).

Let's go over ConMan's code to understand what it does and how to use it. Figure 1 shows the directory structure:

ConMan
Figure 1: Directory structure.

ConManBase

The conman_base.py file contains the ConManBase class. This class serves as a base class to both ConManFile and ConManEtcd. It provides the private _conf attribute that actually stores the configuration, a __repr__() method, which is the text representation of Python objects (for example, it's called when "print" is used), and a dictionary interface that forwards get requests to the internal _conf and raises an exception if a program attempts to modify the configuration:

class ConManBase(dict):
    def __init__(self):
        self._conf = {}

    def __getitem__(self, k):
        return self._conf.__getitem__(k)

    def __setitem__(self, k, v):
        raise NotImplementedError

    def __repr__(self):
        return repr(self._conf)

ConManFile

The conman_file.py class ConManFile deals with configuration files of several formats. It is a little more complicated.

This section has the necessary imports (including the base class) and the list of supported file formats: .ini, .json, and .yaml. I chose not to support XML because it's such a crusty and not human-friendly format that I stopped using it years ago. But it is still very common (especially in the Java world), so it may be reasonable to add XML support later.

import os
import json
import yaml
from ConfigParser import SafeConfigParser
from conman.conman_base import ConManBase

FILE_TYPES = 'ini json yaml'.split()

The ConManFile class subclasses ConManBase to get all its goodies. The __init__() method (called when an object is instantiated) accepts a list of configuration filenames. It initializes the base class, initializes an internal list of configuration files, then calls the add_config_file() for each of the input config files.

class ConManFile(ConManBase):
    def __init__(self, config_files=()):
        ConManBase.__init__(self)
        self._config_files = []
        for config_file in config_files:
            self.add_config_file(config_file)

The add_config_file() is the only public method of ConManFile. It enables the user to dynamically add configuration files in addition to the list of configuration files provided when instantiating a ConManFile object. It also allows periodically refreshing existing config files.

There are many ways to tell add_config_file() about a configuration file. Here are the rules for the various arguments it accepts:

  • The filename contains the path to the config file. The filename may also be read from an environment variable. Either filename is not None or environment_Variable is not None, but not both.
  • If base_dir is not None, then it will be combined with the config filename to create an absolute path.
  • If a file type is provided, it is used to determine how to parse the config file. If no file type is provided, ConMan will try to guess by the extension using the _guess_file_type() method (described later).

If the rules are violated, add_config_file() raises an exception. Listing Two show its workings.

Listing Two

      def add_config_file(self,
                        filename=None,
                        env_variable=None,
                        base_dir=None,
                        file_type=None):
        if filename is None and env_variable is None:
            raise Exception('filename and env_variable are both None')

        if filename is not None and env_variable is not None:
            raise Exception('filename and env_variable are both not None')

        if filename in self._config_files:
            raise Exception('filename is already in the config file list')

        if env_variable:
            filename = os.environ[env_variable]

        if base_dir:
            filename = os.path.join(base_dir, filename)

        if not os.path.isfile(filename):
            raise Exception('No such file: ' + filename)

        if not file_type:
            file_type = self._guess_file_type(filename)

At this point, we have a filename and possibly a file type, too. If there is a file type, ConMan tries to process the file by calling the _process_file() method. If it succeeds, we're done. If it fails or if no file type could be guessed in the first place, then add_config_file() simply iterates over all its known file types and tries to process the input file as each one of them until one succeeds. If the input file failed to be processed as any of the file types, a 'Bad config file' exception is raised, as shown in Listing Three.

Listing Three

        file_types = set(FILE_TYPES)
        if file_type:
            try:
                return self._process_file(filename, file_type)
            except:
                # Remove failed file_type from set of file types to try
                if file_type in file_types:
                    file_types.remove(file_type)

        # If no file type can be guessed or guess failed try all parsers
        for file_type in file_types:
            try:
                return self._process_file(filename, file_type)
            except:
                pass

        raise Exception('Bad config file: ' + filename)

The _guess_file_type() just gets the extension of the input filename and compares it to the supported file types (yaml can have two extensions). It returns the file type or None if there's no match (Listing Four).

Listing Four

    def _guess_file_type(self, filename):
        ext = os.path.splitext(filename)[1][1:]
        return dict(yml='yaml',
                    yaml='yaml',
                    json='json',
                    ini='ini').get(ext, None)

The _process_file() method is just a delegator. Based on the file type, it constructs the corresponding method name and invokes it. The actual process methods follow a uniform naming scheme of _process_<file_type>_file(). The various process methods parse the file and then populate the _conf dictionary (defined in the base class), as in Listing Five.

Listing Five

    def _process_file(self, filename, file_type):
        process_func = getattr(self, '_process_%s_file' % file_type)
        process_func(filename)

    def _process_ini_file(self, filename):
        parser = SafeConfigParser()
        parser.read(filename)
        for section_name in parser.sections():
            self._conf[section_name] = {}
            section = self._conf[section_name]
            for name, value in parser.items(section_name):
                section[name] = value

    def _process_json_file(self, filename):
        self._conf.update(json.load(open(filename)))

    def _process_yaml_file(self, filename):
        self._conf.update(yaml.load(open(filename)))

ConManEtcd

The conman_etcd.py file contains the ConManEtcd class, which allows you to add the content of various etcd keys as nested dictionaries, then expose them as one unified dictionary (via the ConManBase base class).

The first section is just some necessary imports. The etcd import is the etcd client module.

import functools
import etcd
import time
from conman.conman_base import ConManBase

etcd is a distributed server, so it often responds with varying latencies or may even be unreachable temporarily (you can control it, of course, with proper networking and redundancy). A @thrice decorator is an easy way to interact in a logical and controlled way, but it may exhibit transient failure and require a few tries to get right. The idea is that a callable function or method that is decorated with @thrice will automatically be called up to three times. If any of the calls succeed and no exception is raised, it returns immediately with the result. If the first or second calls fail, then it waits a bit to give the service some time to recover and tries again. If the call fails three times, then @thrice gives up and just re-raises the exception, which propagates to the caller (Listing Six). The ConManEtcd class utilizes it when it connects via the etcd.Client.

Listing Six

def thrice(delay=0.5):
    def decorated(f):
        @functools.wraps(f)
        def wrapped(*args, **kwargs):
            for i in xrange(3):
                try:
                    return f(*args, **kwargs)
                except Exception:
                    if i == 2:
                        raise
                    time.sleep(delay)
        return wrapped
    return decorated

The ConManEtcd class exposes two public methods (in addition to dictionary read-only access to the configuration information via the base class). The add_key() reads a key (and recursively all its sub-keys and values), and populates the _conf dictionary of the base class. The refresh() method can refresh a single key or all the keys currently managed. This allows the ability to periodically ensure that the configuration is up-to-date, which is important for long-running systems (Listing Seven).

Listing Seven

class ConManEtcd(ConManBase):
    def __init__(self, host='127.0.0.1', port=4001, allow_reconnect=True):
        ConManBase.__init__(self)
        self._connect(host, port, allow_reconnect)

    @thrice()
    def _connect(self, host, port, allow_reconnect):
        self.client = etcd.Client(
            host=host,
            port=port,
            allow_reconnect=allow_reconnect)

    def _add_key_recursively(self, target, key, etcd_result):
        if key.startswith('/'):
            key = key[1:]
        if etcd_result.value:
            target[key] = etcd_result.value
        else:
            target[key] = {}
            target = target[key]
            for c in etcd_result.children:
                k = c.key.split('/')[-1]
                self._add_key_recursively(target, k, c)

    def add_key(self, key):
        etcd_result = self.client.read(key, recursive=True, sorted=True)
        self._add_key_recursively(self._conf, key, etcd_result)

    def refresh(self, key=None):
        keys = [key] if key else self._conf.keys()
        for k in keys:
            self.add_key(k)

Using ConMan

To use ConMan in your code, simply install the package and its requirements. Get it from GitHub, then run:

pip install conman

For usage examples, refer to the tests. In the source code, etcd_test_util.py includes helper functions to manage a local etcd server.

Conclusion

In this article, I discussed the spectrum of configuration options ranging from simple programs up to large distributed systems. I also introduced a Python package designed to help manage multiple configuration files and distributed configuration for more intricate use cases. I hope my package helps with managing configuration on systems composed of multiple configurable components, or inspires ideas for creating you own.


Gigi Sayfan specializes in cross-platform object-oriented programming in C/C++/ C#/Python/Java with emphasis on large-scale distributed systems.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.