ConMan
ConMan (short for "configuration manager") is an open-source Python package I wrote that manages and provides an easy-to-use interface for complex hierarchical configurations. The source is available on GitHub. It is designed to help when you have many configuration files in various formats or if you want to use etcd as a distributed configuration store. In both cases, it exposes all the configuration information as a Python dictionary.
ConMan doesn't deal with command-line arguments or environment variables because:
- I wanted to focus on the more complex case of hierarchical configuration data shared by multiple programs. Command-line arguments in particular are designed for a single run of a program, and environment variables are typically used for flat key-value pairs (although the values could be something like an XML or JSON string with internal structure).
- There are excellent tools to deal with command-line arguments (argparse), and environment variables are already exposed as a dictionary (os.environ).
Let's go over ConMan's code to understand what it does and how to use it. Figure 1 shows the directory structure:
Figure 1: Directory structure.
ConManBase
The conman_base.py file contains the ConManBase class. This class serves as a base class to both ConManFile and ConManEtcd. It provides the private _conf attribute that actually stores the configuration, a __repr__() method, which is the text representation of Python objects (for example, it's called when "print" is used), and a dictionary interface that forwards get requests to the internal _conf and raises an exception if a program attempts to modify the configuration:
class ConManBase(dict):
def __init__(self):
self._conf = {}
def __getitem__(self, k):
return self._conf.__getitem__(k)
def __setitem__(self, k, v):
raise NotImplementedError
def __repr__(self):
return repr(self._conf)
ConManFile
The conman_file.py class ConManFile deals with configuration files of several formats. It is a little more complicated.
This section has the necessary imports (including the base class) and the list of supported file formats: .ini, .json, and .yaml. I chose not to support XML because it's such a crusty and not human-friendly format that I stopped using it years ago. But it is still very common (especially in the Java world), so it may be reasonable to add XML support later.
import os import json import yaml from ConfigParser import SafeConfigParser from conman.conman_base import ConManBase FILE_TYPES = 'ini json yaml'.split()
The ConManFile class subclasses ConManBase to get all its goodies. The __init__() method (called when an object is instantiated) accepts a list of configuration filenames. It initializes the base class, initializes an internal list of configuration files, then calls the add_config_file() for each of the input config files.
class ConManFile(ConManBase):
def __init__(self, config_files=()):
ConManBase.__init__(self)
self._config_files = []
for config_file in config_files:
self.add_config_file(config_file)
The add_config_file() is the only public method of ConManFile. It enables the user to dynamically add configuration files in addition to the list of configuration files provided when instantiating a ConManFile object. It also allows periodically refreshing existing config files.
There are many ways to tell add_config_file() about a configuration file. Here are the rules for the various arguments it accepts:
- The filename contains the path to the config file. The filename may also be read from an environment variable. Either
filenameis notNoneorenvironment_Variableis notNone, but not both. - If
base_diris notNone, then it will be combined with the config filename to create an absolute path. - If a file type is provided, it is used to determine how to parse the config file. If no file type is provided, ConMan will try to guess by the extension using the
_guess_file_type()method (described later).
If the rules are violated, add_config_file() raises an exception. Listing Two show its workings.
Listing Two
def add_config_file(self,
filename=None,
env_variable=None,
base_dir=None,
file_type=None):
if filename is None and env_variable is None:
raise Exception('filename and env_variable are both None')
if filename is not None and env_variable is not None:
raise Exception('filename and env_variable are both not None')
if filename in self._config_files:
raise Exception('filename is already in the config file list')
if env_variable:
filename = os.environ[env_variable]
if base_dir:
filename = os.path.join(base_dir, filename)
if not os.path.isfile(filename):
raise Exception('No such file: ' + filename)
if not file_type:
file_type = self._guess_file_type(filename)
At this point, we have a filename and possibly a file type, too. If there is a file type, ConMan tries to process the file by calling the _process_file() method. If it succeeds, we're done. If it fails or if no file type could be guessed in the first place, then add_config_file() simply iterates over all its known file types and tries to process the input file as each one of them until one succeeds. If the input file failed to be processed as any of the file types, a 'Bad config file' exception is raised, as shown in Listing Three.
Listing Three
file_types = set(FILE_TYPES)
if file_type:
try:
return self._process_file(filename, file_type)
except:
# Remove failed file_type from set of file types to try
if file_type in file_types:
file_types.remove(file_type)
# If no file type can be guessed or guess failed try all parsers
for file_type in file_types:
try:
return self._process_file(filename, file_type)
except:
pass
raise Exception('Bad config file: ' + filename)
The _guess_file_type() just gets the extension of the input filename and compares it to the supported file types (yaml can have two extensions). It returns the file type or None if there's no match (Listing Four).
Listing Four
def _guess_file_type(self, filename):
ext = os.path.splitext(filename)[1][1:]
return dict(yml='yaml',
yaml='yaml',
json='json',
ini='ini').get(ext, None)
The _process_file() method is just a delegator. Based on the file type, it constructs the corresponding method name and invokes it. The actual process methods follow a uniform naming scheme of _process_<file_type>_file(). The various process methods parse the file and then populate the _conf dictionary (defined in the base class), as in Listing Five.
Listing Five
def _process_file(self, filename, file_type):
process_func = getattr(self, '_process_%s_file' % file_type)
process_func(filename)
def _process_ini_file(self, filename):
parser = SafeConfigParser()
parser.read(filename)
for section_name in parser.sections():
self._conf[section_name] = {}
section = self._conf[section_name]
for name, value in parser.items(section_name):
section[name] = value
def _process_json_file(self, filename):
self._conf.update(json.load(open(filename)))
def _process_yaml_file(self, filename):
self._conf.update(yaml.load(open(filename)))
ConManEtcd
The conman_etcd.py file contains the ConManEtcd class, which allows you to add the content of various etcd keys as nested dictionaries, then expose them as one unified dictionary (via the ConManBase base class).
The first section is just some necessary imports. The etcd import is the etcd client module.
import functools import etcd import time from conman.conman_base import ConManBase
etcd is a distributed server, so it often responds with varying latencies or may even be unreachable temporarily (you can control it, of course, with proper networking and redundancy). A @thrice decorator is an easy way to interact in a logical and controlled way, but it may exhibit transient failure and require a few tries to get right. The idea is that a callable function or method that is decorated with @thrice will automatically be called up to three times. If any of the calls succeed and no exception is raised, it returns immediately with the result. If the first or second calls fail, then it waits a bit to give the service some time to recover and tries again. If the call fails three times, then @thrice gives up and just re-raises the exception, which propagates to the caller (Listing Six). The ConManEtcd class utilizes it when it connects via the etcd.Client.
Listing Six
def thrice(delay=0.5):
def decorated(f):
@functools.wraps(f)
def wrapped(*args, **kwargs):
for i in xrange(3):
try:
return f(*args, **kwargs)
except Exception:
if i == 2:
raise
time.sleep(delay)
return wrapped
return decorated
The ConManEtcd class exposes two public methods (in addition to dictionary read-only access to the configuration information via the base class). The add_key() reads a key (and recursively all its sub-keys and values), and populates the _conf dictionary of the base class. The refresh() method can refresh a single key or all the keys currently managed. This allows the ability to periodically ensure that the configuration is up-to-date, which is important for long-running systems (Listing Seven).
Listing Seven
class ConManEtcd(ConManBase):
def __init__(self, host='127.0.0.1', port=4001, allow_reconnect=True):
ConManBase.__init__(self)
self._connect(host, port, allow_reconnect)
@thrice()
def _connect(self, host, port, allow_reconnect):
self.client = etcd.Client(
host=host,
port=port,
allow_reconnect=allow_reconnect)
def _add_key_recursively(self, target, key, etcd_result):
if key.startswith('/'):
key = key[1:]
if etcd_result.value:
target[key] = etcd_result.value
else:
target[key] = {}
target = target[key]
for c in etcd_result.children:
k = c.key.split('/')[-1]
self._add_key_recursively(target, k, c)
def add_key(self, key):
etcd_result = self.client.read(key, recursive=True, sorted=True)
self._add_key_recursively(self._conf, key, etcd_result)
def refresh(self, key=None):
keys = [key] if key else self._conf.keys()
for k in keys:
self.add_key(k)
Using ConMan
To use ConMan in your code, simply install the package and its requirements. Get it from GitHub, then run:
pip install conman
For usage examples, refer to the tests. In the source code, etcd_test_util.py includes helper functions to manage a local etcd server.
Conclusion
In this article, I discussed the spectrum of configuration options ranging from simple programs up to large distributed systems. I also introduced a Python package designed to help manage multiple configuration files and distributed configuration for more intricate use cases. I hope my package helps with managing configuration on systems composed of multiple configurable components, or inspires ideas for creating you own.
Gigi Sayfan specializes in cross-platform object-oriented programming in C/C++/ C#/Python/Java with emphasis on large-scale distributed systems.
