Channels ▼
RSS

.NET

Lightweight Virtual Environments in Python 3.4


Working with the EnvBuilder Class

The venv module provides a simple API that allows you to customize the creation of the lightweight virtual environments. For example, if you want to easily create virtual environments and control which packages are installed after their creation, you can use the API to automate this procedure. Instead of creating scripts in the different platforms and shells, you can use the Python language to customize the way venv works for your needs.

The scripts use the venv.EnvBuilder class to create the lightweight virtual environments with the provided configuration. It is easy to understand the keyword arguments that the EnvBuilder class accepts because they mimic the different options I've been explaining. However, you need to pay attention to the defaults because EnvBuilder specifies a default False value for all the arguments. The following table shows the instantiation arguments for the class, and the equivalent option in the pyenv.py script for the different Boolean values:

Argument

pyenv.py option for False

pyenv.py option for True

system-site-packages

Default

--system-site-packages

clear

Default

--clear

symlinks

Default

--symlinks

upgrade

Default

--upgrade

with_pip

--without-pip

Default

So, if you want to create an instance of the EnvBuilder class with the default values used when you invoke the pyenv.py script without additional options, you must set the with_pip argument to True:

import venv
builder = venv.EnvBuilder(with_pip=True)

Then, you can call the create method with the desired target directory or folder for a new lightweight virtual environment as an argument. The following line creates a new virtual environment in C:\mypythonenvs\virtualenv03 in a Windows system. As you might guess, if something goes wrong, the create method will raise an exception.

builder.create('C:\\mypythonenvs\\virtualenv03')

The source code for the create method is easy to understand. The following lines show the different methods called to create a lightweight virtual environment.

def create(self, env_dir):
    env_dir = os.path.abspath(env_dir)
    context = self.ensure_directories(env_dir)
    self.create_configuration(context)
    self.setup_python(context)
    if self.with_pip:
        self._setup_pip(context)
    if not self.upgrade:
        self.setup_scripts(context)
        self.post_setup(context)

The good news is that you can extend the venv.EnvBuilder class and override any of the following methods called within the create method:

  • ensure_directories: Creates the directories or folders for the new virtual environment and returns a context object that holds the paths in this environment. The next methods receive the context object as an argument to have access to the paths.
  • create_configuration: Receives the context object returned by ensure_directories. Creates the pyenv.cfg configuration file and saves the values for the keys.
  • setup_python: Receives the context object returned by ensure_directories. Sets up the Python executable in the virtual environment. The method decides whether to symlink or copy the files based on the operating system and the arguments specified in the create method that might force symlinking.
  • _setup_pip: The create method calls _setup_pip if you set with_pip to True when creating the EnvBuilder instance. _setup_pip receives the context object returned by ensure_directories. This method installs or upgrades pip in a virtual environment. The Python documentation for venv provides an example of extending the EnvBuilder class to install pip when you create virtual environments. Because Python 3.4 includes the call to _setup_pip, you don't need to extend EnvBuilder to achieve that goal.
  • setup_scripts: Receives the context object returned by ensure_directories. Installs the default scripts into the new virtual environment. You can override this method to prevent the default installation or to specify a different location for the scripts.
  • post_setup: Receives the context object returned by ensure_directories. post_setup is just a hook for post-setup modifications of the new virtual environment. You can override the method to perform the installation of additional packages or scripts.

The following lines show a simple example of how to extend venv.EnvBuilder to override the post_setup method. You just need to add the necessary code to setup the desired packages in the new virtual environment in the post_setup method. The main method code is borrowed from the venv module and the only thing that changes is that the code creates an instance of the CustomizedEnvBuilder class instead of the venv.EnvBuilder class. This way, the virtual environment creation will use the overridden post_setup method.

import os
import os.path
import sys
import venv


class CustomizedEnvBuilder(venv.EnvBuilder):

    def post_setup(self, context):
        """
        Set up any packages which need to be pre-installed into the
        environment being created.

        :param context: The information for the environment creation request
                        being processed.
        """
        # Add the necessary code to setup required packages in the new virtual environment
        pass

def main(args=None):
    compatible = True
    if sys.version_info < (3, 3):
        compatible = False
    elif not hasattr(sys, 'base_prefix'):
        compatible = False
    if not compatible:
        raise ValueError('This script is only for use with Python >= 3.3')
    else:
        import argparse

        parser = argparse.ArgumentParser(prog=__name__,
                                         description='Creates virtual Python '
                                                     'environments in one or '
                                                     'more target '
                                                     'directories.',
                                         epilog='Once an environment has been '
                                                'created, you may wish to '
                                                'activate it, e.g. by '
                                                'sourcing an activate script '
                                                'in its bin directory.')
        parser.add_argument('dirs', metavar='ENV_DIR', nargs='+',
                            help='A directory to create the environment in.')
        parser.add_argument('--system-site-packages', default=False,
                            action='store_true', dest='system_site',
                            help='Give the virtual environment access to the '
                                 'system site-packages dir.')
        if os.name == 'nt':
            use_symlinks = False
        else:
            use_symlinks = True
        group = parser.add_mutually_exclusive_group()
        group.add_argument('--symlinks', default=use_symlinks,
                           action='store_true', dest='symlinks',
                           help='Try to use symlinks rather than copies, '
                                'when symlinks are not the default for '
                                'the platform.')
        group.add_argument('--copies', default=not use_symlinks,
                           action='store_false', dest='symlinks',
                           help='Try to use copies rather than symlinks, '
                                'even when symlinks are the default for '
                                'the platform.')
        parser.add_argument('--clear', default=False, action='store_true',
                            dest='clear', help='Delete the contents of the '
                                               'environment directory if it '
                                               'already exists, before '
                                               'environment creation.')
        parser.add_argument('--upgrade', default=False, action='store_true',
                            dest='upgrade', help='Upgrade the environment '
                                               'directory to use this version '
                                               'of Python, assuming Python '
                                               'has been upgraded in-place.')
        parser.add_argument('--without-pip', dest='with_pip',
                            default=True, action='store_false',
                            help='Skips installing or upgrading pip in the '
                                 'virtual environment (pip is bootstrapped '
                                 'by default)')
        options = parser.parse_args(args)
        if options.upgrade and options.clear:
            raise ValueError('you cannot supply --upgrade and --clear together.')
        builder = CustomizedEnvBuilder(system_site_packages=options.system_site,
                             clear=options.clear,
                             symlinks=options.symlinks,
                             upgrade=options.upgrade,
                             with_pip=options.with_pip)
        for d in options.dirs:
            builder.create(d)

if __name__ == '__main__':
    rc = 1
    try:
        main()
        rc = 0
    except Exception as e:
        print('Error: %s' % e, file=sys.stderr)
    sys.exit(rc)

Conclusion

The ability to customize the venv.EnvBuilder class using Python code makes it possible to tailor the creation of virtual environments to your needs. The venv module includes features focused for the task of using different packages to generate isolated environments in Python. This support for multiple different lightweight virtual environments in Python is a truly useful feature that has been enhanced in the latest Python 3.4.


Gaston Hillar is a frequent contributor to Dr. Dobb's.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video