Channels ▼
RSS

Design

Lightweight Virtual Environments in Python 3.4


Python 3.3 added the venv module to its standard library, which you can use instead of installing the popular virtualenv third-party module. The new module provides many new features that enable you to create lightweight virtual environments easily, but it departs in many ways from the way the virtualenv works. Recently released Python 3.4 also added new features to the venv module. In this article, I'll examine the usage of venv in common lightweight virtual-environment scenarios.

Lightweight Virtual Environments

Lightweight virtual environments are common for Python developers who need to work on different projects with conflicting library releases or different Python versions. The new venv module is perfect for this because each lightweight virtual environment has its own Python binary. Thus, the module is especially useful when you need to maintain and test code across multiple Python versions.

A key advantages of the venv module compared with the existing third-party modules is that it provides a reliable optional isolation from system site directories, combined with a lightweight approach that reduces overall maintenance. The virtual-environment mechanisms in venv are integrated with Python, and therefore, available in all the platforms where you can run Python. As you might guess, each virtual environment in venv can have its own independent set of installed Python packages in its site directories. Thus, making it easy to control dependency management.

For example, sometimes you have an application that works with version 1.7 of a specific library, but you are also working on another application that uses version 3.5 of the library. If you upgrade the library that is common to both applications to 3.5 in site-packages, you will need to make the necessary upgrades to the application that uses version 1.7. A lightweight virtual environment that doesn't share libraries with other environments is useful in solving dependency problems like this one.

Python 3.3 and 3.4 added the following components to support lightweight virtual environments:

  • pyvenv.cfg configuration file: The existence of this file signals the base of a virtual environment's directory tree.
  • pyvenv script: Used for command-line access and administration of virtual environments. Python 3.4 added activation scripts for the csh and fish shells to the venv activation scripts.
  • venv module: as described, this gives programmatic access to virtual environments. Python 3.4 added a new keyword argument to the virtual-environment creation function to specify whether the builder must ensure that the pip package manager is installed in the virtual environment.

The documentation for venv is a bit confusing because, at the time of this writing, it hadn't been completely updated with the enhancements made in Python 3.4. Thus, some comments in this article might not match the descriptions you find in the documentation. I've prepared the examples considering all the features included in Python 3.4.

Executing the pyvenv Script to Create a Virtual Environment

The easiest way to grasp the possibilities that venv offers is to learn from some simple examples. If you want to create a new lightweight virtual environment with Python 3.4 installed on Windows in C:\Python34, you can run the following command:

C:\Python34\python C:\Python34\Tools\Scripts\pyvenv.py C:\mypythonenvs\virtualenv01

The command invokes the pyenv.py script located in Tools\Scripts within the base Python installation folder. The script creates the target folder C:\mypythonenvs\virtualenv01 and installs pip by invoking ensurepip because I didn't specify the --without-pip option. By default, if the target folder exists, the script will raise an error and stop its execution.

C:\mypythonenvs\virtualenv01 is a new directory tree that contains Python executable files and other files that indicate it is a lightweight virtual environment. This folder includes a pyenv.cfg configuration file specifying different options for the virtual environment and the Include, Lib, Lib\site-packages and Scripts subfolders. In Linux and other POSIX systems, the directories would be include, lib, lib/python3.4/site-packages (lib/pythonX.Y/site-packages), and bin (see Figure 1). As you might guess, the directory trees for the virtual environment in each platform are the same as the layout of the Python installation. Whenever you install third-party packages into a virtual environment, the modules will be located within the initially empty site-packages folder and the executables within either Scripts or bin, according to the system (Windows or POSIX, respectively). The existence of the pyenv.cfg file in the folder indicates Python that there is a virtual environment.

Python
Figure 1: The initial directory trees for virtual environments in Windows and in POSIX systems.

It is also possible to invoke the pyenv.py script in Windows or other Python-supported platforms with the following lines.

  • Windows sample: python -m venv C:\mypythonenvs\virtualenv01
  • Linux and other POSIX systems sample: python -m venv /home/gaston/pythonenvs/virtualenv01

The following lines show the contents of the C:\mypythonenvs\virtualenv01\pyenv.cfg file for the virtual environment created in the C:\mypythonenvs\virtualenv01 folder with the previously shown command line in Python 3.4:

home = C:\Python34
include-system-site-packages = false
version = 3.4.0

Notice that the include-system-site-packages key is set to false by default, unless you specify the --system-site-packages option when invoking pyenv.py. Thus, if you want your virtual environment to have access to the system site-packages folder, you will need to specify that option when creating it.

The ability to upgrade a lightweight virtual environment is very useful because it makes it easy to move to newer Python versions in each virtual environment. This option is especially handy in Windows and OS X installations because the Python binary and the related DLLs are copied to the virtual-environment structure instead of being symlinked. There is a --symlink option to force the use of symlink equivalents in Windows, but it isn't done by default. In addition, OS X framework builds will never use symlinks, so you will need to use the --upgrade option. If you want to override the default usage of symlinks in your platform, you can use the new --copies option included in Python 3.4.

If you create a lightweight virtual environment without symlinks and then you want to upgrade it to a newer Python version, you can easily do so by executing the pyenv.py script with the target folder and the --upgrade option. For example, if you started using lightweight virtual environments with Python 3.3 and then you want to upgrade some environments to Python 3.4, you can do so using the following lines:

  • Windows sample: python -m venv C:\mypythonenvs\virtualenv01 --upgrade
  • Linux and other POSIX systems sample: python -m venv /home/gaston/pythonenvs/virtualenv01 --upgrade

Activating and Deactivating Virtual Environments

Once you create your virtual environment, you can optionally run a platform-specific script to activate it. The activation prepends the virtual environment's binary folder to the path. This way, you can avoid using the full paths for the Python interpreter and the scripts. Obviously, there is also a deactivate mechanism to remove the changes in the environment variables.

The following table summarizes the commands included in a virtual environment generated with Python 3.4 to activate a virtual environment. In the examples, I use C:\mypythonenvs\virtualenv01 as the base folder for Windows scripts and /home/gaston/pythonenvs/virtualenv01 in POSIX scripts.

Platform and shell

Command that you must execute to activate the virtual environment

Windows Command Prompt

C:\mypythonenvs\virtualenv01\Scripts\activate.bat

Windows PowerShell

C:\mypythonenvs\virtualenv01\Scripts\Activate.ps1

POSIX bash or zsh

source /home/gaston/pythonenvs/virtualenv01/bin/activate

POSIX csh or tcsh

source /home/gaston/pythonenvs/virtualenv01/bin/activate.csh

POSIX fish

. /home/gaston/pythonenvs/virtualenv01/bin/activate.fish

In any POSIX shell, you just need to type deactivate in the shell to deactivate the virtual environment because the activation scripts define a command with this name. However, at the Windows command prompt, you have to execute C:\mypythonenvs\virtualenv01\Scripts\deactivate.bat, and in Windows PowerShell C:\mypythonenvs\virtualenv01\Scripts\Deactivate.ps1. (Obviously, you need to replace the folder I'm using with the path to your own virtual environment for all the scripts.)

I'm sure you're wondering which mechanism is the best to remind you to deactivate a virtual environment after you activated it. Don't worry, you won't need sticky notes or other kinds of reminders. The script changes your shell's prompt to add the virtual environment folder as a prefix. For example, if you execute the activate.bat activation script in Windows that is located in C:\mypythonenvs\virtualenv01\Scripts, the command prompt will display (virtualenv01) as a prefix to remind you that you have activated the virtual environment located in the virtualenv01 folder. The following lines show how the command prompt changes:

C:\mypythonenvs\virtualenv01>C:\mypythonenvs\virtualenv01\Scripts\activate.bat
(virtualenv01) C:\mypythonenvs\virtualenv01>

Once you have built and activated the desired virtual environment, any package installation you perform will only affect this environment. For example, if you use pip to install the popular simplejson package, it will affect the site-packages folder within your activated virtual environment. The installation won't make changes to either the default Python installation or other virtual environments you may have created.


Related Reading


More Insights






Currently we allow the following HTML tags in comments:

Single tags

These tags can be used alone and don't need an ending tag.

<br> Defines a single line break

<hr> Defines a horizontal line

Matching tags

These require an ending tag - e.g. <i>italic text</i>

<a> Defines an anchor

<b> Defines bold text

<big> Defines big text

<blockquote> Defines a long quotation

<caption> Defines a table caption

<cite> Defines a citation

<code> Defines computer code text

<em> Defines emphasized text

<fieldset> Defines a border around elements in a form

<h1> This is heading 1

<h2> This is heading 2

<h3> This is heading 3

<h4> This is heading 4

<h5> This is heading 5

<h6> This is heading 6

<i> Defines italic text

<p> Defines a paragraph

<pre> Defines preformatted text

<q> Defines a short quotation

<samp> Defines sample computer code text

<small> Defines small text

<span> Defines a section in a document

<s> Defines strikethrough text

<strike> Defines strikethrough text

<strong> Defines strong text

<sub> Defines subscripted text

<sup> Defines superscripted text

<u> Defines underlined text

Dr. Dobb's encourages readers to engage in spirited, healthy debate, including taking us to task. However, Dr. Dobb's moderates all comments posted to our site, and reserves the right to modify or remove any content that it determines to be derogatory, offensive, inflammatory, vulgar, irrelevant/off-topic, racist or obvious marketing or spam. Dr. Dobb's further reserves the right to disable the profile of any commenter participating in said activities.

 
Disqus Tips To upload an avatar photo, first complete your Disqus profile. | View the list of supported HTML tags you can use to style comments. | Please read our commenting policy.
 

Video