Python 3.3 added the venv module to its standard library, which you can use instead of installing the popular virtualenv third-party module. The new module provides many new features that enable you to create lightweight virtual environments easily, but it departs in many ways from the way the virtualenv works. Recently released Python 3.4 also added new features to the venv module. In this article, I'll examine the usage of venv in common lightweight virtual-environment scenarios.
Lightweight Virtual Environments
Lightweight virtual environments are common for Python developers who need to work on different projects with conflicting library releases or different Python versions. The new venv module is perfect for this because each lightweight virtual environment has its own Python binary. Thus, the module is especially useful when you need to maintain and test code across multiple Python versions.
A key advantages of the venv module compared with the existing third-party modules is that it provides a reliable optional isolation from system site directories, combined with a lightweight approach that reduces overall maintenance. The virtual-environment mechanisms in venv are integrated with Python, and therefore, available in all the platforms where you can run Python. As you might guess, each virtual environment in venv can have its own independent set of installed Python packages in its site directories. Thus, making it easy to control dependency management.
For example, sometimes you have an application that works with version 1.7 of a specific library, but you are also working on another application that uses version 3.5 of the library. If you upgrade the library that is common to both applications to 3.5 in
site-packages, you will need to make the necessary upgrades to the application that uses version 1.7. A lightweight virtual environment that doesn't share libraries with other environments is useful in solving dependency problems like this one.
Python 3.3 and 3.4 added the following components to support lightweight virtual environments:
pyvenv.cfgconfiguration file: The existence of this file signals the base of a virtual environment's directory tree.
pyvenvscript: Used for command-line access and administration of virtual environments. Python 3.4 added activation scripts for the csh and fish shells to the venv activation scripts.
- venv module: as described, this gives programmatic access to virtual environments. Python 3.4 added a new keyword argument to the virtual-environment creation function to specify whether the builder must ensure that the pip package manager is installed in the virtual environment.
The documentation for venv is a bit confusing because, at the time of this writing, it hadn't been completely updated with the enhancements made in Python 3.4. Thus, some comments in this article might not match the descriptions you find in the documentation. I've prepared the examples considering all the features included in Python 3.4.
Executing the pyvenv Script to Create a Virtual Environment
The easiest way to grasp the possibilities that venv offers is to learn from some simple examples. If you want to create a new lightweight virtual environment with Python 3.4 installed on Windows in
C:\Python34, you can run the following command:
C:\Python34\python C:\Python34\Tools\Scripts\pyvenv.py C:\mypythonenvs\virtualenv01
The command invokes the
pyenv.py script located in
Tools\Scripts within the base Python installation folder. The script creates the target folder
C:\mypythonenvs\virtualenv01 and installs pip by invoking
ensurepip because I didn't specify the
--without-pip option. By default, if the target folder exists, the script will raise an error and stop its execution.
C:\mypythonenvs\virtualenv01 is a new directory tree that contains Python executable files and other files that indicate it is a lightweight virtual environment. This folder includes a
pyenv.cfg configuration file specifying different options for the virtual environment and the
Scripts subfolders. In Linux and other POSIX systems, the directories would be
bin (see Figure 1). As you might guess, the directory trees for the virtual environment in each platform are the same as the layout of the Python installation. Whenever you install third-party packages into a virtual environment, the modules will be located within the initially empty
site-packages folder and the executables within either
bin, according to the system (Windows or POSIX, respectively). The existence of the
pyenv.cfg file in the folder indicates Python that there is a virtual environment.
Figure 1: The initial directory trees for virtual environments in Windows and in POSIX systems.
It is also possible to invoke the
pyenv.py script in Windows or other Python-supported platforms with the following lines.
- Windows sample:
python -m venv C:\mypythonenvs\virtualenv01
- Linux and other POSIX systems sample:
python -m venv /home/gaston/pythonenvs/virtualenv01
The following lines show the contents of the
C:\mypythonenvs\virtualenv01\pyenv.cfg file for the virtual environment created in the
C:\mypythonenvs\virtualenv01 folder with the previously shown command line in Python 3.4:
home = C:\Python34 include-system-site-packages = false version = 3.4.0
Notice that the
include-system-site-packages key is set to
false by default, unless you specify the
--system-site-packages option when invoking
pyenv.py. Thus, if you want your virtual environment to have access to the system
site-packages folder, you will need to specify that option when creating it.
The ability to upgrade a lightweight virtual environment is very useful because it makes it easy to move to newer Python versions in each virtual environment. This option is especially handy in Windows and OS X installations because the Python binary and the related DLLs are copied to the virtual-environment structure instead of being symlinked. There is a
--symlink option to force the use of symlink equivalents in Windows, but it isn't done by default. In addition, OS X framework builds will never use symlinks, so you will need to use the
--upgrade option. If you want to override the default usage of symlinks in your platform, you can use the new
--copies option included in Python 3.4.
If you create a lightweight virtual environment without symlinks and then you want to upgrade it to a newer Python version, you can easily do so by executing the
pyenv.py script with the target folder and the
--upgrade option. For example, if you started using lightweight virtual environments with Python 3.3 and then you want to upgrade some environments to Python 3.4, you can do so using the following lines:
- Windows sample:
python -m venv C:\mypythonenvs\virtualenv01 --upgrade
- Linux and other POSIX systems sample:
python -m venv /home/gaston/pythonenvs/virtualenv01 --upgrade
Activating and Deactivating Virtual Environments
Once you create your virtual environment, you can optionally run a platform-specific script to activate it. The activation prepends the virtual environment's binary folder to the path. This way, you can avoid using the full paths for the Python interpreter and the scripts. Obviously, there is also a deactivate mechanism to remove the changes in the environment variables.
The following table summarizes the commands included in a virtual environment generated with Python 3.4 to activate a virtual environment. In the examples, I use
C:\mypythonenvs\virtualenv01 as the base folder for Windows scripts and
/home/gaston/pythonenvs/virtualenv01 in POSIX scripts.
Platform and shell
Command that you must execute to activate the virtual environment
Windows Command Prompt
POSIX bash or zsh
POSIX csh or tcsh
In any POSIX shell, you just need to type
deactivate in the shell to deactivate the virtual environment because the activation scripts define a command with this name. However, at the Windows command prompt, you have to execute
C:\mypythonenvs\virtualenv01\Scripts\deactivate.bat, and in Windows PowerShell
C:\mypythonenvs\virtualenv01\Scripts\Deactivate.ps1. (Obviously, you need to replace the folder I'm using with the path to your own virtual environment for all the scripts.)
I'm sure you're wondering which mechanism is the best to remind you to deactivate a virtual environment after you activated it. Don't worry, you won't need sticky notes or other kinds of reminders. The script changes your shell's prompt to add the virtual environment folder as a prefix. For example, if you execute the
activate.bat activation script in Windows that is located in
C:\mypythonenvs\virtualenv01\Scripts, the command prompt will display
(virtualenv01) as a prefix to remind you that you have activated the virtual environment located in the
virtualenv01 folder. The following lines show how the command prompt changes:
C:\mypythonenvs\virtualenv01>C:\mypythonenvs\virtualenv01\Scripts\activate.bat (virtualenv01) C:\mypythonenvs\virtualenv01>
Once you have built and activated the desired virtual environment, any package installation you perform will only affect this environment. For example, if you use pip to install the popular simplejson package, it will affect the
site-packages folder within your activated virtual environment. The installation won't make changes to either the default Python installation or other virtual environments you may have created.