Try   HackMD

Python-virtual-environment

PRACTICAL Introduction to Python Virtual Environment

If you just start your project using python for programming and data analysis, you might come to a common problem for beginners — why your project no longer runs after working on something else for a while. You may also not know where your python packages are stored or how to manage different versions of python between projects. Well, give a sigh of relief, virtual environments have come to the rescue.
In this blogpost, I will delve into the significance of Python virtual environments, providing a step-by-step tutorial on how to create and activate the virtual environment using Python’s venv module and then install packges. I will also describe how to export the installed packages with their specific versions so as to recreate the same Python virtural environment.

Contents of this documents and quicklinks

1. Python Virtual Environments

Python is a powerful programming language that offers flexibility and efficiency, and now it has been used in a variety of applications, including web development, machine learning, and scientific computing. One of the most significant features of Python is its ability to install and use packages, which extend the functionality of this language. However, managing packages and their dependencies are quite complex, especially when working on multiple projects that require different versions of the same packages. This is where virtual environments come into play.

A Python virtual environment is a self-contained space where you can run Python code and manage its dependencies, libraries, and configurations separately from the system-wide Python installation. This isolation allows you to work on different projects with different requirements without worrying about conflicts between them. Each virtual environment has its own set of installed packages and can be activated or deactivated as needed.

There are several reasons why you might want to use a virtual environments in your Python projects:

  • Isolation: A virtual environment provides an isolated environment for the project, ensuring that the dependencies for this project don’t interfere with those for the other projects.
  • Dependency management: The virtual environment allows you to install, upgrade, or remove packages without affecting the global Python installation, making it easier to manage project-specific dependencies.
  • Portability: A virtual environment encapsulates all dependencies within a project folder, making it easy to share projects with others without worrying about conflicting dependencies.

2. Creating A Python Virtual Environment

There are two primary tools available for creating a Python virtual environments: virtualenv and venv. These tools are essentially interchangeable, but they have slight differences in their compatibility and installation methods.

  • Virtualenv supports older versions of Python and must be installed using the pip command.
  • venv is integrated into Python 3.3 and above as part of the standard library, eliminating the need for separate installation.

Here I will walk through the process of creating a Python virtual environment using venv.

2.1 Create a project folder

First, we need to create a project folder to contain your project files and the virtual environment.

$ mkdir project_A
$ cd project_A

2.2 Create a Python virtual environment

Use the venv module to create a Python virtual environment inside your project folder.

$ python3 -m venv venv_4_Proj_A

The -m flag is necessary because we’re running the venv module as a script. This command should generate a folder venv_4_Proj_A, which contains the Python virtual environment.

Running the tree -L 2 command inside this folder will show the file structures in the venv_4_Proj_A/ folder.

$ cd venv_4_Proj_A/
$ tree -L 2
.
├── bin
│   ├── Activate.ps1
│   ├── activate
│   ├── activate.csh
│   ├── activate.fish
│   ├── pip
│   ├── pip3
│   ├── pip3.10
│   ├── python -> python3
│   ├── python3 -> /Users/<NAME>/anaconda3/bin/python3
│   └── python3.10 -> python3
├── include
├── lib
│   └── python3.10
│       └── site-packages
└── pyvenv.cfg

The most important things in this folder are the site-packages sub-folder within lib and the activate scripts within bin directories.

  • site-packages is the place where all third-party packages you install will be stored
  • activate is the script that you should run before using the Python virtual environment.

2.3 Activate the Python virtual environment

To activate the Python virtual environment, you should run the command:

$ source bin/activate
(venv_4_Proj_A) (base) 

After activation, the Python virtual environment’s name (here it is venv_4_Proj_A) will appear below the command, indicating that you are now working inside the Python virtual environment.

1-activate

We can verify this by checking the version of Python and where it is stored via the command below.

$ which python3

This should display something like this:

2-python-version

Two notes should be addressed here:

  • the version of Python3 being referenced here is the one within the Python virtual environment. Any packages and libraries installed with the pip command should be contained within this virtual environment, and inaccessible outside of it.
  • we should distinguish between a Python project folder and a Python virtual environment folder.
    • the project folder holds the source code for your project, while the virtual environment folder contains the Python interpreter, packages, and tools (that is why I created a folder venv_4_Proj_A to store everything relevant to the virtual environment).
    • to maintain clarity and organization, it’s highly recommanded to keep these two folders separate and never mix your project files within the virtual environment folder.

3. Installing Packages in A Python Virtual Environment

Inside the Python virtual environment, you can install packages using pip, which is the package installer for Python.

Before installing any packages or libraries, let's check the pre-installed packages after we created the Python virtual environment.

3.1 Check pre-installed packages

Running the command below we can get a list of the pre-installed packages as shown below.

$ pip list

3-1st-pip-list

Sometimes you might get the output info like [notice] A new release of pip available.. If so, you can upgrade pip to the latest version via running the command.

$ pip install --upgrade pip

3.2 Install Python packages

Running the command below to install three packages (numpy, pandas, and matplotlib) inside the Python virtual environment.

$ pip install numpy pandas matplotlib

This should display something like this:

4-install-packages

If you want to install a specific version of a package, such as the version 2.2.0 for pandas, you should specify the version number of this package in the command line.

$ pip install pandas==2.2.0

You can also install a range of versions using comparison operators.

$ pip install 'pandas<2.2'

After the installation of three packaged mentioned above, we will get the updated list of the all installed packages and their specific versions via the command below.

$ pip list

5-2nd-pip-list

4. Reproducing A Python Virtual Environment

It’s common to reproduce a Python virtual environment to ensure consistency across different development environments or group members.

4.1 Freeze dependencies

You can export a list of all installed packages along with their versions to a requirements file using the command below.

$ pip freeze > requirements.txt

This command creates a requirements.txt file containing all the packages and their exact versions as shown below.

contourpy==1.3.1
cycler==0.12.1
fonttools==4.55.3
kiwisolver==1.4.8
matplotlib==3.10.0
numpy==2.2.1
packaging==24.2
pandas==2.2.3
pillow==11.0.0
pyparsing==3.2.1
python-dateutil==2.9.0.post0
pytz==2024.2
six==1.17.0
tzdata==2024.2

4.2 Recreate the Python virtual environment from the requirements.txt file

You can distribute the requirements.txt file to your group members, and they can later use this file to recreate the same Python virtual environment and install all packages listed in the requirements.txt file via the commands below.

$ mkdir project_B/
$ cd project_B/
$ python3 -m venv venv_4_Proj_B/
$ source venv_4_Proj_B/bin/activate
$ pip install -r requirements.txt

5. Deactivating and Deleting A Python Virtual Environment

Once you are done working with your Python virtual environment, you can deactivate it or delete it.

Running the command below returns you to the system-wide Python environment.

$ deactivate

To delete the Python virtual environment, simply remove its directory.

$ rm -rf project_A/venv_4_Proj_A

6. Summary

Python virtual environments are essential for managing dependencies and preventing conflicts between projects. By creating isolated environments for each project, you can ensure that each application has its specific packages and versions. This enables you to maintain a clear and efficient development environment, allowing you to focus on building great software without worrying about dependency conflicts.

Besides using venv module, you can also use other methods (virtualenv, pipenv,conda, etc.) to create the Python virtual environment. You can refer to this WEBPAGE for more details.