Python - package management

Virtual environment

https://realpython.com/python-virtual-environments-a-primer/
https://docs.python.org/3/tutorial/venv.html

Can install packages in the virtual environment using command lines. For example, writing linux scripts setup.sh and teardown.sh to create env and install packages via conda.

setup.sh

#!/bin/bash

# Show conda information
conda update -y conda
conda info
conda env list

# Create environment with specific python version and activate
conda create --name scraper_work_env python=3.9 -y
conda activate scraper_work_env

# Install required packages into the virtual environment
conda install -c conda-forge --name scraper_work_env git -y

# Show the installed packages
conda list

teardown.sh

#!/bin/bash

conda deactivate
conda env remove --name scraper_work_env

setup.py

https://docs.python.org/3/distutils/setupscript.html
https://www.educative.io/answers/what-is-setuppy

The setup script is the centre of all activity in building, distributing, and installing modules using the Distutils. The main purpose of the setup script is to describe your module distribution to the Distutils.

When there are multiple folders in the projects (each is a module), need to list them into the package keywords. (For example, /pages and /scraper)

List necessary packages for this project into the install_requires keyword.

Solving errors

CondaValueError: prefix already exists

Happens when creating the virtual environment. The reason is that the env has been deactivated but the folder still exists.

Solution
https://stackoverflow.com/questions/40180652/condavalueerror-value-error-prefix-already-exists

Due the command --force to force creating environment does not always work, need to delete the env folder instead.

# Print the base envs
ENV_BASE=$(conda-env list | awk '/base/ { print $NF }')
echo $ENV_BASE
# Delete the target env
rm -rf "$ENV_BASE/envs/scraper_work_env"

‘ImportError: No module named …' When running pytest to do unit tests

https://thewebdev.info/2022/04/09/how-to-fix-path-issue-with-pytest-importerror-no-module-named-error-with-python/

Solution
Step 1
Make sure pytest is installed. Include pytest in the install_requires keyword in the setup.py

Run the command pip install . to execute setup.py in the current running virtual environment.

Step 2

Set the PYTHONPATH variable to the Python executable path, to run pytest on the tests folder with python -m to Python add the current directory in the PYTHONPATH environment variable.

Run this command to run pytest:

python -m pytest tests/

Instead of running the command directly:

pytest

tags: python