Installing Python modules on Bianca - UPPMAX

Bianca is a research system dedicated for analyzing sensitive personal data from large-scale molecular experiments. Bianca is part of the SNIC-SENS project.

Since Bianca is designed to handle sensitive personal data security is a key aspect of the configuration and features restricted access and lack of direct Internet access to and from the cluster. This make it particularly difficult to use common tools to maintain python modules or software installations.

Here are some tips and advises on different way to approach the problem.

• Installing Python modules on Bianca - UPPMAX
  • User installs via pip
    • Download the necessary modules - docs.
    • Install the modules on Bianca
    • Setup
  • User installs via python-venv
  • Using conda
    • Create conda environment with selected Python version
  • Contacts:

User installs via pip

pip documentation

Download the necessary modules - docs.

You need to collect the required python modules on a computer with Internet access then transfer the files to Bianca for installation. The most convenient way is to

1. Login on transit.uppmax.uu.se
2. Mount your project wharf folder i.e. mount_wharf sens2016001
3. Enter the wharf folder on transit and download the necessary modules.

On transit (computer with Internet)

# Load python from modules
$ module load python/3.12.7

# Create folder where to download the dependencies for the module
$ mkdir mdownloads
$ cd mdownloads
$ python3 -m pip download module_name

Install the modules on Bianca

On Bianca (computer without Internet)

$ module load python/3.12.7
$ export PYTHONUSERBASE=/proj/snicxxxxx/pythontools

$ python3 -m pip install -U -f ./ --no-index  module_name.whl

Setup

# make sure you have `PYTHONPATH` and `PATH` setup accordingly.
$ export PYTHONUSERBASE=/proj/snicxxxxx/pythontools
$ export PATH=/proj/snicxxxxx/pythontools/bin:$PATH

Look at this example in details.
Alternatively, check this guide and/or the video at the end.

User installs via python-venv

Note that this might be done better by using python's venv
https://docs.python.org/3/tutorial/venv.html

Also: User practices: Installation and use of python packages on UPPMAX
Just tar the installation folder and untar it on Bianca and use the same python version.

TL;DR - installing https://github.com/nanoporetech/bonito

On Rackham (source machine)

Collect the module package following the instructions from the previous step.

Copy mdownload.tar to Bianca

On Bianca (target machine)

$ module load python/3.9.5

$ cd /proj/PATH/
$ tar -xvf mdownload.tar
$ python3 -m venv bonito
$ source bonito/bin/activate

$(bonito) cd mdownload
$(bonito) python3 -m pip install -f ./ --no-index bonito-xxx.xx.whl
$(bonito) $ bonito -v
bonito 0.3.6

Using conda

Note: On Bianca there is nearly complete mirror of most common channels, which means that you might be able to manage your conda installations the usual way.

The methods above will not allow you to use python version that is not provided by the system setup i.e.

Available Python versions on UPPMAX (2020.12.21)

$ module spider python

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  python:
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
     Versions:
        python/2.7.6
        python/2.7.9
        python/2.7.11
        python/2.7.15
        python/3.3
        python/3.3.1
        python/3.4.3
        python/3.5.0
        python/3.6.0
        python/3.6.8
        python/3.7.2

To use another specific version you need to use conda to create new environment with selected Python version.

Conda User Guide@UPPMAX
Conda-pack - command line tool for creating relocatable conda environments.

Hint: You can install packages with pip that will be packed with the conda environment…

Create conda environment with selected Python version

On Rackham or transit /or other linux computer/
(*computer with Internet and same amd64 architecture *)

# activate conda
$ source /sw/apps/conda/latest/rackham_stage/etc/profile.d/conda.sh

# redirect the installation to folder in your project
$ export CONDA_ENVS_PATH=/proj/snicxxxxx/conda_enviroments

# create environment with selected Python version
$ conda create -n my_env python=x.x module_name
$ conda activate my_env
$(my_env) 
# make your installation then pack your environment

# Follow the guide at https://conda.github.io/conda-pack/
$(my_env) conda install -c conda-forge conda-pack
# Deactivate the environment
$(my_env) conda deactivate
# Pack environment my_env into out_name.tar.gz
$ conda pack -n my_env -o out_name.tar.gz
...

On Bianca (computer without Internet)

# activate conda
$ source /sw/apps/conda/latest/rackham_stage/etc/profile.d/conda.sh
$ export CONDA_ENVS_PATH=/proj/snicxxxxx/conda_enviroments

# Unpack environment into directory `my_env`
#$ cd $CONDA_ENVS_PATH
$ mkdir -p $CONDA_ENVS_PATH/my_env
$ tar -xzf my_env.tar.gz -C $CONDA_ENVS_PATH/my_env

# Activate the environment. This adds `my_env/bin` to your path
$ source $CONDA_ENVS_PATH/my_env/bin/activate

# Cleanup prefixes from in the active environment.
(my_env) $ conda-unpack

# Deactivate the environment to remove it from your path
(my_env) $ source my_env/bin/deactivate

Contacts:

• Pavlin Mitev
• UPPMAX
• SNIC AE@UPPMAX - related documentation

tags: UPPMAX, SNIC, RT223073, conda, pip