conda and mamba on Rackham@UPPMAX, Dadel@PDC

Note: the following instructions do not apply for Bianca!

In case you insist using conda from the module system, please follow these instructions. Conda - best practices @ UPPMAX

If you experience unexpected problems with the conda provided by the module system on Rackham or anaconda3 on Dardel, you can easily install your own and maintain it yourself.

:exclamation:Important: Please, install miniforge3 or miniconda3 in your project folder!:exclamation:

Home folders at UPPMAX have 32GB (25GB on Dardel) which is rather limited storage space for most common conda installations.

:exclamation:Important: Please, install miniforge3 or miniconda3 in your project folder!:exclamation:

Take some precautions by (re)moving the .conda directory and .condarc in your home folder if you had ones from previous use. Look in your ~/.bashrc and remove leftovers from other conda installations. Delete everything between ## >>> conda initialize >>> and ## <<< conda initialize <<< including these two lines. Remove any exports related to conda i.e. VARIABLES starting with CONDA_

Install miniforge3 https://github.com/conda-forge/miniforge
Download the installation media, run the installation, read and accept the user agreement, and redirect the installation to your project folder, preferably in /proj/naiss2023-XXX-XX/nobackup/miniforge3. Choose to initialize the shell.

# Navigate to your project folder
cd /proj/naiss2023-XXX-XX/nobackup

# Download the installation file
wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh

# Start the interactive installation
sh Miniforge3-Linux-x86_64.sh

# ... or non-interactive  
# sh Miniforge3-Linux-x86_64.sh        -b -p /proj/naiss2023-XXX-XX/nobackup/miniforge3
...

Logout and login again to pickup the new conda setup, then disable the auto activation of the base environment. Working from the conda base environment might interfere with the software modules provided by the computer center.

(base) conda config --set auto_activate_base false

Logout and login again to get the final setup active.

Check that conda works

# Check where is conda
which conda
/proj/naiss2023-XXX-XX/nobackup/miniforge3/condabin/conda

# Activate base environment
conda activate

# Display a health report for your environment.
(base) conda doctor
Environment Health Report for: /proj/naiss2023-XXX-XX/nobackup/miniforge3
1. Missing Files:
✔ There are no packages with missing files.
2. Altered Files
✔ There are no packages with altered files.

Check

# Activate base environment
conda activate

# Check the output from mamba -V
(base) mamba -V
mamba 1.5.0
conda 23.7.3

(base) mamba info

OPTIONAL: Define CONDA_ENVS_PATH if you want to specify different location than the default miniconda3/miniforge3 sub-folder for the new conda environments.

# Add the line bellow in your $HOME/.bashrc
export CONDA_ENVS_PATH=/proj/naiss2023-XXX-XX/user/CONDA_ENVS
mkdir -p $CONDA_ENVS_PATH

Advantages and disadvantages of user/project installation

Advantages:

Full control over the installation.
No specific setup i.e. one can follow the regular installation instructions provided by developers.
Environments will be installed in subfolder of the installation i.e. in your project folder.
Faster solutions for installations - conda does not have to search through all available indexes Rackham but only the one you provide.
You can install in the (base) environment (although not a good practice).

Disadvantages

:point_right: You need to maintain the installation - setup, cleaning cache, update, etc.
Not applicable solution for isolated clusters like Bianca.

If you insist on using conda from the module system

Conda - "best practices" - UPPMAX

Beyond conda and mamba

Alternatively, this might be more than enough to get you working - micromamba
Tired of conda - try pixi - faster, python independent package manager and more utilizing the conda packaging ecosystem.
- 7 Reasons to Switch from Conda to Pixi
- What is the difference with conda, mamba, poetry, pip
- just my personal opinion 🤏🧂: I hope this or similar project becomes more popular or standard…

Troubleshooting conda provided on Rackham

If it complains - it is time to clean the mess.

Look in your ~/.bashrc for conda configuration and remove these lines - if uncertain - copy them in another file, for safety…

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/xxxxx/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/xxxxx/conda.sh" ]; then
        . "/xxxx/etc/profile.d/conda.sh"
    else
        export PATH="/xxx/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<

run conda init --reverse to reverse the damage. This is problematic only on Rackham and Bianca, where this is not the correct way to activate conda.

logout and login again.

conda and mamba on Rackham@UPPMAX, Dadel@PDC

Advantages and disadvantages of user/project installation

Advantages:

Disadvantages

If you insist on using conda from the module system

Beyond conda and mamba

Troubleshooting conda provided on Rackham

Contacts:

tags: `UPPMAX`, `PDC`, `conda`, `mamba`

conda and mamba on Rackham@UPPMAX, Dadel@PDC

Advantages and disadvantages of user/project installation

Advantages:

Disadvantages

If you insist on using conda from the module system

Beyond conda and mamba

Troubleshooting conda provided on Rackham

Contacts:

tags: UPPMAX, PDC, conda, mamba

Read more

nf-core pipelines on Bianca

ParseBiosciences-Pipeline.1.5.0 on Bianca

Building Singularity containers on Transit

Conda - best practices - UPPMAX

tags: `UPPMAX`, `PDC`, `conda`, `mamba`