conda and mamba on Rackham@UPPMAX, Dadel@PDC

Note: the following instructions do not apply for Bianca!

In case you insist using conda from the module system, please follow these instructions. Conda - best practices @ UPPMAX

If you experience unexpected problems with the conda provided by the module system on Rackham or anaconda3 on Dardel, you can easily install your own and maintain it yourself.

Important: Please, install miniforge3 or miniconda3 in your project folder!

Home folders at UPPMAX have 32GB (25GB on Dardel) which is rather limited storage space for most common conda installations.

Important: Please, install miniforge3 or miniconda3 in your project folder!

1. Take some precautions by (re)moving the .conda directory and .condarc in your home folder if you had ones from previous use. Look in your ~/.bashrc and remove leftovers from other conda installations. Delete everything between ## >>> conda initialize >>> and ## <<< conda initialize <<< including these two lines. Remove any exports related to conda i.e. VARIABLES starting with CONDA_
2. Install miniforge3 https://github.com/conda-forge/miniforge
   Download the installation media, run the installation, read and accept the user agreement, and redirect the installation to your project folder, preferably in /proj/naiss2023-XXX-XX/nobackup/miniforge3. Choose to initialize the shell.
   ​​​​# Navigate to your project folder
   ​​​​cd /proj/naiss2023-XXX-XX/nobackup
   ​​​​
   ​​​​# Download the installation file
   ​​​​wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
   ​​​​
   ​​​​# Start the interactive installation
   ​​​​sh Miniforge3-Linux-x86_64.sh
   ​​​​
   ​​​​# ... or non-interactive  
   ​​​​# sh Miniforge3-Linux-x86_64.sh        -b -p /proj/naiss2023-XXX-XX/nobackup/miniforge3
   ​​​​...
   

   • Logout and login again to pickup the new conda setup, then disable the auto activation of the base environment. Working from the conda base environment might interfere with the software modules provided by the computer center.
   ​​​​(base) conda config --set auto_activate_base false
   

   • Logout and login again to get the final setup active.
3. Check that conda works
   ​​​​# Check where is conda
   ​​​​which conda
   ​​​​/proj/naiss2023-XXX-XX/nobackup/miniforge3/condabin/conda
   ​​​​
   ​​​​# Activate base environment
   ​​​​conda activate
   ​​​​
   ​​​​# Display a health report for your environment.
   ​​​​(base) conda doctor
   ​​​​Environment Health Report for: /proj/naiss2023-XXX-XX/nobackup/miniforge3
   ​​​​1. Missing Files:
   ​​​​✔ There are no packages with missing files.
   ​​​​2. Altered Files
   ​​​​✔ There are no packages with altered files.
   

4. Check
   ​​​​# Activate base environment
   ​​​​conda activate
   ​​​​
   ​​​​# Check the output from mamba -V
   ​​​​(base) mamba -V
   ​​​​mamba 1.5.0
   ​​​​conda 23.7.3
   ​​​​
   ​​​​(base) mamba info
   

5. OPTIONAL: Define CONDA_ENVS_PATH if you want to specify different location than the default miniconda3/miniforge3 sub-folder for the new conda environments.
   ​​​​# Add the line bellow in your $HOME/.bashrc
   ​​​​export CONDA_ENVS_PATH=/proj/naiss2023-XXX-XX/user/CONDA_ENVS
   ​​​​mkdir -p $CONDA_ENVS_PATH  
   

Advantages and disadvantages of user/project installation

Advantages:

• Full control over the installation.
• No specific setup i.e. one can follow the regular installation instructions provided by developers.
• Environments will be installed in subfolder of the installation i.e. in your project folder.
• Faster solutions for installations - conda does not have to search through all available indexes Rackham but only the one you provide.
• You can install in the (base) environment (although not a good practice).

Disadvantages

• You need to maintain the installation - setup, cleaning cache, update, etc.
• Not applicable solution for isolated clusters like Bianca.

If you insist on using conda from the module system

Conda - "best practices" - UPPMAX

Beyond conda and mamba

• Alternatively, this might be more than enough to get you working - micromamba
• Tired of conda - try pixi - faster, python independent package manager and more utilizing the conda packaging ecosystem.
  • 7 Reasons to Switch from Conda to Pixi
  • What is the difference with conda, mamba, poetry, pip

  • just my personal opinion 🤏🧂: I hope this or similar project becomes more popular or standard…

Troubleshooting conda provided on Rackham

• If it complains - it is time to clean the mess.
  • Look in your ~/.bashrc for conda configuration and remove these lines - if uncertain - copy them in another file, for safety…
  ​​# >>> conda initialize >>>
  ​​# !! Contents within this block are managed by 'conda init' !!
  ​​__conda_setup="$('/xxxxx/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
  ​​if [ $? -eq 0 ]; then
  ​​    eval "$__conda_setup"
  ​​else
  ​​    if [ -f "/xxxxx/conda.sh" ]; then
  ​​        . "/xxxx/etc/profile.d/conda.sh"
  ​​    else
  ​​        export PATH="/xxx/bin:$PATH"
  ​​    fi
  ​​fi
  ​​unset __conda_setup
  ​​# <<< conda initialize <<<
  

  • run conda init --reverse to reverse the damage. This is problematic only on Rackham and Bianca, where this is not the correct way to activate conda.
• logout and login again.

Contacts:

• Pavlin Mitev
• UPPMAX
• AE@UPPMAX - related documentation

tags: UPPMAX, PDC, conda, mamba