changed 2 months ago
Published Linked with GitHub

Conda - "best practices" - UPPMAX

Disclaimer: Instructions below assume that conda provided by the module system is working. Please, submit a support request if something does not work or you want to know why is not working.

Alternatively - follow this tutorial on how to install own conda on Rackham.

The default conda module is not complete setup and will, by default, cache and install in your $HOME folder which is only 32GB and pretty much, not enough for anything. Follow the instructions below to redirect conda cache and environments to your project folder.

Prepare your conda setup.

UPDATE 2024.03.07: Instead of the official instructions below

# module load conda
# source conda_init.sh

run this

source /sw/apps/conda/latest/rackham_stage/etc/profile.d/conda.sh

Shell environment

# Start fresh by cleaning your cache
conda clean -ay

# Location where the new conda environments will be created
# You can add this line in your ~/.bashrc
# Edit the line, so it contains a correct path
export CONDA_ENVS_PATH=/proj/naiss2023-XXX-XX/user/CONDA_ENVS
mkdir -p $CONDA_ENVS_PATH
# redirect where the packages will be cached (default is in your $HOME folder)
# run and add this next line as well in your ~/.bashrc
export CONDA_PKGS_DIRS=/proj/naiss2023-XXX-XX/user/CONDA_PKGS
mkdir -p $CONDA_PKGS_DIRS

This, most probably will activate the (base) conda environment.
(base) environment conflicts with other tools especially with python!

  • Disable the auto activation of the (base) environment.
(base) conda config --set auto_activate_base false
  • Logout and login again!
  • Make sure you do not have (base) activated.
  • Check if conda is available by runningconda -V - If it prints the version you might be OK
  • Test conda env list. If it works, then you are all setup. Congratulations!
  • UPDATE 2024.04.08: mamba's solver is "baked" into conda , so mamba is not available anymore.

Tips

module load conda provides conda with system-wide configuration /sw/apps/conda/latest/rackham_stage/.condarc that includes local mirrors that are necessary for offline installation on Bianca. Unfortunately, this puts unnecessary load on the conda dependence solving process which leads to unresolved or toooooo slow installations.
Try these options when possible.

  • Add --override-channels to you command. Note, not all commands support it.
    Example for installing https://github.com/harvardinformatics/degenotate
    ​​mamba create -n degenotate -c bioconda -c conda-forge -c defaults degenotate --override-channels
    
  • Consider your own local conda installation on Rackham - link

Troubleshooting

  • If it complains - it is time to clean the mess.
    • Look in your ~/.bashrc for conda configuration and remove these lines - if uncertain - copy them in another file, for safety
    ​​# >>> conda initialize >>>
    ​​# !! Contents within this block are managed by 'conda init' !!
    ​​__conda_setup="$('/xxxxx/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
    ​​if [ $? -eq 0 ]; then
    ​​    eval "$__conda_setup"
    ​​else
    ​​    if [ -f "/xxxxx/conda.sh" ]; then
    ​​        . "/xxxx/etc/profile.d/conda.sh"
    ​​    else
    ​​        export PATH="/xxx/bin:$PATH"
    ​​    fi
    ​​fi
    ​​unset __conda_setup
    ​​# <<< conda initialize <<<
    
    • run conda init --reverse to reverse the damage. This is problematic only on Rackham and Bianca, where this is not the correct way to activate conda.
  • logout and login again.
  • Now every time you need conda
# module load conda
# source conda_init.sh
source /sw/apps/conda/latest/rackham_stage/etc/profile.d/conda.sh
# make sure you have 
export CONDA_ENVS_PATH=/proj/naiss2023-XXX-XX/user/CONDA_ENVS
export CONDA_PKGS_DIRS=/proj/naiss2023-XXX-XX/user/CONDA_PKGS
  • Check you can run conda env list.

DONE!

Contacts:

tags: UPPMAX
Select a repo