A Hands-On Introduction to Conda

# A Hands-On Introduction to Conda **When:** Thursday, March 4, from 10:00 am-12:00 pm PST **Instructors:** Marisa Lim and Saranya Canchi **Moderator:** Abhijna Parigi **Helpers:** Jose Sanchez, Jeremy Walter **Zoom:** https://zoom.us/j/7575820324?pwd=d2UyMEhYZGNiV3kyUFpUL1EwQmthQT09 **Description:** This free 2 hour hands-on tutorial will introduce you to the package and environment manager, Conda. You'll learn how to set up conda environments and manage software installations. We'll discuss other applications, such as packaging software, running workflows, or creating binders all with conda! [toc] ::: warning Links for Moderator to share: - workshop notes: https://hackmd.io/1ivIgqVYSCupC21MuAW9FA?view - pre-workshop survey: https://forms.gle/XuFwaGDemWyCLUYh7 - exercise 1: https://hackmd.io/1ivIgqVYSCupC21MuAW9FA?view#Exercise-1- - exercise 2: https://hackmd.io/1ivIgqVYSCupC21MuAW9FA?view#Exercise-2- - post-workshop survey:https://forms.gle/qKZj61wwZYx8fkQ27 - CFDE training website: https://training.nih-cfde.org/en/latest/ - workshop additional resources/FAQ: https://hackmd.io/RweX2WZ7RGGtfLfCDuEubQ ::: :::success While we wait to get started -- 1. :pencil: Please fill out our pre-workshop survey if you have not already done so! Link here: https://forms.gle/XuFwaGDemWyCLUYh7 2. :computer: Open a web browser - e.g., Firefox, Chrome, or Safari. Open the binder by clicking this button: [![Binder](https://binder.pangeo.io/badge_logo.svg)](https://binder.pangeo.io/v2/gh/nih-cfde/training-rstudio-binder/conda-workshop-march2021?urlpath=rstudio). It may take a few minutes to load. ::: ## Introductions We are both postdocs at UC Davis and part of the training and engagement team for the [NIH Common Fund Data Ecosystem](https://nih-cfde.org/), a project supported by the NIH to increase data reuse and cloud computing for biomedical research. You can contact us at mcwlim@ucdavis.edu and srcanchi@ucdavis.edu. ## Why Conda ? ![](https://i.imgur.com/2R6UDu3.jpg) - OS-agnostic tool for package and environment management - clean software installation along with dependencies - searches for compatible software versions - many packages available - python, R, etc. - user has admin permissions, good for use on HPC where you'd otherwise need to request software installations ## What is an Environment ? Think of environments like zoom breakout rooms and the main room as your laptop OS. You can have many rooms and each independent from one another yet all are connected to your main room. You can join a room, chat, share screen etc and those remain part of the room with no cross over to the main room or any other room. The equivalent of zoom rooms for software are directories that contain the specific software/tools and their dependencies for a given environment. This is a great way to maintain different versions of the same software ! ### Why do we need isolated software environments ? - your research/project requires specific versions of software packages (**versioning**) - you want to experiment with new packages or features of existing packages without compromising your current workflow (**reproducibility**) - you want to replicate results from a publication (**repeatability**) - your collaborators require you to use certain software/tools that are incompatibile with your current system (**compatibility**) There are many actions you can perform on conda environments. We will cover a few today: - [x] create - [x] list - [x] remove - [x] update - [x] revert - [x] export ::: info Goals for today: - learn about conda and how to use it - learn the basics of software installation, software dependencies, and isolation environments - learn about managing virtual environments ::: --- ## Let's get started! If not done already: - :computer: Open a web browser - e.g., Firefox, Chrome, or Safari. - Open the binder by clicking this button: [![Binder](https://binder.pangeo.io/badge_logo.svg)](https://binder.pangeo.io/v2/gh/nih-cfde/training-rstudio-binder/conda-workshop-march2021?urlpath=rstudio) :::success :heavy_check_mark: Did the binder load? Put up a :hand: on Zoom if you see Rstudio in your web browser. ::: ### Initialize conda :keyboard: Copy/paste commands into the terminal OR run the commands from the `workshop_commands.sh` file in the binder. Setup the conda installer and initialize the settings: ```bash=1 source $(conda info --base)/etc/profile.d/conda.sh conda init conda config --set auto_activate_base false ``` :::info If you have a WindowsOS, the copy/paste function may not work in the Terminal panel. Instead, change the code language in the Source panel to `Shell`. Then, copy/paste code and run line by line from Source. ![](https://i.imgur.com/2jdCO82.png) ::: We will shorten command prompt to `$`: ```bash=4 echo "PS1='\w $ '" >> .bashrc ``` Re-start terminal for the changes to take effect (type `exit` and then open a new terminal). Activate the base conda environment: ```bash=5 conda activate base ``` :::success :heavy_check_mark: Put up a :hand: on Zoom when you've got a new terminal window and see this at the prompt: ` (base) ~ $` ::: --- ### Conda channels: Searching for software The channels are places that conda looks for packages. The default channels after conda installation is set to Anaconda Inc's channels (Conda's Developer). ```bash=6 conda config --show channels conda info # get channel URLs ``` Channels exist in a hierarchial order. By default: :::info Channel priority > package version > package build number ::: ![](https://i.imgur.com/GE8nY0a.png) Image credit: [Gergely Szerovay](https://www.freecodecamp.org/news/why-you-need-python-environments-and-how-to-manage-them-with-conda-85f155f4353c/) :::info - conda-forge and bioconda are channels that contain commuity contributed software - Bioconda specializes in bioinformatics software (*supports only 64-bit Linux and Mac OS*) - conda-forge contains many dependency packages - In absence of other channels, conda [searches the default repository](https://docs.anaconda.com/anaconda/user-guide/tasks/using-repositories/) which consists of ten official repositories. - You can even install R packages with conda! - See [Resources](https://hackmd.io/1ivIgqVYSCupC21MuAW9FA?view#Resources) for links. ::: We will update the channels to include bioconda since our demo includes downloading some bioinformatic tools. **The order of the channels matters !** First add bioconda channel which defaults to top of the list: ```bash=8 conda config --prepend channels bioconda conda config --get channels ``` Then re-add the conda-forge channel to move it to top of the list to follow the [Bioconda recommended channel order](https://bioconda.github.io/user/install.html#set-up-channels): ```bash=10 conda config --prepend channels https://conda.anaconda.org/conda-forge conda config --get channels ``` --- ### Install Software We will install FastQC which is a software tool that provides a simple way to run quality control checks on raw sequencing data. Search for software (fastqc): ```bash=12 conda search fastqc ``` Create conda environment and install fastqc: ```bash=13 conda create -y --name fqc fastqc ``` Activate environment: ```bash=14 conda activate fqc ``` Check fastqc version: ```bash=15 fastqc --version ``` :::success :heavy_check_mark: Put up a :hand: on Zoom if your command prompt shows the name of the environment in parentheses: `(fqc) ~ $` ::: :::info To go back to `(base) ~ $` environment: ``` conda deactivate ``` ::: --- Our QC steps for the pipeline involve fastqc and trimmomatic, which is useful for read trimming (i.e., adapters). There are multiple ways we could create the conda environment that contains both software programs. #### Method 1: install software in existing environment We could add `trimmomatic` to the `fqc` environment: ```bash= conda install -y trimmomatic=0.36 conda list # check installed software ``` We can specify the exact software version :point_up_2: The default is to install the most current version, but sometimes your workflow may depend on a different version. --- #### Method 2: install both software during environment creation When you switch conda environments, conda changes the PATH (and other environment variables) so it searches for software packages in different places. Let's check the PATH for method 1: ``` echo $PATH ``` You should see that the first element in the PATH changes each time you switch environments! ```bash= conda deactivate conda create -y --name fqc_trim fastqc trimmomatic=0.36 conda activate fqc_trim conda list # check installed software echo $PATH # path for method 2 ``` --- The following methods use an external file to specify the packages to install. #### Method 3: specify software to install with a YAML file Often, it's easier to create environments and install software using a YAML file (YAML is a file format) that specifies all the software to be installed. For our example, we create a file called `test.yml`. Let's start back in the `(base)` environment. ```bash=1 conda deactivate ``` The `test.yml` file contains the following in YAML format: ```yaml name: qc_yaml #this specifies environment name channels: - conda-forge - bioconda - defaults dependencies: - fastqc - trimmomatic=0.36 ``` Create the environment - note the difference in conda syntax: ```bash=2 conda env create -f test.yml #since environment name specified in yml file, we do not need to use -n flag here conda activate qc_yaml conda list # check installed software ``` --- :::success :writing_hand: Try Method 4 (below) on your own! For this approach, we export a list of the exact software package versions installed in a given environment and use it to set up new environments. This set up method won't install the latest version of a given program, for example, but it will replicate the exact environment set up you exported from. ::: #### Method 4: Install exact environment ``` conda activate fqc conda list --export > packages.txt conda deactivate ``` Two options - 1) install the exact package list into an existing environment: ``` conda install --file=packages.txt ``` 2) set up a new environment with the exact package list: ``` conda env create --name qc_file --file packages.txt ``` --- ### Managing Environments At this point, we have several conda environments! To see a list, there are 2 commands (they do the same thing!): ```bash conda env list ``` or ```bash conda info --envs ``` :::warning Generally, you want to avoid installing too many software packages in one environment. It takes longer for conda to resolve compatible software versions for an environment the more software you install. For this reason, in practice, people often manage software for their workflows with multiple conda environments. ::: --- ### FastQC demo If not already done, activate one of the environments we created, e.g.,: ```bash conda activate fqc ``` Let's make sure the software was installed correctly: ```bash fastqc --help ``` :::success Output should look like: ``` FastQC - A high throughput sequence QC analysis tool SYNOPSIS fastqc seqfile1 seqfile2 .. seqfileN fastqc [-o output dir] [--(no)extract] [-f fastq|bam|sam] [-c contaminant file] seqfile1 .. seqfileN ... ``` ::: Download data ```bash= curl -L https://osf.io/5daup/download -o ERR458493.fastq.gz ``` Check out the data: ```bash=2 gunzip -c ERR458493.fastq.gz | wc -l ```  :::success :heavy_check_mark: How many lines are in this fastq file? (add number to Zoom chat) ::: What does the fastq file look like? ```bash=3 gunzip -c ERR458493.fastq.gz | head ``` Run FastQC! ```bash=4 fastqc ERR458493.fastq.gz ``` :::success :heavy_check_mark: Put up a :hand: on Zoom if you see an html file in the directory. ::: --- ## Exercise 1 :writing_hand: - **First**, install the `blast` software with version 2.9.0 using conda. :::spoiler Hint! :bulb: You can use one of these approaches to install `blast`: - install the software in an existing env using `conda install -y <name of the software>` - create a new env using `conda create -y --name <name of env> <software to install>` ::: - **Second**, try this example BLAST analysis in the conda environment. In this example, we are comparing the sequence similarity of a mouse protein sequence to a reference zebra fish sequence - as you might imagine, they are not that similar! But for today, this exercise will demonstrate running a quick analysis in a conda environment and bonus points if you find out how similar/dissimilar they are! (More details on BLAST and what each step is for [here](https://training.nih-cfde.org/en/latest/Bioinformatics-Skills/Command-Line-BLAST/BLAST4/)). Run each line of code below in the terminal: - Make a directory for the exercise files: ```bash= mkdir exercise1 cd exercise1 ``` - Download with `curl` command and unzip data files: ```bash=3 curl -o mouse.1.protein.faa.gz -L https://osf.io/v6j9x/download curl -o zebrafish.1.protein.faa.gz -L https://osf.io/68mgf/download gunzip *.faa.gz ``` - Subset the data for a test run: ```bash=6 head -n 11 mouse.1.protein.faa > mm-first.faa ``` - Format zebra fish sequence as the blast database to search against: ```bash=7 makeblastdb -in zebrafish.1.protein.faa -dbtype prot ``` - Run a protein blast search with `blastp`! ```bash=8 blastp -query mm-first.faa -db zebrafish.1.protein.faa -out mm-first.x.zebrafish.txt -outfmt 6 ``` What does the output look like? :::info Note that if you `conda deactivate`, you can still access the input/intermediate/output files from the BLAST analysis. They are not 'stuck' inside the conda environment! ::: --- ## Exercise 2 :writing_hand: Conda allows you to revert to a previous version of your software using the `--revision` flag: Usage: ``` # list all revisions conda list --revisions # revert to previous state conda install --revision <number> # for example: conda install --revision 1 ``` Earlier, we installed an older version of trimmomatic (0.36). Try updating it to the most recent version and then revert back to the old version. :::spoiler Hint! :bulb: You can do this exercise in any of the conda environments we created earlier with trimmomatic. You can update software with `conda update <software name>` ::: --- **Tidying up** To remove old conda environments: ``` conda env remove --name <conda env name> ``` To remove software: ``` conda remove <software name> ``` :::warning Be sure to save any work/notes you took in the binder to your computer. Any new files/changes are not available when the binder session is closed! For example, select a file, click "More", click "Export": ![](https://i.imgur.com/js8eX0l.png) ::: ## What else can we use Conda for? - **Reproducing analyses** - **Why**: removes the guesswork on what version of software to use or how to install it! :::spoiler For example, it's a lot easier to replicate a software environment from a publication or share software versions used in your own analysis using conda (particularly method 3 & 4 mentioned above ways to create conda environments from YAML or exact package list files). ::: - **Packaging software** - **Why**: to share your cool software with the world! :::spoiler The gist is that you write a recipe that has all the specs about your software that is submitted to a channel, e.g., conda-forge or bioconda. Once correctly formatted and tested (with continuous integration automation) so it works on different operating systems, it's added to the channel and the world can use and install your software with conda! - https://python-packaging-tutorial.readthedocs.io/en/latest/conda.html ::: - **Running parts of analysis workflows in their own environment** - **Why**: avoid software version conflicts, easier to keep track of software/versions used for a specific analysis, easier for others to reproduce :::spoiler This is a very helpful feature for workflows. For example, in [Snakemake](https://training.nih-cfde.org/en/latest/Bioinformatics-Skills/Snakemake/) workflows, you can specify that each step (called a "rule") is executed in an isolated conda environment by adding a `conda:` directive: ```python rule fastqc_raw: input: "rnaseq/raw_data/{sample}.fq.gz" output: "rnaseq/raw_data/fastqc/{sample}_fastqc.html" params: outdir="rnaseq/raw_data/fastqc" conda: "rnaseq-env.yml" shell: """ fastqc {input} --outdir {params.outdir} """ ``` ::: - **Creating binders** - **Why**: teaching tool for software or analysis demos, share reproducible analysis :::spoiler Examples: - the Rstudio binder we're using today was created with https://binder.pangeo.io/ - [metENP binder](https://github.com/metabolomicsworkbench/MetENP): tool for metabolite enrichment analysis and their associated enriched pathways ::: ## Q&A :::success :pencil: Please fill out our post-workshop survey! We use the feedback to help improve our training materials and future workshops. Link here: https://forms.gle/qKZj61wwZYx8fkQ27 ::: Questions? ## Resources - [CFDE training webite](https://training.nih-cfde.org/en/latest/) - Common conda commands: - [CFDE conda cheat sheets](https://training.nih-cfde.org/en/latest/General-Tools/Cheat-Sheets/conda_cheatsheet/) - [official conda cheat sheet](https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html) - [Workshop additional resources/FAQ](https://hackmd.io/RweX2WZ7RGGtfLfCDuEubQ) - [CFDE events & workshops](https://www.nih-cfde.org/events/) - [DIB Lab video about conda](https://www.youtube.com/watch?v=Ef1QwhELuMs)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.