Conda workshop additional resources/FAQ

# Conda workshop additional resources/FAQ [toc] ## Using the binder ### What happens if I get a 502, 503, or 504 error? Try clicking on the launch button again. The binder or internet connection may have timed out. ### How do I save work from a binder? Be sure to save any work/notes you took in the binder to your computer. Any new files/changes are not available when the binder session is closed! For example, select a file, click "More", click "Export", to save individual files to your computer: ![](https://i.imgur.com/js8eX0l.png) ### I can't copy/paste code to the RStudio Terminal. How do I run commands? On some versions of WindowsOS, the copy/paste function may not work in the Terminal panel. Instead, change the code language in the Source panel to `Shell`. Then, copy/paste code and run line by line from Source. ![](https://i.imgur.com/2jdCO82.png) For the workshop, you should be able to run commands in the Source panel with the `workshop_commands.sh` file. ### How do I make a binder like the one used in the workshop? We used a Pangeo Binder for the workshop: https://binder.pangeo.io/ To make a binder, you need to create a [Github account](https://github.com/) and a Github repository to put files you want to use for the binder. See our tutorial here: https://github.com/nih-cfde/training-and-engagement/wiki/Making-an-RStudio-binder Here are some examples for rstudio binders: https://github.com/binder-examples/r-conda You can also make binders that open with jupyter notebooks or terminal interfaces! ## Conda channels - `conda-forge` and `bioconda` are channels that contain community contributed software - `Bioconda` specializes in bioinformatics software (*supports only 64-bit Linux and Mac OS*) - package list: https://anaconda.org/bioconda/repo - `conda-forge` contains many dependency packages - package list: https://anaconda.org/conda-forge/repo - In absence of other channels, conda [searches the default repository](https://docs.anaconda.com/anaconda/user-guide/tasks/using-repositories/) which consists of ten official repositories. - You can even install R packages with conda! ## Getting started on your system Whether you're installing conda on your own computer, a cloud instance, or high performance computer server, you'll need to consider the following: 1. Conda installer: Miniconda vs Anaconda - Both are free versions with Miniconda being the light weight version 3. OS: Windows, MacOS, Linux 4. Bit-count: 32 vs 64-bit - macOS is 64-bit only 6. Python version for root environment (2.x vs 3.x) - version 3.x is the default option since it is newer - choose 2.7 version if you have mostly 2.7 code or use packages that do not have a 3.x version (but keep in mind that python 2.x sunsetted - https://www.python.org/doc/sunset-python-2/) :::info We have a lesson on installing Miniconda on MacOS: [Click here for lesson](https://training.nih-cfde.org/en/latest/Bioinformatics-Skills/install_conda_tutorial/). You can also learn more from the conda user guide for installation guidelines: [Click here for user guide](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html). ::: ![](https://i.imgur.com/DuVrjaB.jpg) Installer sets up two things: Conda and the root environment. The root environment contains the selected python version and some basic packages. ![](https://i.imgur.com/NM4wtre.png) Image credit: [Gergely Szerovay](https://www.freecodecamp.org/news/why-you-need-python-environments-and-how-to-manage-them-with-conda-85f155f4353c/) **Conda distributions** - [Anaconda](https://www.anaconda.com/products/individual) - [Miniconda](https://docs.conda.io/en/latest/miniconda.html) **Conda package distributions** - [conda-forge](https://conda-forge.org/) - [bioconda](https://bioconda.github.io/) - [Paper introducing bioconda](https://www.nature.com/articles/s41592-018-0046-7) ## How can I make sure the latest version of a chosen software package is always installed? To ensure conda installs the latest version of a package from any listed channel, configure conda to set the channel priority to FALSE: ```bash conda config --set channel_priority false ``` Doing so allows the package version to take precedence over channel priority followed by package build number. :::info Package version > channel priority > package build number ::: ## How do I recreate an environment with exactly the same versions of all software packages? Continuing from the 3 methods of creating and installing software in conda environments in the workshop, here is a 4th method. For this approach, we export a list of the exact software package versions installed in a given environment and use it to set up new environments. This set up method won't install the latest version of a given program, for example, but it will replicate the exact environment set up you exported from. ### Method 4: Install exact environment ``` conda activate fqc conda list --export > packages.txt conda deactivate ``` Two options - 1) install the exact package list into an existing environment: ``` conda install --file=packages.txt ``` 2) set up a new environment with the exact package list: ``` conda env create --name qc_file --file packages.txt ``` ## Workshop exercise solutions There are many ways to solve the exercises from our workshop. Here is one approach! ### Exercise 1 Input: ```bash= conda search blast conda create -y -n exercise1 blast=2.9.0 conda activate exercise1 mkdir exercise1 cd exercise1 curl -o mouse.1.protein.faa.gz -L https://osf.io/v6j9x/download curl -o zebrafish.1.protein.faa.gz -L https://osf.io/68mgf/download gunzip *.faa.gz head -n 11 mouse.1.protein.faa > mm-first.faa makeblastdb -in zebrafish.1.protein.faa -dbtype prot blastp -query mm-first.faa -db zebrafish.1.protein.faa -out myoutput.txt -outfmt 6 ``` Output looks like this: ``` YP_220550.1 NP_059331.1 69.010 313 97 0 4 316 10 322 1.24e-150 426 YP_220551.1 NP_059332.1 44.509 346 188 3 1 344 1 344 8.62e-92 279 YP_220551.1 NP_059341.1 24.540 163 112 3 112 263 231 393 5.15e-06 49.7 YP_220551.1 NP_059340.1 26.804 97 65 2 98 188 200 296 0.10 35.8 ``` The 3rd column has the percent of matching amino acids between the query mouse protein sequence file and the zebra fish reference database. Not surprisingly, :mouse: and :fish: are not very similar! For this particular mouse sequence, it's only 69.01% similar to the zebra fish reference sequences. ### Exercise 2 ```bash= conda activate fqc conda list # check version conda update trimmomatic # should now be 0.39 conda list --revisions conda install --revision 1 # want to revert back to [rev 1] which had 0.36 conda list # now it's back to 0.36! ``` Note that conda told us that it was upgrading to 0.39 and downgrading to 0.36 - useful outputs! ## How do I install R packages with conda? Why would you install R packages with conda rather than `install.packages()`? Here's one use case (a comment from this [blog](https://www.johndcook.com/blog/2019/05/30/r-with-conda/)): > Everytime I start a new project, I create a new conda environment with the R packages I am going to need. I can add more if I need them later on. It saves me from the headaches of having to deal with one unique installation of R and all the dependencies I need for the diferent projects I am working on. This is a fantastic blog/tutorial on this topic: https://alexanderlabwhoi.github.io/post/anaconda-r-sarah/ As an example, you can open the workshop binder: [![Binder](https://binder.pangeo.io/badge_logo.svg)](https://binder.pangeo.io/v2/gh/nih-cfde/training-rstudio-binder/conda-workshop-march2021?urlpath=rstudio) - click the Terminal tab - set up/initialize conda and add channels as we did in the workshop - create a directory for our test files: ```bash= mkdir test cd test ``` - then create a .yml file. You can use the source Rstudio panel to make a new file, save it as `renv.yml` (make sure it saves a .yml and not R script file). The YAML file should have the channels and dependencies below. We are only installing the R package `ggplot2` for this example, but experiment with adding others! Note that conda will automatically install any dependencies that `ggplot2` needs. ``` name: r-test channels: - conda-forge - bioconda - defaults dependencies: - r-ggplot2 ``` - create and activate the environment ```bash=4 conda env create -f renv.yml conda activate r-test ``` - open R in the terminal ```bash=6 R ``` - in the R shell, type the following ```r= library(ggplot2) df <- data.frame('x'=c(1:10), 'y'=c(41:50)) ggplot(df, aes(x, y)) + geom_point() #it's ok if you get an X11 error - that's the terminal saying it can't display the plot. we're going to save the plot and then view it so this isn't an issue! ggsave('testplot.jpg', height=5, width=5, units='in', dpi=400) ``` - on the binder file panel, open the jpg file! ```r= quit() #to exit the R interface # type 'n' - do not need to save the workspace image ``` ## Beyond Conda...Mamba! Conda is very popular and there is a lot of documentation, but it can get slow when trying to resolve compatibility for a lot of software programs. [Mamba](https://github.com/mamba-org/mamba) is an even faster reimplementation of conda. This command installs `mamba` into the base environment, so it's available across environments: ``` conda install -n base -c conda-forge mamba ``` If using `mamba`, the commands usually replace `conda` with `mamba`, for example: ``` mamba env create -f environment.yml ``` However, environments are still be activated using `conda activate`. ## Q&A from March 18 workshop - What does it mean to be OS agnostic? Thinking about reproducibility, if I install some software using conda on Mac OS, do I get the same software as someone who uses my env definition (e.g. with a yaml file) on linux or windows? - 'OS agnostic' means that conda works on Windows, Mac, and Linux! - yes, the software should be the same as long as they specify the same software packages. You can also have them replicate the exact environment (see [here](#How-do-I-recreate-an-environment-with-exactly-the-same-versions-of-all-software-packages)) - does the channel order set for all your environments? or just the current environment? - it sets the order for all environments - you can also specify channel with the `-c` flag in the `conda install` command - what is the difference between the version and the build? - Version of the software reflects the main updates of the software while builds can imply after things like build for a particular OS (linux vs Mac) or some patch fixes to a major version or if it built for different versions of software ( like py 2 vs py3) - More information about what goes into building software for conda : https://docs.conda.io/projects/conda-build/en/latest/resources/package-spec.html - This highlights the different aspects like version, build etc - What is base? - ‘Base’ refers to the base environment. It’s basically the conda installation that you start out with when first installing and setting up conda. From there, we can create multiple virtual environments, like the fqc and fqc_yaml that were just created! - What programs can you make a .yml file in? - You can make .yml in any text editor program! the YAML format is nice because it’s both computer and human-readable - Suppose I have a pipeline with several steps. The steps require different software but they are not incompatible. Is it considered better practice to create one environment for the whole pipeline for simplicity, or a separate environment for each step (e.g. to allow for easy reuse of the individual steps in other pipelines). If the same software is in more than one environment, does it get installed multiple times, or is there a central installation that is reused by different environments? - different env for different steps could be better. The sw are not accessed across envs - This is a bit of user choice, but our lab’s workflows use multiple conda environments. It’s actually better because the more software packages to install in 1 env, the slower conda gets - and yes, it would be separate installations. Conda stores installations for each env in a separate folder - Are packages accessed through conda guaranteed to be there 'forever'? - the conda env you create on your machine will remain so till you remove them - For packages in the channel, it generally is kept around but depends on the developers who added the pkgs - Sometimes I come across instructions to install software using a different package manager, presumably because the software is not available through conda. Can software that is not downloaded through conda be installed in a conda environment to keep it isolated? Or is there a better way to do this? - that’s a good question. Partly the answer is yes. Any software installed with conda or otherwise ends up in the env folder so you end up having it isolated. - More information on installing non-conda programs and such —> https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-pkgs.html#installing-similar-packages - I've heard about singularity and docker containers but haven't ever used them. Could you say a bit more about them (esp in comparison to conda environments) and what they would be useful for? - You can think of docker/singularity as useful for creating shareable images of the OS — a snapshot of your work that can be recreated in any other system. Conda works the same way except you cannot include files with conda env - docker requires super-user permissions and will not work, for example, on an HPC because most users won't have root access (often you will see Singularity on HPCs instead as an alternate container system)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.