---
tags: workshop, notepad
---
# Notepad_2019-04-11-gwu
===
Data Carpentry Genomics Workshop @ GWU 2019
Date: April 11-12th 2019
Location: Gelman Library, 2130 H St NW, Room 301/302, Washington, DC 20052
[**Workshop lessons (click on this link)**](https://gwu-omics2019.readthedocs.io/en/latest/)

---

[**Pre-workshop survey**](https://bit.ly/2Kqgsdh)
[**Post-workshop survey**](https://www.surveymonkey.com/r/dcpostworkshopassessment?workshop_id=2019-04-11-gwu)

---

## We will use this HackMD document to share links and snippets of code, take notes, ask and answer questions, and whatever else comes to mind.

# Quick Markdown Introduction

### This is a Heading
===
This is a [link](http://angus.readthedocs.io/en/2018/)
* point one
* point two
* point three

> See this [awesome page](https://hackmd.io/s/features) for more Markdown features

# Announcements

[Code of conduct](https://docs.carpentries.org/topic_folders/policies/code-of-conduct.html)

Bathrooms:

# Instructors:
* Niel Infante
* Sateesh Peri

# TA/Helpers
* Will Bonnett
* Adam Wong
* Rebecca Clement
* Keylie Gibson

# Sign-In Here

+ > *Name, Major, Email, Twitter (optional). What is the last thing that you made?
+ Yi-Wen Chen,Molecular Pathology, [e-mail]
+ Ian Toma, Clinical Research & Leadership, Dept. of Medicine, Genomic Medicine Division, [e-mail]
+ Sateesh Peri
+ Gabriella Rudy, Msc in Bioinformatics, [e-mail], Python script
+ Richa Joshi, MS in Data Science, [e-mail]
+ Angela Maggio, Bioinformatics and Biochemsitry Masters, [e-mail], R loop
+ Jonathan LoTempio, Genomics and Bioinformatics PhD Program, [e-mail], breakfast
+ Kristen Kocher, Genomics & Bioinformatics PhD Program, [e-mail], burrito
+ Cameron Himes, Biology, [e-mail]
+ Stephanie Gomez, Microbiology & Immunology PhD Program, [e-mail], nanoparticles
+ Hiroki Morizono, Genomics and Precision Medicine, [e-mail], @morizono, coffee
+ Mollie Manier, Assist. Prof in Biology, [e-mail], @maniermk, toast
+ Niel Infante, Bioinformatics, I made a hedgehog.
+ Gitanjali Roy, MS Bioinformatics & Biochemistry, [e-mail], https://twitter.com/roy_gitanjali, coffee
Bugs Bunny, [e-mail]
* Dhatri Badri, Biology, [e-mail]
* nzahadat CyVerse username
* Marcin Gierdalski, Div. Biostatistics, [e-mail]
+ Chia Han Lee, Bioinformatics, [e-mail], Breakfast

General notes
==
Don't use spaces in filenames. You can't use spaces in variable names. Suggest using_underscores_instead. YouCanAlsoUseCamalCase

Append to a file within a loop:

    
Write the output of a loop to a file

Example of writing the filename to the file before the contents of the file

Using trimmomatic: see <https://gwu-omics2019.readthedocs.io/en/latest/trimming.html> for common trimming parameters
    Remember that you need bioconda installed in your environment to use trimmomatic.
    Trimming is very easy to parallelize because you are doing the **same** thing to **many** things.
    Any command that displays in [ ] is optional. That doesn't mean you shouldn't use it.
    Anything with a | in it is an OR (you can do one OR the other)
    
bwa mem referenceGenome input1 input2 > output
    There are 2 inputs because these are paired files

Snakemake (a popular genomics workflow manager) 
===
rule rule_name:
    input:
    output:
    shell:
    conda: (optional connection to the yml file outlining your environment)
snakefile must be named Snakefile (needs s to be capitalized)

You still need to do some shell scripting to figure out what output files to anticipate.  I.e. you need to know what kind of files to expect from a tool.

Snakemake looks at the 1st rule by default
    otherwise must give snakemake the specific rule that you want it to run

Rule all says "hey snakemake, I rely on these outputs in order to do my work, so they are your inputs."

Order of operations for running rules:
1. rule all (any outputs then dependencies)
2. first rule
3. stop

Try:

will send different samples to different cores (4 == number of cores)

Other Popular Workflow management options include:
+ Common workflow language;
+ Make;
+ Nextflow (supposed to be best on HPCs)

Questions for instructors/helpers?
===
Q) What's the difference between Reboot and Redeploy for instance actions?

A) You reboot your machine while it is already running but you can redeploy a stopped machine with the same image to start it up.

Q) What does the black line on the quality score graph mean?

Q) Yesterday, Niel mentioned that he never trims sequences for RNAseq because poor quality reads don't affect downstream processes. Can he or someone else elaborate on that? I'd love to get some more information on best practices like that for RNAseq analysis for differential expression.

Q) I would love to be able to implement RStudio through CyVerse to do some differential expression analysis on bam files for the class I'm teaching on Wednesday (!!!!) -- is this even possible in that timeframe??

Q) Will we have access to the gwu-omics website after the workshop?

A) YES, Forever

Q) Is there a good rule of thumb to use to predict what instance size you need?

A) There is really no good way to predict the instance sizes you might need. You will have to try different sizes. Here is the Cyverse wiki about AUs (https://wiki.cyverse.org/wiki/display/atmman/Requesting+More+Atmosphere+Resources)

Q) What does "force a refresh" in the dashboard do? Does it just update the browser or is it like a redeploy?

Q) I ran into a Fail2Ban issue by trying to ssh into my vm when the network was up but the machine wasn't fully deployed.

A) Need to use the web-cli interface, and unban. From Julian in support "If you don't mind, please ask the instructor to let the other students know about the web shell + unban_help trick if they can't SSH into their instances."

Log of shell commands
==

curl -L https://osf.io/vuk5y/download -o shell_data.zip

unzip shell_data.zip

ls   # lists contents in directory
pwd  # path location
cd   # change directory
*    # wildcard. Good for searching through directories
head # returns the first 10 lines
head -n 8 file.name # returns the first 8 lines
tail # returns the last 10 lines
less # scrollable content (hit q to leave)
/   # search within results
rm  # removes the file forever!
cp  # copy. Syntax: cp original_file new_file
mkdir # make directory
mv  # move file
chmod  # change command permissions. 
    To take away write permissions:
        chmod -w file.name
        Good for data from Sequencing Center :)
    Grant write permissions:
        chmod +w file.name
grep  # search with pattern matching
grep -c # count of lines that match a given pattern
echo # returns whatever you give the command as an output
for _ in _ # loop
do # what we want it to do
# String loops together with ;
example: for x in a b c d e; do echo $x; done
{start stop step} # create a range of numbers
cat # displays the output of a file onto the terminal
zcat # like cat but for gzipped files
set -e # fail safe within a shell script that says "stop here if something is not working right"


# with flags

ls -a # show hidden files
cd . # current directory
cd .. # up one directory

# Get data from the web
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_2.fastq.gz 

handy trick for finding runnable files:
/usr/bin/*.sh


# Nextera adapters
/opt/miniconda3/pkgs/trimmomatic-0.38-1/share/trimmomatic-0.38-1/adapters/NexteraPE-PE.fa

cd ~/dc_workshop
mkdir data/ref_genome

curl -L -o data/ref_genome/ecoli_rel606.fasta.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/017/985/GCA_000017985.1_ASM1798v1/GCA_000017985.1_ASM1798v1_genomic.fna.gz

curl -L -o sub.tar.gz https://ndownloader.figshare.com/files/14418248
tar xvf sub.tar.gz


for f in *.gz; do echo ${%_*}; zcat $f; | head -n 4; done > heads

---

# Resources

### BASH/UNIX
Shell novice guide: https://unr-dcg19.slack.com/archives/CDZ9692LA/p1547588745016300

https://www.tldp.org/LDP/Bash-Beginners-Guide/html/

https://linuxconfig.org/bash-scripting-tutorial-for-beginners

6 unix commands worth knowing: https://astrobiomike.github.io/bash/six_commands#tr

### Conda & Snakemake
+ Conda Cheatsheet: https://docs.conda.io/projects/conda/en/latest/_downloads/1f5ecf5a87b1c1a8aaf5a7ab8a7a0ff7/conda-cheatsheet.pdf
+ Snakemake tutorial using Singularity containers [here](https://snakemake2019.readthedocs.io/en/latest/awesome.html)
+ [Creating workflows with snakemake and conda](https://www.biostars.org/p/335903/)

### Tutorials

+ Lessons for RNA-sequencing Analysis from an earlier data carpentry workshop taught at Uni of Nevada, Reno - [Click Here](https://unr-omics.readthedocs.io/en/latest/) - you can follow these lessons on cyverse atmosphere

### Analyzing High Throughput Next-Gen Sequencing Data [ANGUS 2019](http://ivory.idyll.org/dibsi/ANGUS.html)

### Carpentries
+ [About Carpentries](https://carpentries.org/)
+ [**Become an Instructor**](https://carpentries.org/become-instructor/)
+ [Carpentry Con 2020](https://carpentrycon.github.io/carpentrycon2020/)
+ Snakemake Carpentry [lesson](https://hpc-carpentry.github.io/hpc-python/)

### CyVerse

+ [ASK CyVerse](http://ask.iplantcollaborative.org/questions/)
+ CyVerse Learning Center [Wiki](https://wiki.cyverse.org/wiki/dashboard.action;jsessionid=6DA6124290F22E541A21CC6EE6F5E66C)
+ [Resources Request Forms](https://user.cyverse.org/forms)


Sateesh
===

From keylie - Other snakemake files.
Telescope - pilot run example for someone in our group: https://github.com/gwcbi/AMPEL_telescope_pilot/blob/master/Snakefile
Fontana: https://github.com/kmgibson/FONTANA/blob/master/Snakefile

Ian
====

Richa
===
Hiroki
===
### Atmosphere Dashboard
Mouse over the allocation percents to see quotas<br>
Spawning a new image takes 5-10 min

Niel's command line introduction a great read is Neal Stephenson's "In the Beginning was the Commandline" esp the bit about Unix coders wanting to avoid carpal tunnel.

Slide for ls -l and permissions
o used for owner, g for group, u for users
probably better to have example be
u for user, g for group, o for other<br>
bc chmod u-w is change permission for the user
chmod o-w is change for all users

## Marcin's clever solution to getting to your html files if you don't have a way to download to your computer.

Navigate to the directory with your html files on your virtual machine and then issue the command
$ python -m http.server 8000
This launches a simple httpd in the directory where you issued the command that is visible from port 8000

then from your local computer point your web browser to
http://IP_address_of_your_machine:8000
you should see the files in that directory

This has no security whatsoever so would be a BAD IDEA to leave running.
You can hit Ctrl-C to kill the webserver process (but that will not affect your files)