Notepad_2019-04-11-gwu

===
Data Carpentry Genomics Workshop @ GWU 2019
Date: April 11-12th 2019
Location: Gelman Library, 2130 H St NW, Room 301/302, Washington, DC 20052
Workshop lessons (click on this link)

Pre-workshop survey
Post-workshop survey

Quick Markdown Introduction

This is a Heading

===
This is a link

  • point one
  • point two
  • point three

See this awesome page for more Markdown features

Announcements

Code of conduct

Bathrooms:

Instructors:

  • Niel Infante
  • Sateesh Peri

TA/Helpers

  • Will Bonnett
  • Adam Wong
  • Rebecca Clement
  • Keylie Gibson

Sign-In Here

General notes

Don't use spaces in filenames. You can't use spaces in variable names. Suggest using_underscores_instead. YouCanAlsoUseCamalCase

Append to a file within a loop:
for f in *gz; do zcat $f | head -n 4 >> heads ; done

Write the output of a loop to a file
for f in *gz; do zcat $f | head -n 4 ; done > heads

Example of writing the filename to the file before the contents of the file
for f in *.gz; do echo ${f%.fastq.gz}; zcat $f | head -n 4; done > heads

Using trimmomatic: see https://gwu-omics2019.readthedocs.io/en/latest/trimming.html for common trimming parameters
Remember that you need bioconda installed in your environment to use trimmomatic.
Trimming is very easy to parallelize because you are doing the same thing to many things.
Any command that displays in [ ] is optional. That doesn't mean you shouldn't use it.
Anything with a | in it is an OR (you can do one OR the other)

bwa mem referenceGenome input1 input2 > output
There are 2 inputs because these are paired files

Snakemake (a popular genomics workflow manager)

rule rule_name:
input:
output:
shell:
conda: (optional connection to the yml file outlining your environment)
snakefile must be named Snakefile (needs s to be capitalized)

You still need to do some shell scripting to figure out what output files to anticipate. I.e. you need to know what kind of files to expect from a tool.

Snakemake looks at the 1st rule by default
otherwise must give snakemake the specific rule that you want it to run

Rule all says "hey snakemake, I rely on these outputs in order to do my work, so they are your inputs."

Order of operations for running rules:

  1. rule all (any outputs then dependencies)
  2. first rule
  3. stop

Try:
snakemake -j 4
will send different samples to different cores (4 == number of cores)

Other Popular Workflow management options include:

  • Common workflow language;
  • Make;
  • Nextflow (supposed to be best on HPCs)

Questions for instructors/helpers?

Q) What's the difference between Reboot and Redeploy for instance actions?

A) You reboot your machine while it is already running but you can redeploy a stopped machine with the same image to start it up.

Q) What does the black line on the quality score graph mean?

Q) Yesterday, Niel mentioned that he never trims sequences for RNAseq because poor quality reads don't affect downstream processes. Can he or someone else elaborate on that? I'd love to get some more information on best practices like that for RNAseq analysis for differential expression.

Q) I would love to be able to implement RStudio through CyVerse to do some differential expression analysis on bam files for the class I'm teaching on Wednesday (!!!) – is this even possible in that timeframe??

Q) Will we have access to the gwu-omics website after the workshop?

A) YES, Forever

Q) Is there a good rule of thumb to use to predict what instance size you need?

A) There is really no good way to predict the instance sizes you might need. You will have to try different sizes. Here is the Cyverse wiki about AUs (https://wiki.cyverse.org/wiki/display/atmman/Requesting+More+Atmosphere+Resources)

Q) What does "force a refresh" in the dashboard do? Does it just update the browser or is it like a redeploy?

Q) I ran into a Fail2Ban issue by trying to ssh into my vm when the network was up but the machine wasn't fully deployed.

A) Need to use the web-cli interface, and unban. From Julian in support "If you don't mind, please ask the instructor to let the other students know about the web shell + unban_help trick if they can't SSH into their instances."

Log of shell commands

curl -L https://osf.io/vuk5y/download -o shell_data.zip

unzip shell_data.zip

ls   # lists contents in directory
pwd  # path location
cd   # change directory
*    # wildcard. Good for searching through directories
head # returns the first 10 lines
head -n 8 file.name # returns the first 8 lines
tail # returns the last 10 lines
less # scrollable content (hit q to leave)
/   # search within results
rm  # removes the file forever!
cp  # copy. Syntax: cp original_file new_file
mkdir # make directory
mv  # move file
chmod  # change command permissions. 
    To take away write permissions:
        chmod -w file.name
        Good for data from Sequencing Center :)
    Grant write permissions:
        chmod +w file.name
grep  # search with pattern matching
grep -c # count of lines that match a given pattern
echo # returns whatever you give the command as an output
for _ in _ # loop
do # what we want it to do
# String loops together with ;
example: for x in a b c d e; do echo $x; done
{start stop step} # create a range of numbers
cat # displays the output of a file onto the terminal
zcat # like cat but for gzipped files
set -e # fail safe within a shell script that says "stop here if something is not working right"


# with flags

ls -a # show hidden files
cd . # current directory
cd .. # up one directory

# Get data from the web
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_2.fastq.gz 

handy trick for finding runnable files:
/usr/bin/*.sh


# Nextera adapters
/opt/miniconda3/pkgs/trimmomatic-0.38-1/share/trimmomatic-0.38-1/adapters/NexteraPE-PE.fa

cd ~/dc_workshop
mkdir data/ref_genome

curl -L -o data/ref_genome/ecoli_rel606.fasta.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/017/985/GCA_000017985.1_ASM1798v1/GCA_000017985.1_ASM1798v1_genomic.fna.gz

curl -L -o sub.tar.gz https://ndownloader.figshare.com/files/14418248
tar xvf sub.tar.gz


for f in *.gz; do echo ${%_*}; zcat $f; | head -n 4; done > heads

Resources


BASH/UNIX

Shell novice guide: https://unr-dcg19.slack.com/archives/CDZ9692LA/p1547588745016300

https://www.tldp.org/LDP/Bash-Beginners-Guide/html/

https://linuxconfig.org/bash-scripting-tutorial-for-beginners

6 unix commands worth knowing: https://astrobiomike.github.io/bash/six_commands#tr

Conda & Snakemake

Tutorials

  • Lessons for RNA-sequencing Analysis from an earlier data carpentry workshop taught at Uni of Nevada, Reno - Click Here - you can follow these lessons on cyverse atmosphere

Analyzing High Throughput Next-Gen Sequencing Data ANGUS 2019

Carpentries

CyVerse




Sateesh

From keylie - Other snakemake files.
Telescope - pilot run example for someone in our group: https://github.com/gwcbi/AMPEL_telescope_pilot/blob/master/Snakefile
Fontana: https://github.com/kmgibson/FONTANA/blob/master/Snakefile

Ian

Richa

Hiroki

Atmosphere Dashboard

Mouse over the allocation percents to see quotas

Spawning a new image takes 5-10 min

Niel's command line introduction a great read is Neal Stephenson's "In the Beginning was the Commandline" esp the bit about Unix coders wanting to avoid carpal tunnel.

Slide for ls -l and permissions
o used for owner, g for group, u for users
probably better to have example be
u for user, g for group, o for other

bc chmod u-w is change permission for the user
chmod o-w is change for all users

Marcin's clever solution to getting to your html files if you don't have a way to download to your computer.

Navigate to the directory with your html files on your virtual machine and then issue the command
$ python -m http.server 8000
This launches a simple httpd in the directory where you issued the command that is visible from port 8000

then from your local computer point your web browser to
http://IP_address_of_your_machine:8000
you should see the files in that directory

This has no security whatsoever so would be a BAD IDEA to leave running.
You can hit Ctrl-C to kill the webserver process (but that will not affect your files)

Select a repo