# DIBSI Notes for Tiger's Room: Day 1-2
[I'm a link to a markdown cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

- **Day 3** HackMD page: https://hackmd.io/CwDgZgpg7AJgTHAtBAxgVgIaOARhAZkRDQh0TQDYZwwoUAGCtfIA
- **Day 4** HackMD page: https://hackmd.io/MwdgrCBGAMCmCcBaEATAxgM0QFniaiAHMAIwBMiwhGY2aa2h8AhmUA==
- **Day 8** HackMD page: https://hackmd.io/MwMwDARgTBAmCGBaAjANgKwE5EBZn3UXgFN0IUSAOTYCYKVYqIA=
- **Day 10** HackMD page: https://hackmd.io/MYNgrMCcDsBMAMBaYAjeAORAWAZgQz0TyxViPmklgGYQBTAE2tCA?view
- **Day 11** HackMD page: https://hackmd.io/OwicEZQIwYwWgMwDMoDY4BYCm50A48s84dwE8ktUBDGLGIA=
- **Day 12** HackMD page: https://hackmd.io/GwYwhgHAjA7AJgIwLRQKxWEgLBGyEKrICmqAnAgMxkAM8MUxQA==?both

## Roadmap of our process

1. Sequence super awesome biological DNA/RNA for fun and amazing science!
2. Raw data = `.fastq` output
3. Quality assessment of the reads with **FastQC**
4. Trim off the adaptors and low quality reads with **trimmomatic**
5. Alignment of sequences with **BWA/Bowtie**
6. Aligned Data **SAM/BAM**

**If your goals is to do RNAseq**
- Transcriptome assembly!
- Differential Expression!
- Variant Calling!

**If you have ChipSEQ**

## Day 1

### Running command-line BLAST

*If you want extra explanation on this part of the tutorials, write here!*

**it is so easy to add notes to this thing!!!**

### Running large and long command line jobs - using shmlast!

* `&&` allows you to execute commands after one another
* ubuntu comes with an older version of python. We want a new one, so we install a newer one (python 3.5)
* `~` is a shortcut for your home directory (the directory you end up in when you connect to the server)
* `curl` is a command to download things from the internet. The `-O` option redirects the downloaded data into a file instead of printing it to your screen! (can also use `wget`)
* `less` is a command to read text files
* `nohup` can be used to run commands that take a long time. It avoids to lose your work if you get disconnect from the server / the internet

#### To get you caught up on the schmalst example

* `apt get`'s `-y` parameter allows you to skip entering 'yes'

# Install the necessary programs
sudo apt-get -y update && \
sudo apt-get install -y python3.5-dev python3.5-venv make \
    libc6-dev g++ zlib1g-dev last-align parallel
python3.5 -m venv ~/py3
. ~/py3/bin/activate
pip install -U pip
pip install shmlast>=1.2

# Download data
curl -O ftp://ftp.ncbi.nih.gov/refseq/M_musculus/mRNA_Prot/mouse.1.rna.fna.gz
# loop thru cow prot 1-8
# The -O option re-names the file in the destination folder the same 
for i in 1 2 3 4 5 6 7 8
do
   curl -O ftp://ftp.ncbi.nih.gov/refseq/B_taurus/mRNA_Prot/cow.$i.protein.faa.gz
done
gunzip -c cow.*.faa.gz > cow.faa

# Run shmlast!
shmlast crbl -q mouse.1.rna.fna.gz -d cow.faa --n_threads=6

#### A note on virtual environments

Creating a python environment is good for packages and dependencies. If you have a
pipeline that runs best with particular versions of programs- make a python environment
and install all the particular versions of programs. Then, when you leave this environment,
the intalled programs are contained. That way you do not have to install over your commonly
used programs.

> **Tip:** Use `deactivate` to exit a virtual environment!

#### A note on loops

"Loops" are a construct to execute the same command - or a set of commands - on a list of files or variables. Let's decompose our `curl` loop together:

for i in 1 2 3 4 5 6 7 8
do
   curl -O ftp://ftp.ncbi.nih.gov/refseq/B_taurus/mRNA_Prot/cow.$i.protein.faa.gz
done

for i in 1 2 3 4 5 6 7 8
do

means that `for` each of the numbers `1` to `8`, we will do something (i.e. executing a command). For each *iteration* of the loop, `i`, our variable, will take the value 1, 2, ... and 8.

   curl -O ftp://ftp.ncbi.nih.gov/refseq/B_taurus/mRNA_Prot/cow.$i.protein.faa.gz

will download data (with the `curl` command we've used earlier). Since `i` will take values 1 to 8, it means we'll download the files:

* cow.1.protein.faa.gz
* cow.2.protein.faa.gz
* ...
* cow.8.protein.faa.gz

done

tells our loop construct is `done`

for htop
sudo apt - get install -y htop

## RStudio Server

Several panels - each corresponding to different views of our environment.

Run the command: `getwd()`. It shows the path to working directory (similar to the UNIX command `pwd`).

> **Tip:** If your Institutional cluster/data center doesn't provide RStudio - but has R installed, use the infrastructure to do the heavy lifting, and install RStudio on your machine for the visuals/graphs.
>

If we have a variable data frame in R (in other words, a place containing structured information), we can do the following:

`dim(df)`: gives the dimensions of the variable (how many rows and columns)
`str(df)`: gives an overview of the structure. Each column is denoted with a `$`.
`df$column`: it will print out all the data from that one column (in this case, named `column`).

### Notes from R lesson

The data carpentry lessons on R can be found [here](http://www.datacarpentry.org/R-ecology-lesson/01-intro-to-r.html). The lessons are written in a self teaching structure so you can go over them while in the course or anytime you need a refresher.
- functions
- dplyr
- challenges
- etc.

## Day 2 (Wed, June 28)

### Unix lesson with Elijah

Some helpful commands in the shell
- `pwd`: print working directory
- `ls -a` : will show you *all* of the contents in the current directory
- `-l`: output in the long format .
- `.`: current directory
- `..`: Up a directory
- `mkdir`: makes a new directory
- `cd`: change directory
- <path> e.i. `cd /usr/Desktop/`
- `-`: previous directory
- `~`: home directory
> **Pro Tip:** Often when you make a directory, you next enter it. It is possible to make the directory and enter in one line: `mkdir <dir_name> && cd $_`. $ indicates that you are referring to a variable and $_ variable is the last word typed, so in our example <dir_name>.
- `touch test.txt`: creates an empty file named *text.txt*. touch can also be used to refresh the last time a file has been accessed which my prevent a file from being deleted by system admins that delete data on a timed basis
- `rm <filename>`: removes the filename. For ever! No turning back...
- -r: recursive, used for folders (on some systems the folder must be empty)
- -f: force

> **Tip:** A good idea to make sure you have the right file is to `ls <filename>` first, to make sure you have the right one, and then run the actual deletion `rm <filename>`.
>

- `cp`: copy

#### How to look at a file?

- `cat` displays to the screen
- `less` opens in a scrollable format
- `more` the same as less, but with less functionality
- `head` look at first 10 lines of a file
- -num: display first num of lines ex. `head -3` displays the first 3 lines
- `tail` look at last 10 lines of a file
- -num: display last num of lines ex. `head -3` displays the last 3 lines

`>` will write/overwrite the a file. It will take the output from a command to and direct it to a file.
`>>` will append the output to a file.

- `sort -n`: do a numerical sort
- `grep -v saturn planets.txt > goodbye.saturn.txt` : look for *saturn* in the *planets.txt* file
- `-i`: ignore case
- `-v`: invert match

`wc`: default counts the lines, words, bytes in a file or from standard input

> **Pro Tip:** Often when you make a directory, you next enter it. It is possible to make the directory and enter in one line: `mkdir <dir_name> && cd $_`. $ indicates that you are referring to a variable and $_ variable is the last word typed, so in our example <dir_name>.
>
> To remove all spaces from files in current directory:`for f in *\ *; do mv "$f" "${f// /_}"; done`
>
### Website with cool command-line trick for selecting and reformating data: [Grad Hustle](http://likit.github.io/category/linuxunix.html).

### Trimming and quality scores with Tristan

* `md5sum`
* command to verify if data that was downloaded was corrupted during the transfer
* `md5sum <file1> <file2>`
* `clear`
* erase contents of the terminal shell
* `chmod` = change modification
* command to modify file permissions
* read only = 444
* user | group | everyone
* 4 | 4 | 4 = read only for all
* if read and write = 6
* if read and write and executable = 7
* 777 = all users have read, write and executable permission for the file

## Will Trimble's Super Awesome Squencing Talk
Find Videos here: http://www.mcs.anl.gov/~trimble/flowcell/

## FastQC output

**What is fastQC?** We use FastQC to assess the overall quality of raw sequencing data, and determine the parameters that are important specifically to metagenomes!

**Overrepresented sequences:** If there is adapter content in your samples, it will show up here!

`k-mer`: a sequence of bases (usually 7 in FastQC - so essentially a 7-mer)

The quality of the reads for the first 10-15 bases, is usually lower, but this is not necessarily due to the actual quality of the read, but more of a technical decision from the Sequencer Manufacturer.

## Trimmomatic!

**What is trimmomatic you ask?** When DNA sequencing takes place, errors are inevitable. No sequencing method is perfect and some are drastically different than others in regard to sequence length and quality. We're going to trim the poor quality tail end sections of our Illumina reads. Although in general Illumina reads are very high quality, this degradation at the end of the sequencing run is typical of the Illumina sequencing platforms.

> **HINT** Using `which <command>` will show you if the software is installed, and where it's actually located.
>

Types of files:
>- **SAM files**: Sequence alignment map
>- **BAM files**: binary alignment map
>