# ANGUS Day 1 & 2 (07/02/18-07/03/18) notes - TIGERS

ANGUS notes http://angus.readthedocs.io/en/2018/

Jetstream https://use.jetstream-cloud.org/application

Basically Jetstream is a NSF funded cloud infrastructure

Task for today is to learn about Jetstream
* Now everyone has logged into the dibtiger account
* Start a new project under own name
* Select new instance
* Select DIBSI 2018 image
* Set instance name and say okay
* Wait for instance to be deployed and become active to use

Now we are on Cloud!!! Woo hoo!!

This is a **test of writing stuff** on HackMD

This is a [link in markdown](https://use.jetstream-cloud.org/application)

If you are having trouble signing into the hackmd on Safari, try a different browser -> Firefox or Opera or Chrome

Once the instance is active:
* Check size -> it is the size of the computer assigned to you
* IP address -> associated with instance
* Status -> active means ready to use
* Suspend -> kind of like pause on the computation so as to not consume resources
* Shelve -> For long term pause
* Stop -> To stop the computer, rebooting on start up, Don't Use when programs running in the background !!
* Reboot -> softer reboot of the system
* Redeploy -> Restart of the instance from scratch

Most commonly used options are stop and reboot

But we will leave the instances running -> Do not use any options yet!

How to choose the right limit for instance size ?
Depends on the type of analysis, but if you choose a large instance you will burn through the allocation quickly -> Choose wisely!!!

Compared to HPC (High Performance Computing), Instances in Jetstream are owned by you. You are are MASTER!

If things get stuck or going wrong -> delete and restart!

To actually start the instance, On your instance click on `open old web shell`

Once you see the black screen, check for the spiky wheel logo on the right hand top corner -> Settings
* Here you can change the size of the font -> Font Size
* Change screen color -> Theme

Other way to connect to the instance is via SSH key

SSH is Secure Shell
* Have a unique string of characters that defines the key
* Set up a protocol such that the unique key identifies your computer
* Use this to connect to Jetstream instance

To set up SSH, download the private key from slack channel to your home directory

More details on looging into jetstream [here](http://angus.readthedocs.io/en/2018/jetstream/login.html).

Go to terminal in Mac or in windows use BASH shell (Type Bash in search)

Example command to login:

`ssh -i angus_private_key dibtiger@129.114.16.114`

everyone has the same account -> dibtiger
Copy your unique IP address on your instance and paste for ssh

You should see the 'Atomsphere' & `dibtiger@js-xx-xxx:~$` on the screen

Now you are connected to the jetstream instance Yaay!!!!

Remember -> IP address is unique to the instance, but since here we are all using the same account, we can all share our workspace and instances if needed.

But in real world scenario, your IP address and account will be unique and to share with others, they have to give you permission to access

Now moving on to BLAST!!!

UNIX is full of puns ! :D

Some UNIXy things
* cd -> change directory
* pwd -> path of current working directory
* mkdir -> make directory ( -p stands for parent)
* ls -> list of things in the current directory
* curl -> see urls (-o for giving a name)
* wget -> world wide web and get it
* man -> manual (use it to get information about any command)
* gunzip -> to unzip `.gz` files
* head -> shows the first 10 lines of a file by default (use -n to show more than 10)
* less -> to look at the text type files in UNIX (newer version of `more`)

Note -> UNIX is not hard bent about the file suffixes. You can save say for example `.png` as `bob`

But it will be confusing for YOU latter so don'tt do it -> Better to give useful filenames to know what it is about

Remember -> If no comments or words after a command in UNIX is a GOOD THING -> It means nothing bad happened :)

What is blast ?

It is a program that takes two sequences of DNA and checks how similar or dissimilar they are.

Use tab complete -> It will save you time typing in UNIX
* Type the starting letter of the folder or file you want and press TAB and if the file exists, it will complete

Conda -> app manager -> Make it do the work for you!!
Just tell it the name of the program you want and it will take care of all installation that will be needed to run blast

NCBI BLAST -> https://blast.ncbi.nlm.nih.gov/Blast.cgi

Blast on NCBI vs on terminal ?

There are the same, but the advantage of using the terminal is automation! No more single sequence loading and clicking!

`ls -lh` -> for human legible listing
Note -> for the two flags for `ls` can grouped toether.
But in some cases the flag has a specific meaning and order in which case DO NOT GROUP. For example, in the `curl` command `-o` and `-l` cannot be combined.

`*` -> means anything
To be specific to match any and all patterns matching `.gz` files -> `*.gz`

FASTA -> a format for sequences

`>` -> alligator symbol!! Takes stuff on LHS and puts in RHS
blastx
* p -> proteins
* n -> nucleotides
* x -> proteins from one and nucleotides from another

## DAY 2 - Tuesday, 07/03/2018

### Unix commands we have seen so far
* `ssh` log into remote machine
* `curl` see url! download something from the web
* `ls` list the content in the folder
* ls -l show long format
* ls -a show hidden file
* ls -t show files by the modified time
* `cd` change directory
* `mkdir` make a directory
* `head` the first n rows of a file
* `man` check the manual of a command
* man ls and then type `/`, you can then search the string, press `n` for the next occurence.

* `conda install` install software through [conda](https://conda.io/docs/)
* `gunzip` g(zip)-unzip decompress a `.gz` file
* `zcat` print out a `.gz` file to the screen
* `makeblastdb` make a blast database
* `blastp` do blast for protein sequence
* `less` peek at your large file without loading all the content into memory.
* I do `less -S` every day. If you have lines that are very long, less -S will not wrap the lines.
* `screen` it is a program to tell the remote computer to ignore the internet connection and keep running even if you lost the internect.
* `echo` echo the content.
* `clear` to clear the screen.

In a screen session, you do `control + a` then press `d` to detach. `screen -r` to retach to your screen session.

If you are running your screen and droped the internet.
The next time when you log back (when your internect is back) to the remote machine by `ssh -i angus_key.txt [e-mail]` and then `screen -r` to re-attach to the screen session.

#### How to tell if you are in screen session

https://serverfault.com/questions/257975/how-to-check-if-im-in-screen-session

echo $STY
29883.pts-0.js-170-21

if you are in a screen session, it will print out something. Otherwise, it will be blank.

or

echo $TERM
screen

### some other useful concepts.

In command line, the space is very important.

head -n 11 myfile.txt

unix uses the space to separate the name of the command, the flag and the argument.

`head -n11 myfile.txt` will not work.

flag: `-n` is a flag which can change the behavior of the head command. A unix command can have many flags.
Flags start with a single dash (`-`) or two dashes (`--`).

argument: `myfile.txt` is the argument of the head command. A command can be called with more than one flag and more than one argument: but a command doesn’t always require an argument or a flag.

e.g if you just run `ls` it will list all the files in the current directory. If you do `ls -l myfile.txt`, `ls` will list the `long` format of the `myfile.txt`

**Do not have space in your file name**

wildcard: `*` glob
redirect: `>` save the output of a command to a file

pipe: `|` send the output of a command to a different command.
`zcat mouse.1.protein.faa.gz | head -n 498 > mm-second.faa`

In this example, we pipe the output of `zcat` to `head` and then redirect/save to `mm-second.faa`.

Usually, you can save some space to not have intermeidate files by piping. In this case, we do not decompress the `.gz` file and save some space.

if you do `ls -a`, hidden files will show up in the current directory, you will also see two special things: `.` and `..`.

`.` : is a short cut for current directory
`..`: is a short cut for the parent directory

you can do `cd ..` to go up one level of the directory.

`$`: it has different meaning depending where it is. When you log in, you will see `$` in your prompt. It can also refer to a variable.

e.g. `echo My username is $USER`
`echo` command recognizes the `$` and expand the `$USER` to the value of the variable, which is `dibtiger` in our case.

### why blast command line?

transcriptome.
5000 genes are differentially expressed.
Download the database, and blast to see what those genes are

Random sampling the reads. blast to see if the reads are for the species you sequenced. (contamination or the sequencing core gave you the wrong data)

NCBI database can have adaptor contamination.

### Rstudio

in the Source panel, type commands like in a regular text editor.

In mac, `command + enter` to execute the command where the cursor is located.

In Windows, use `Control + enter`.

In the `Console` panel, use `getwd()` to find where you are. It is similar to unix command `pwd`.

blast_out <- read.table('blast/mm-second.x.zebrafish.tsv', sep='\t')

The `blast/mm-second.x.zebrafish.tsv` path is relative to your current directory. use `getwd()` to confirm you are in your home directory. The `blast` folder is inside your home directory, you only need to specify `blast/mm-second.x.zebrafish.tsv` for R to find the file.

### R commands we have seen

* `read.table` read in the file to R.
* `colnames` get the column names of your dataframe or assign column names of your dataframe.
* `head` the first n rows of your dataframe, default is 6 rows.
* `dim` the dimension of your dataframe or matrix.
* `hist` make a histogram.
* `plot` make a scatter plot.

There are many **functions** to read a file into R. `read.table` is one of them. In the Console, type `?write.table` to get help for this function. Similiarly, type `?colnames` to get the help for this function.

use a `$` sign to access the column of a dataframe. e.g. `blast_out$evalue`

### quality control and trim reads

go to your home directory

#full path
cd /home/dibtiger

# relative path ~ is a short cut for home directory
cd ~/

# without any arguments will bring you to home directory as well

cd 

# make sure you are in the right place
pwd

[fastq format](https://en.wikipedia.org/wiki/FASTQ_format).

[phred score](https://en.wikipedia.org/wiki/Phred_quality_score) for fastq

* `md5sum -c` will check a file with the md5sum content.
* `mv` move/rename a folder/file. `man mv` for more
* `rm` delete a file or folder.
`rm` can only remove an empty folder, if it is not. you have to do `rm -f` to force remove. However, becareful, if you `rm` a file or folder, it is gone (forever). There is no trash bin in unix system. With greater power comes greater responsibility, use `rm -i` for interactive remove.

* `ln -s` make a soft link.
you can save space; no need to type the full path to the file.

* `nano` a text editor

**It is known that for illumina, the first several bases of RNAseq have low quality** http://seqanswers.com/forums/showthread.php?t=61936