# CSU Monterey Bay
# Data Carpentry: Genomics
## 21-22 Mar 2019

## Workshop website: https://kweitemier.github.io/2019-03-21-csumb/

### Welcome!
This is a document that any and all of us can use to take notes and share difficult-to-type information (e.g. links). To start editing, click on the icons above that look either like a pencil (editing only) or a book (both editing and viewing). You can just start typing anywhere in the dark panel.

Try it out! Type your name below:
1. **Brook Moyers**
2. **Kevin Weitemier**
3. **Simon Titen**
4. **Austin Diaz**
5. **Isabella McCrory**
7. **Aparna Sreenivassamp**
8. **Jose M Andrade Lopez**
9. **Cecile Mioni**
10. **Vanessa Garcia**
11. **Joaquin Sanchez**
12. **Chris Preston**
13. **Paul Minor**
14. **Josue Duque**
15. **Paul Bump**
16. **ⒼⒶⒷⒺ Chavez**
17. **Setsuka Aust**

### Accessing your Amazon Web Services (AWS) instance
Data Carpentry has created a temporary AWS instance for each student to use during the workshop. For directions on how to access an AWS instance visit the lesson [Logging onto Cloud](https://datacarpentry.org/cloud-genomics/02-logging-onto-cloud/index.html).

Each instance has its own unique address. All of the addresses for this workshop are listed in the table below. Add your name next to an address to mark which you will be using for the duration of the workshop.

All instances have the same username and password.
Username: **dcuser**
Password: ***see the whiteboard***

| AWS Address | Name |
| ------------------------------------------ | ---------- |
| ec2-3-83-107-33.compute-1.amazonaws.com |*Vanessa* |
| ec2-54-146-175-116.compute-1.amazonaws.com |*ⒼⒶⒷⒺ* |
| ec2-54-174-29-243.compute-1.amazonaws.com |*Skyler* |
| ec2-3-93-218-145.compute-1.amazonaws.com |*Simon* |
| ec2-3-82-146-221.compute-1.amazonaws.com |*Simon* |
| ec2-18-208-115-125.compute-1.amazonaws.com |*JMAL* |
| ec2-3-86-246-208.compute-1.amazonaws.com |*Paul Minor*|
| ec2-3-95-193-101.compute-1.amazonaws.com |*Paul Bump* |
| ec2-3-91-57-118.compute-1.amazonaws.com |*Josue D* |
| ec2-54-165-187-154.compute-1.amazonaws.com |*Cecile* |
| ec2-3-91-218-48.compute-1.amazonaws.com |*Joaquin* |
| ec2-52-90-237-195.compute-1.amazonaws.com |*Chris* |
| ec2-34-207-87-99.compute-1.amazonaws.com |*Aparna* |
| ec2-35-171-163-141.compute-1.amazonaws.com |*SSet* |
| ec2-54-157-224-166.compute-1.amazonaws.com |*Preethy* |
| ec2-3-87-72-153.compute-1.amazonaws.com |*Bella* |
| ec2-54-211-103-192.compute-1.amazonaws.com |*Jacob* |
| ec2-54-87-236-46.compute-1.amazonaws.com |*Nicholas* |
| ec2-184-72-137-79.compute-1.amazonaws.com |*Austin* |
| ec2-52-201-234-228.compute-1.amazonaws.com |*Sasha* |

---

### Data Tidiness

#### Discussion
With the person next to you, discuss:

**What kinds of data and information might have been generated before sending your DNA/RNA off for sequencing?**

Write some of your answers below:

- Group members: Answers
- Group #1 Aparna and Chris -- (sample numbers, barcode, location, time series, replicates, normalization, primer type/target genes, metadata about the sample, sample collection, sample processing)
- Group #2: Simon & Cecile (sample type, sample ID/name, source, date of collection, concentration/quantity/purity, primers used for sequencing, method of preparation/extraction/purification, sample storage condition, sequencing well #/sequencing order #)
- Group #3: Vanessa, Austin & Bella (sampling site, type of organism, tissue type, environmental conditions, date sample is collected, date of pcr)

- Group Josue Jose: (Primers, DNA kit used, adapters, sample ID, genes of interest, tissue type, species name, control vs test)

- Group: Paul² -- sample name, collection date, descriptive data (species name, quantity, staging), concentration, quality, indexing barcode
- Group: Happy Muffins: - primer info, gene name, length of sequence, chromosome number,

- Group: Joaquin , skyler- where the sample came from,

#### Exercise

[Download](https://github.com/datacarpentry/organization-genomics/raw/gh-pages/files/Ecoli_metadata_composite_messy.xlsx) or view the potential spreadsheet data generated about a sequencing experiment. You can look at the image, or download the file to you computer via this [link](https://github.com/datacarpentry/organization-genomics/raw/gh-pages/files/Ecoli_metadata_composite_messy.xlsx) and open it in a spreadsheet reader like Excel.
**With the person next to you, for about 2 minutes, discuss some of the problems with the spreadsheet data.**

- Group #2: inconsistency in formatting of variable titles/data, space in variable title/long name, multiple metadatabases on the same tab, multiple dataset per columns (should be separate columns), incomplete/inconsistent description

- Group: Paul²: Consistency across spreadsheets in columns and rows, plain formatting - no color, no empty cells, no random titles/descriptions outside of the columns and rows
- Group Josue Jose: data not in consistent columns, empty cells in the data, data that should be in columns is above the tables, reference could be its own column
- Group: Joaquin and Skyler - color coded, two sheets of data, inconsistant columns throughout the different groups, spelling of column headers is different in the different groups.

### Planning for NGS Projects

#### Discussion
Discuss the following with your neighbor.
- **What challenges do you think you’ll face (or have already faced) in working with a large sequence dataset?**
- **What is your strategy for saving and sharing your sequence files?**
- **How can you be sure that your raw data have not been unintentionally corrupted?**
- **Where/how will you (did you) analyze your data - what software, what computer(s)?**

- Group #2: Error in copying/pasting when large metadatabase that don't fit in the window screen, out of date software, errors when multiple people use/update the same database, separate backup to prevent corrupted file/virus, share as uneditable/read only, use lab computer for official raw data storage rather than personal computer, SOP/training on how to enter/edit data

- Group: Paul²: Data storage (local vs remote), have backups, know how to analyze data..., server access, convincing your boss you need the $$$ for server/software

<!---
#### Exercise
(We skipped this exercise for time)

Download the following [sample submission sheet](https://datacarpentry.org/organization-genomics/files/sample_submission.txt) and answer the following questions.

1. What are some errors you can spot in the data? Typos, missing data, inconsistencies?
2. What improvements could be made to the choices in naming?
3. What are some errors in the spreadsheet that would be difficult to spot? Is there anyway you can test this?
--->

#### Exercise
When the data come back from the sequencing facility, you will receive some documentation (metadata) as well as the sequence files themselves. Download and examine the following example file - here provided as a [text file](https://datacarpentry.org/organization-genomics/files/sequencing_results_metadata.txt) and [Excel file](https://datacarpentry.org/organization-genomics/files/sequencing_results_metadata.xls).

1. How are these samples organized?
2. If you wanted to relate file names to the sample names submitted above (e.g. wild type…) could you do so?
3. What do the _R1/_R2 extensions mean in the file names?
4. What does the ‘.gz’ extension on the filenames indicate?
5. What is the total file size - what challenges in downloading and sharing these data might exist?

- Group #1:

- Group #2: tab separated (tsp), we skipped the sample submition exercise,

- Group: Paul²: data organized by sample ID, paired ends reads, R1 and R2 represent forward and reverse sequences of the paired end read, gzip file, 1113.60gb (lots of data!)

- Group: Jose Josue: Data sorted by sample ID aand every row describes one output file.

---

SRA- where raw sequence reads are stored through NCBI.
European nucleotide archive and NCBI- clearinghouses of information
Could possibly build projects from the data available from these archives.

### Thursday afternoon

#### Exercise
Starting in the `shell_data/untrimmed_fastq/` directory, do the following:

1. Make sure that you have deleted your backup directory and all files it contains.
2. Create a copy of each of your FASTQ files. (Note: You’ll need to do this individually for each of the two FASTQ files. We haven’t learned yet how to do this with a wild-card.)
3. Use a wildcard to move all of your backup files to a new backup directory.
4. Change the permissions on all of your backup files to be write-protected.

#### Exercise
1. How many sequences in SRR098026.fastq contain at least 3 consecutive Ns?
2. Search for the sequence GNATNACCACTTCC in the SRR098026.fastq file. Have your search return all matching lines and the name (or identifier) for each sequence that contains a match.
3. Search for the sequence AAGTT in both FASTQ files. Have your search return all matching lines and the name (or identifier) for each sequence that contains a match.

#### Exercise
Now that we know about the pipe (|), write a single command to find the number of reads in the SRR098026.fastq file that contain at least two regions of 5 unknown nucleotides in a row, separated by any number of nucleotides. Do this without creating a new file.

For a challenge, you can try writing a regular expression for at least two regions of 5 unknown nucleotides in a row, separated by any number of **known** nucleotides (A,C,G,T).

This works as long as you interpret **any** as at least one `grep NNNNN[ACGT]*NNNNN SRR098026.fastq`

#### Exercise
How many single-end libraries are in our samples?

#### Exercise
1. How many different sample load dates are there?
2. How many samples were loaded on each date?

#### Exercise
We want the script to tell us when it’s done.
Open bad-reads-script.sh and add the line echo "Script finished!" after the grep command and save the file.
Run the updated script.

Optional extras:
1. How many bad reads are there in the two FASTQ files combined?
2. How many bad reads are in each of the two FASTQ files? (Hint: You will need to use the cut command with the -d flag.)

### Downloading data
wget ftp://ftp.ensemblgenomes.org/pub/release-37/bacteria/species_EnsemblBacteria.txt

----

## Friday

#### Data Wrangling and Processing

Use the following commands to download files onto your AWS instance.

mkdir -p ~/dc_workshop/data/untrimmed_fastq/
cd ~/dc_workshop/data/untrimmed_fastq

curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/003/SRR2584863/SRR2584863_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/004/SRR2589044/SRR2589044_2.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_1.fastq.gz
curl -O ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR258/006/SRR2584866/SRR2584866_2.fastq.gz

Where to find trimmomatic, in case it is not in your path:

cp ~/.miniconda3/pkgs/trimmomatic-0.38-0/share/trimmomatic-0.38-0/adapters/NexteraPE-PE.fa .

---

### Please take the [Post-workshop survey](https://www.surveymonkey.com/r/dcpostworkshopassessment?workshop_id=2019-03-21-csumb)!