Remove Human Reads from Metagenomic Sample FASTQ files (Test with 1 sample)

# Remove Human Reads from Metagenomic Sample FASTQ files (Test with 1 sample) Using information from:https://sites.google.com/site/wiki4metagenomics/tools/short-read/remove-host-sequences ## Overall Steps 1. Download Human Reference Genome 2. Build Bowtie2 index using the human reference genome 3. Map sample fastqs against the reference genome using bowtie2 4. Convert resulting SAM files to BAM files using samtools 5. Get unmapped pairs (did not match Human genome) using Samtools 6. Split bam files back into separated fastq files using bedtools #### Download Human Reference Genome & uncompress ``` wget -c ftp://ftp.ncbi.nlm.nih.gov/refseq/H_sapiens/annotation/GRCh38_latest/refseq_identifiers/GRCh38_latest_genomic.fna.gz gunzip GRCh38_latest_genomic.fna.gz ``` #### Build Bowtie2 index using the human reference genome ``` bowtie2-build GRCh38_latest_genomic.fna host_db ``` > Total time for backward call to driver() for mirror index: 00:59:58 #### bowtie2 mapping against host sequence database, keep both mapped and unmapped reads ``` bowtie2 -x host_db -1 SH5971_SA48774_S4_L008_R1_trimmed.fastq -2 SH5971_SA48774_S4_L008_R2_trimmed.fastq \ -S SH5971_SA48774_S4_L008_mapped_and_unmapped.sam ``` > 5856764 reads; of these: 5856764 (100.00%) were paired; of these: 3352513 (57.24%) aligned concordantly 0 times 2202085 (37.60%) aligned concordantly exactly 1 time 302166 (5.16%) aligned concordantly >1 times ---- 3352513 pairs aligned concordantly 0 times; of these: 345368 (10.30%) aligned discordantly 1 time ---- 3007145 pairs aligned 0 times concordantly or discordantly; of these: 6014290 mates make up the pairs; of these: 5790862 (96.29%) aligned 0 times 92875 (1.54%) aligned exactly 1 time 130553 (2.17%) aligned >1 times 50.56% overall alignment rate #### convert file .sam to .bam ``` samtools view -bS SH5971_SA48774_S4_L008_mapped_and_unmapped.sam > SH5971_SA48774_S4_L008_mapped_and_unmapped.bam ``` ### Filter required unmapped reads #### SAMtools SAM-flag filter: get unmapped pairs (both ends unmapped) ``` samtools view -b -f 12 -F 256 SH5971_SA48774_S4_L008_mapped_and_unmapped.bam > SH5971_SA48774_S4_L008_bothEndsUnmapped.bam ``` -f 12 Extract only (-f) alignments with both reads unmapped: <read unmapped><mate unmapped> -F 256 Do not(-F) extract alignments which are: <not primary alignment> #### Sort bam file by read name (-n) to have paired reads next to each other as required by bedtools ``` samtools sort -n SH5971_SA48774_S4_L008_bothEndsUnmapped.bam > SH5971_SA48774_S4_L008_bothEndsUnmapped_sorted.bam ``` #### Split sorted unmapped BAM file to R1 & R2 Fastq files ``` bedtools bamtofastq -i SH5971_SA48774_S4_L008_bothEndsUnmapped_sorted.bam -fq SH5971_SA48774_S4_L008_host_removed_r1.fastq -fq2 SH5971_SA48774_S4_L008_host_removed_r2.fastq ``` ## Success!!! ## Testing with non-trimmed fq & host_db in diff directory ``` cd /home/chiranjit/new_pipeline/batch8_fastqs mkdir host_rem ln -s SH5971_SA48774_S4_L008_R1_001.fastq.gz host_rem/sample1_R1_001.fastq.gz ln -s SH5971_SA48774_S4_L008_R2_001.fastq.gz host_rem/sample1_R2_001.fastq.gz cd host_rem bowtie2 -x /home/chiranjit/databases/hg38/host_db -1 SH5971_SA48774_S4_L008_R1_001.fastq.gz -2 \ SH5971_SA48774_S4_L008_R2_001.fastq.gz -S SH5971_SA48774_S4_L008_mapped_and_unmapped.sam samtools view -bS SH5971_SA48774_S4_L008_mapped_and_unmapped.sam > SH5971_SA48774_S4_L008_mapped_and_unmapped.bam samtools view -b -f 12 -F 256 SH5971_SA48774_S4_L008_mapped_and_unmapped.bam > SH5971_SA48774_S4_L008_bothEndsUnmapped.bam samtools sort -n SH5971_SA48774_S4_L008_bothEndsUnmapped.bam > SH5971_SA48774_S4_L008_bothEndsUnmapped_sorted.bam bedtools bamtofastq -i SH5971_SA48774_S4_L008_bothEndsUnmapped_sorted.bam -fq SH5971_SA48774_S4_L008_nohuman_R1.fastq \ -fq2 SH5971_SA48774_S4_L008_nohuman_R2.fastq grep -c "@" SH5971_SA48774_S4_L008_nohuman_R1.fastq ``` > 3349850 ## Success! # Test with small fileset ``` cd /home/chiranjit/batch8_fastqs mkdir hum_test gunzip -c SH5971_SA48774_S4_L008_R1_001.fastq.gz > sample1_R1_001.fastq gunzip -c SH5971_SA48774_S4_L008_R2_001.fastq.gz > sample1_R2_001.fastq head -1000 sample1_R1_001.fastq > hum_test/sample1a_R1_001.fastq head -1000 sample1_R2_001.fastq > hum_test/sample1a_R2_001.fastq tail -1000 sample1_R1_001.fastq > hum_test/sample1b_R1_001.fastq tail -1000 sample1_R2_001.fastq > hum_test/sample1b_R2_001.fastq cd hum_test find . -name "*fastq" -exec gzip '{}' \; mkdir host_rem_test mv hum_test host_rem_test cd host_rem_test /home/chiranjit/scripts/jpl_metg_hum_rem.sh 8 hum_test 12 ``` > total 180K drwxrwxr-x 4 chiranjit chiranjit 4.0K Oct 19 18:34 . drwxrwxr-x 4 chiranjit chiranjit 4.0K Oct 19 18:28 .. drwxrwxr-x 2 chiranjit chiranjit 4.0K Oct 19 18:29 batch8_human_removed drwxrwxr-x 2 chiranjit chiranjit 4.0K Oct 19 18:29 batch8_hum_intermeds -rw-rw-r-- 1 chiranjit chiranjit 15K Oct 19 18:28 sample1a_R1_001.fastq.gz -rw-rw-r-- 1 chiranjit chiranjit 17K Oct 19 18:28 sample1a_R2_001.fastq.gz -rw-rw-r-- 1 chiranjit chiranjit 15K Oct 19 18:28 sample1b_R1_001.fastq.gz -rw-rw-r-- 1 chiranjit chiranjit 20K Oct 19 18:28 sample1b_R2_001.fastq.gz ## Success!! find /home/chiranjit/batch8_fastqs/host_rem_test/batch8_fastqs/batch8_human_removed/ -maxdepth 1 -type f -exec ln -s {} \;

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.