UE21CS343AB2 Big Data Lab 2

# UE22AM251B Big Data Lab 2 Parallel Computations through Map-Reduce ## Assignment Objectives and Outcomes MapReduce is a programming model and a parallel processing framework for distributed computing that was originally developed by Google. It is designed to process and generate large datasets that can be distributed across clusters of computers in a highly scalable and fault-tolerant manner. The objective of this lab is to understand and apply MapReduce for basic genomic data analysis tasks. ## Software/Languages to be used 1. Python ```3.10.x``` 2. Hadoop ```3.3.6``` ## Environment Setup 1. Set up a Hadoop environment, if it's already not setup and ready to go from Lab 1. 2. You will be working with the condensed and clean version of this [NCBI Dataset](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/reference-docs/data-packages/genome/) available to download from this [link](https://drive.google.com/file/d/15UsuKaN0t4HqVXIsSMNDTVcaOVbRphsN/view?usp=sharing). The above database (GDrive link) shall be used for all the three tasks below. > .fna extension stands for the FASTA format DNA and protein sequence alignment file that stores DNA information that can be used by molecular biology software. ## Tasks Background MapReduce can be used in genomics data analysis to efficiently process and analyze large genomic datasets. Genomics involves studying DNA sequences, gene expressions, and other biological data, and the sheer volume and complexity of this data make it well-suited for a parallel processing framework like MapReduce. Genomics data analysis is done through the following steps: 1) Data Preparation 2) **Mapping** 3) Variant Calling 4) Quality Control 5) Functional Annotation 6) Population Genetics 7) Pathway Analysis 8) Data Storage and Management 9) Parallel Processing You will work on the **Mapping** Phase and learn how to perform basic DNA sequence analysis tasks using the MapReduce framework. ## Task Specifications 1. **Finding Motifs** * Write a MapReduce program to identify the specific sequence motif '**ATGTG'** in the genomic data. > Motifs are short, recurring patterns of nucleotides that have biological significance. ### Expected output ``` ATGTG 56 ``` 2. **Finding GC Content** * Write a MapReduce program to calculate the GC content (the percentage of G and C nucleotides) in the genomic data. * *Assumptions:* * Take the threshold for GC-rich regions to be classified so to be **0.7** * Take the window size for the sequence to be **10** ### Expected output ``` 1-10 0.9 10-19 1.0 11-20 1.0 12-21 1.0 13-22 1.0 14-23 0.9 15-24 1.0 16-25 1.0 17-26 1.0 18-27 1.0 19-28 1.0 2-11 1.0 20-29 0.9 21-30 0.9 22-31 0.9 23-32 1.0 24-33 1.0 25-34 1.0 26-35 1.0 27-36 1.0 28-37 1.0 29-38 1.0 3-12 1.0 30-39 0.9 31-40 0.9 32-41 0.9 33-42 0.9 34-43 0.9 35-44 0.9 36-45 0.9 37-46 0.9 38-47 0.9 39-48 0.9 4-13 1.0 40-49 0.9 41-50 0.9 42-51 1.0 43-52 1.0 44-53 0.9 45-54 0.9 46-55 0.9 47-56 0.9 48-57 0.9 49-58 0.9 5-14 1.0 50-59 1.0 51-60 1.0 52-61 1.0 53-62 1.0 54-63 1.0 55-64 0.9 56-65 0.9 57-66 0.9 58-67 0.9 59-68 0.9 6-15 1.0 60-69 0.9 61-70 0.9 62-71 0.9 63-72 0.9 64-73 0.9 65-74 1.0 66-75 1.0 67-76 0.9 68-77 0.9 69-78 0.9 7-16 1.0 70-79 0.9 71-80 0.9 8-17 1.0 9-18 1.0 ``` 3. **Nucleotide Distribution by Position** * Write a MapReduce program that calculates the distribution of each nucleotide at different positions in the genomic sequence. * *Assumptions:* * Calculate the distribution for the first **10** positions only ### Expected output ``` 10_A 160 10_C 153 10_G 121 10_N 125 10_T 167 10_a 88 10_c 54 10_g 54 10_t 77 1_A 180 1_C 125 1_G 122 1_N 125 1_T 174 1_a 87 1_c 60 1_g 48 1_t 78 2_A 176 2_C 135 2_G 146 2_N 125 2_T 145 2_a 103 2_c 49 2_g 45 2_t 75 3_A 174 3_C 142 3_G 141 3_N 125 3_T 142 3_a 90 3_c 47 3_g 49 3_t 89 4_A 169 4_C 126 4_G 130 4_N 125 4_T 173 4_a 87 4_c 55 4_g 55 4_t 79 5_A 140 5_C 147 5_G 147 5_N 125 5_T 170 5_a 91 5_c 59 5_g 32 5_t 88 6_A 161 6_C 153 6_G 122 6_N 125 6_T 168 6_a 87 6_c 68 6_g 40 6_t 75 7_A 149 7_C 134 7_G 141 7_N 125 7_T 178 7_a 94 7_c 55 7_g 34 7_t 89 8_A 141 8_C 160 8_G 145 8_N 125 8_T 152 8_a 96 8_c 59 8_g 36 8_t 85 9_A 157 9_C 162 9_G 107 9_N 125 9_T 169 9_a 92 9_c 42 9_g 50 9_t 95 ``` ## Task Deliverables 1. Load the data into HDFS. 3. Run your code on the dataset until you get the right answer 3. After completing Tasks 1,2 and 3, you shall name the 6 pairs of mapper-reducer python files with each pair pertaining to 1 task with the following naming conventions - ``` SRN_mapper1.py SRN_reducer1.py SRN_mapper2.py SRN_reducer2.py SRN_mapper3.py SRN_reducer3.py ``` 4. Please save all these files in a folder with the following naming convention- ``` Section_SRN_Lab2 ``` Zip this folder and submit it in the form link mentioned in your assessment announcement. ## Submission Guidelines You will need to make the following changes to your ```mapper.py``` and `reducer.py` scripts to run them on the portal. 1. Include the following shebang on the first line of your code ```#!/usr/bin/env python3``` 2. Convert your files to executables ```chmod +x mapper*.py reducer*.py``` 3. Convert line breaks in DOS format to Unix format (this is necessary if you are coding on Windows - your code will not run on our portal otherwise) ```dos2unix mapper*.py reducer*.py``` ## Running the MapReduce Job without Hadoop A MapReduce job can also be run without Hadoop. Although slower, this utility helps you debug faster and helps you isolate Hadoop errors from code errors. ``` cat path_to_dataset | python3 mapper.py [command line arguments] | sort -k 1,1 | python3 reducer.py [command line arguments] > output.txt ``` ## HDFS Operations The HDFS supports all file operations and is greatly similar to the file system commands available on Linux. You can access HDFS on command line using `hdfs dfs` and use the `-` prefix before the file system command to execute general Linux file system commands. ### Loading a file into HDFS A file can be loaded into HDFS using the following command. `hdfs dfs -put path_to_file /hdfs_directory_path` ### Listing files on HDFS Files can be listed on HDFS using ```hdfs dfs -ls /hdfs_directory_path``` Similarly, HDFS also supports `-mkdir` ,` -rm `and more. ### Displaying contents of files on HDFS File contents can be displayed using `hdfs dfs -ls /hdfs_directory_path` ## Running a MapReduce Job A MapReduce job can be run using the following command ``` hadoop jar path-to-streaming-jar-file \ -input path_to_input_file_on_hdfs \ -output path_to_output_folder_on_hdfs \ -mapper absolute_path_to_mapper.py command_line_arguments \ -reducer absolute_path_to_reducer.py command_line_arguments ``` ```path-to-streaming-jar-file``` is ```$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-3.3.6.jar``` if you have followed the previous lab's tutorial on installing Hadoop.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.