Try   HackMD

Labs: Prerequisites

Welcome to the course of Big Data. This document will describe the list of software you will need in this course. Straight ahead!

Software used in the course:

Hardware Requirements

Processor: Virtualization Enabled (Check Virtualization on Windows, On Linux: lscpu). Sometimes it is disabled in BIOS.
Memory: We recommend at least 8Gb of RAM
Storage: To complete the first half of the course, you will need roughly 20Gb of free storage on your hard disk.
OS: macOS or Linux (latest Ubuntu LTS) are fine. You will have some additional issues with Hadoop and Spark on Windows, but it will work. If you want to use Linux - install it on hardware, do not use nested virtualization.

If you cannot meet these hardware requirements, please consult with your TA.

Refresh your knowledge of bash.

First Lab

For the first lab, install Virtual Box, Vagrant, and Docker.

Download Vagrant Box Image from Vagrant Repository.

Useful Software

Books:

Last changed by 
 
BigDataInnopolis
0
1024

Read more

Lab Block 1.1. VMs with Vagrant.

Lab Block 1: VMs with Vagrant This tutorial guides you through creating your first Vagrant project. Vagrant is a virtual machine management software. It allows us to create instructions for automatic configuration and deployment of virtual machines at scale. The configuration of virtual machines is easy to understand. We start with this topic and later will move further to learn how containers can replace virtual machines. We will do the following: Spin up a generic Ubuntu VM Install Apache server Perform port forwarding Learn how to create a multi-machine environment Connect multiple virtual machines with VPN

Dec 10, 2020
Spark, Recommender System

Spark, Recommender System The goal of this task is to finalize the implementation of a movie recommendation system. In the process, you will get more experience with programming in Scala and working with RDDs. Dataset: Modified MovieLens dataset (Mirror), also available in HDFS hdfs://namenode:9000/movielens-mod Project Template: Download the project and try compiling it. Compilation should complete without errors. (Mirror) Extra files: Download list of movies for collecting user preferences Inspecting and Running The Template Inspect the project template content.

Nov 16, 2020
Lab Block 1.2. Docker.

Lab Block 1: Docker This tutorial guides you through creating your first Docker container. It was heavily drawn from the official Docker Getting Started Guide. :::info This tutorial comes with a troubleshooting section that lists the most common problems. ::: How is Docker different from VMs? Virtual Machine emulates a fully working isolated OS. It requires the same resources from the host as a normal OS would, meaning it would load its kernel into the memory, load all necessary kernel modules, all the libraries to work with the software and only then will allocate resources for a user application. If you run 100 identical VMs, they would occupy 100 times more resources.

Nov 9, 2020
Assignment 2. Simple Sentiment Classification with Spark.

Assignment №2. Introduction to Big Data. Stream Processing with Spark Due Date: 8-th October 2019, 23:55 Teams: no less than three and no more than four students. Team representative should send list names to your TA Rule for new team: no ex-teammates. Assignment Details: Should be implemented in Scala. Cluster address is the same. Using this link, you can check the cluster status. In general, this task is not computationally intensive. (But we recommend you to perform hyperparameter search and cross-validation on the cluster) Stream Address: 10.90.138.32:8989 Report: Read Non-Technical Guide to Writing a Report to understand how to present your work in the best way. Submission Format: report, link to GitHub, compiled binaries, and an example of the output. Store your full outputs and your trained model in your groups' folder in HDFS. Grading policy: Individual grade is based on the role in the team (contribution). The description of the personal contribution should be provided in the report.

Sep 28, 2020
Read more from BigDataInnopolis
Published on HackMD