# Done: Big Data & ML Fundamentals - 2 Mar 2021

> This Document: https://tinyurl.com/bdml-2mar2021 or https://hackmd.io/_0zrmB9vRsmP58DWPnH_tw?both

> Slides: https://1drv.ms/p/s!Aq6hYeVV5o6Dh4ARi1E1GOZ66r_Xjg?e=164CE6

## Class Resources & Notes
Revist today's course online (4 different ways):
1. https://www.coursera.org/learn/gcp-big-data-ml-fundamentals (Audit the course then you can do it for free)
2. https://cloudonair.withgoogle.com/events/apac-gcp-fundamentals-series
3. https://cloudonair.withgoogle.com/events/cloud-onboard-data-fundamentals
4. https://www.youtube.com/playlist?list=PLY7sQ59Bufns3VafkhnHpbdbGBrTxSXwi

#### General GCP Resources
- Check the learning paths: https://cloud.google.com/training#learning-paths
- https://www.gcpweekly.com/gcp-resources/
- https://github.com/gregsramblings/google-cloud-4-words

Types of Jobs on Cloud
1. **Migration**. We move the codebase's place of execution from on prem to on cloud
- Google Cloud has partner ecosystem and patterns are heavily involved in migration.
- Many of the technical guides related to migration: Find them here https://cloud.google.com/docs/tutorials
2. **Cloud Native Development** or New application: Generally startups do this or MNCs for a new application
3. Feature Addition on prexisting cloud env product
4. Maintenance

Components of Big Data Systems
* Databases SQL, No SQL, New SQL
* Data Lake & Data Warehouse
* Data Processing, ETL Pipeline
* Business Intelligence
* Artificial Intelligence: Machine Learning or Deep Learning

For each Product, Study
- Know what it does
- When to use it
- When not use it

OnPrem vs Cloud vs Serverless Cloud
- OnPrem - User configured, user managed and user maintained
- Cloud - User configured, provider managed and provider maintained
- Different Ways of Using Cloud
- (on prem is user bought, user configured & user maintained)
- Infrastructure as a Service: User configured, user maintained & Provider provided
- Platform as a Service / Managed Product: User configured, provider managed & maintained but partial work still is needed by user
- Fully Managed / Serverless: Everything is done by the provider. User just codes
- Example restaurant
- IaaS: You cook in the restaurant.
- PaaS: Buffet self service
- Serverless: Waiter serves you prepared food
- https://www.episerver.com/articles/pizza-as-a-service & https://www.bmc.com/blogs/saas-vs-paas-vs-iaas-whats-the-difference-and-how-to-choose/
- Serverless Cloud - Fully automated and no configuration required

### Module 2: Recommendations & Predictions with Hadoop
1. DataProc: Managed Hadoop. Nothing but Hadoop on GCP. (Called EMR/Elastic Mapreduce in aws and HDinsight in Azure)
- Cluster: Group of machines who work together parallely. It’s divide the work and do it parallel. That is the basis of big Data
- Migrate on prem Hadoop to On cloud Dataproc
- https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-jobs
- https://cloud.google.com/solutions/migration/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc
- https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-overview
2. Cloud SQL: Managed RDBMS (MySql, SQL server and PostgreSQL)
- OLTP
- Migrate on prem sql to Cloud SQL
- Migrate oracle to cloud sql https://cloud.google.com/solutions/migrating-data-from-oracle-to-cloud-sql-for-mysql or https://cloud.google.com/solutions/migrating-mysql-to-cloudsql-concept
- Migrate others

### Module 3: Data Warehouse, BigQuery, BigQuery ML
Keywords
1. AI or Artificial Intelligence
- AI is superset
- ML is subset of AI
- DL is subset of ML
- Data Science is ML or DL on Cloud using all related tools for the job
2. Data Science
3. ML or Machine Learning
- Tabular Data
4. DL or Deep Learning
- Unstructured Data. Images, Video, Audio, Text etc

Every AI is made up of
1. Data
2. Model
3. Infra to run the Model

Types of AI Problem
1. Recommendation AI
2. Value Prediction AI
3. Class Prediciton AI
4. Analmoly Detection AI

https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create
https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml

https://go.qwiklabs.com/qwiklabs-free

Extra
- https://www.youtube.com/watch?v=I64CQp6z0Pk&ab_channel=TED

---
## Questions
- [ ] test question
- [ ] test question 2
- [ ] xyz