# Done: Big Data & Machine Learning Fundamentals - 14 April 2021
> This Document: is.gd/ivunax
> Slides: https://1drv.ms/b/s!Aq6hYeVV5o6Dh4BdXur3Iy1c-PjbtQ?e=VM3Ona
---
## Notes
### Gcp Introduction
General GCP Resources
- Check the learning paths: https://cloud.google.com/training#learning-paths
- Details of Certification: https://www.evernote.com/shard/s295/sh/ab8acf7b-98b0-46b3-afbd-3756b46a825e/ffb53c4f70d0fe7fb85f56a9a80bad2f
- No recording allowed, but Revist today’s course online (4 different ways):
1. https://www.coursera.org/learn/gcp-big-data-ml-fundamentals (Audit the course then you can do it for free. https://www.classcentral.com/report/coursera-signup-for-free/) or on pluralsight
2. https://cloudonair.withgoogle.com/events/apac-gcp-fundamentals-series
3. https://cloudonair.withgoogle.com/events/cloud-onboard-data-fundamentals
4. https://www.youtube.com/playlist?list=PLY7sQ59Bufns3VafkhnHpbdbGBrTxSXwi
- Compare Cloud Providers
1. http://comparecloud.in/
2. https://cloud.google.com/docs/compare/azure
3. https://cloud.google.com/docs/compare/aws

Different roles in Data Team Hypothetical Scenario
- 1x or 2x Infra (Physical, IaaS or Cloud)
- 1x DevOps (Stack automation, Containers and Platform as a service)
- 2x or 3x Data Engineer (ETL, Data Pipelines, Data Automation, Data as a Service, Data Ingestion)
- 2x Analytics (1x Batch Analytics, 1x Real-Time Analytics and Predictive APIs)
- 1x AI and ML Data Scienst (Machine Learning and AI algorithms)
- 1x Front-End Dev (Web and Js developer, web and mobile apps)
- Specialized Roles
- 1x Network Architect
- 1x Security Engineer
- 1x Data Viz Developer

Types of Jobs on Cloud
1. Migration. We move the original codebase from on prem to on cloud
- Google Cloud has partner ecosystem and parterns are heavily involved in migration.
- Migration Jobs
1. Step 1: Move application as is (Lift & Shift). We are simply running the same application but at different place
1. Most demand for Cloud Enabled people
2. Step 2: Improve tdhe application to take advantage of Cloud Products.
3. Step 3: (Cloud Native / Serverless) Rewrite everything for cloud Native. No maintainence required
4. Can find many guide here: https://cloud.google.com/docs/tutorials/
2. Cloud Native Development or New application: Generally startups do this
- Cloud Architect is Design & Develop the system
- Design consideration for Scale for security for future growth all these are part of architect’s job
- Cloud Developer just Develop the system
- Architect vs Civil Engineer
- Civil Engineer Builds the building according to the blueprint.
- Architect designs the blueprint considering the feasibility of the building.
- Movies Reference: Inception Dream Designer: Architect
3. Feature Addition on prexisting cloud hosted product
4. Maintainance of the cloud hosted Product

Evolution to Cloud
1. First On Prem either server or data center
- My Infra, I buy it, I manage it
2. Then On Cloud. (First migration called lift & shift)
- Someone else’s infra
- Renting not Buying. But renting gives flexibility
- House Rent vs Buy?
- Renting gives us abilrity to change
3. Then Cloud Native / Serverless / Fully Managed
- Someone else’s Infra, but completely automatically managed
- Iron Man? Mark I vs Mark 85 in End Game. nano tech.

Migration to Cloud
1. Lift & Shift. Move as is
2. Optimize for Cloud. Tiny rewrite to improve the performance
3. Complete Rewrite for Cloud Native or Containerize for Hybrid cloud via Container

OnPrem vs Cloud vs Serverless Cloud
- OnPrem - User configured, user managed and user maintained
- Cloud - User configured, provider managed and provider maintained
- Different Ways of Using Cloud
- (on prem is user bought, user configured & user maintained)
- Infrastructure as a Service: User configured, user maintained & Provider provided
- Managed Product (Partially Managed by us and Partially by the Provider): User configured, provider managed & maintained but partial work still is needed by user
- Fully Managed / Serverless: Everything is done by the provider. User just codes
- Example restaurant
- IaaS: You cook in the restaurant.
- PaaS: Buffet self service
- Serverless: Waiter serves you prepared food
- https://www.episerver.com/articles/pizza-as-a-service & https://www.bmc.com/blogs/saas-vs-paas-vs-iaas-whats-the-difference-and-how-to-choose/
- Serverless Cloud - Fully automated and no configuration required

### Module 1

Big Data
- data analysed is just 1% of all data produced, thus huge potential for growth in this field
- Cloud helps analyze such a large scale data easier compared to on prem systems
Data Scientist vs Data Engineer

- Overlapping skillset but different specialisation
- Data Science is overcrowded,if you are really good then go for it. Otherwise choose Data Engineering if interested.
- Demand Ratio is 3:1 , where as supply ratio is 1:3.
- Concentrate and focus less crowded areas otherwise be the best in crowd.
Google Cloud different products

- Compute
- Custom Hardware TPU, best choice for ML workloads
- ASIC -> Used by Red Hat OpenShift Container Platform(OCP) & IBM CP4A as well is nothing but a TPU.
- Storage
- Networking
- Networking in Google Cloud is powerful
- Private fiber optic cables for better networking with high bandwidth in Google Cloud
- Security.
- Security is a shared responsibilityl
- IBM Security Services product a.k.a Managed Security Services (M.S.S) is a Global Market Leader in Web and Cyber Security space worldwide
- Big Data & ML Product
Summary:

- Explosion or Growth of Data,
- Cloud playing an important role in Big Data,
- Data Scientist v/s Data Engineers (Demand and Supply),
- Big Data & Analytics + Cloud Migration/Modernisation
- With advent of technologies like Kubernetes, Docker containers and Red Hat OpenShift powered with IBM CP4A — High Availability is achievable.

### Module 2:

- Scale
- Autoscale
- Change according to demand automatically either increases or decreases without user intervention
- Eg: room of requirements from Harry Potter
- Scale
- You have to execute the command which can resize the resources either up or down. This is not automatic
- eg: Ant Man. Red is shrinking and blue for big. Scale not auto scale. It needs to be called. Doesn’t happen automatically
- Two types of scaling
- Vertical Scaling
- Increased demand is met by increasing the power of the same machine
- Single machine computing wow
- Horizontal Scaling
- Increased demand is met by increasing the number of machines working together
- All Distributed computing products are horizontal Scaling
- Dataproc
- SQL for write is not horizontal Scale because it’s not distributed computing
- SQL for write can only do vertical scale
- Spanner is distributed computing, hence it’s horizontal scale
- This is a sustainable way because there is a upper limit to vertical scaling and it’s very expensive to build more powerful machines
Machine Learning Recommendation via DataProc Code

- https://github.com/GoogleCloudPlatform/training-data-analyst/blob/1d9e223185c26d85357ad77f0053616e4b23e52b/CPB100/lab3b/sparkml/train_and_apply.py#L26-L32
- https://github.com/GoogleCloudPlatform/training-data-analyst/blob/1d9e223185c26d85357ad77f0053616e4b23e52b/CPB100/lab3b/sparkml/train_and_apply.py#L50
- Other algorithms using Apache ML Lib: https://spark.apache.org/docs/latest/ml-classification-regression.html

Storage: Apache HDFS/Amazon S3/Google Cloud Storage
Processing: Hadoop/MapReduce/Spark/YARN/HiveQL/Pig Latin
Recommendation is Part of AI
Any AI needs 3 things
1. Data
- Can be saved in databases sql or no sql
- Can be saved in data lake or data warehouse
- Data needs to be processed to bring it in format needed for ML model
2. Model
- Many programming languages can be used to write ML model. Tensorflow, Pytorch, MLLib, Scikit learn, xgboost, BigqueryML, R, …
3. Infrastructure to run Model

Products
1. DataProc: Managed Hadoop. Nothing but Hadoop on GCP. (Called EMR/Elastic Mapreduce in aws and HDinsight in Azure)
- Cluster: Group of machines who work together parallely. It’s divide the work and do it parallel. That is the basis of big Data
- Migrate on prem Hadoop to On cloud Dataproc
- https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-jobs
- https://cloud.google.com/solutions/migration/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc
- https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-overview
2. Cloud SQL: Managed RDBMS (MySql, SQL server and PostgreSQL)
- OLTP
- Migrate on prem sql to Cloud SQL
- Migrate oracle to cloud sql https://cloud.google.com/solutions/migrating-data-from-oracle-to-cloud-sql-for-mysql or https://cloud.google.com/solutions/migrating-mysql-to-cloudsql-concept
- Migrate others

- DataBase: R&W also called as OLTP. Lots R & W
- Database & Table in it..
- SQL
- Structured Data
- Not a Big Data, Not a distributed System
- NoSQL
- Structured or Unstructed Data
- Big Data
- Is a distributed System
- Wasn't easy to query & wasn't transaction
- NewSQL
- Structured but Big Data
- Combines Distributed computing computing of NOSQL with easy querying of SQL
- DataWarehouse: OLAP. 80% times you are doing read queries on a huge Data
- Dataset & Table in it.
- Dataset & Table in it.
- Querying should be easy & should query large volume of Data
DataLake

BigQueery
- https://www.evernote.com/shard/s295/sh/187523fb-c79a-49dc-8a17-275607437d60/177187f129c9a7d39667c10c2b635610 & https://www.evernote.com/shard/s295/sh/9a1eb7e2-0e14-40ac-ae8b-40e3e07d470a/8e560d7db7f60c28d20d2e72ab7c69b1

AI Platform
- AI Models
- APIs
- Same APIs but more languages supported with better accuracy
- Sometimes Lacks some features of native processing in api, need to do processing ourself
- Transcription with timestamp in video transcribe
- Better features in some cases
- Transcription accuracy in noisy environment too
- eg
- NLP supports 17 vs 6 in aws vs 17 in azure
- Translation supports 109 vs 55 in aws vs 74 in azure
- Speech to text supports 120 vs 20 in aws vs 25 in azure
- Auto ML
- Tabular, NLP, Video, Image + Translation (5 compared to 3 in aws and azure)
- Tabular
- Bigquery Table, & I want to predict x column.
- Custom Model
- Tensorflow
- Other Open Source
- Pytorch, Keras, XGBoost, Scikit learn
- BigQuery ML
- Using SQL query to create & train ML Model
- AWS has this feature 2 years after GCP has
- Experimentation Tracking
- Not natively in AI Platform, implemented in Kubeflow
- Model Interpretability
- Natively using Explainable AI (but open source)
- Marketplace solutions h2o.ai (Available for aws too.)
- Bias Analysis
- Using What if tool (But open source)
- Custom Hardware
- TPU. (ASIC Hardware, latest generation) vs FPGA in others (Older version)
- Generations of Hardware
- CPU
- GPU
- FPGA
- ASIC
- Reinforcement Learning
- Partial Support on GCP vs better support on aws
- AI Marketplace
- All 3
- ML IDE
- Via Jupyter
- Same as aws. Azure has Azure ML Studio which is an UI tool

---
## Questions
- Ask the questions here
How ETL and Data Engineering differ?
- is serverless same as a serverless function?
- Serverless as a topic is about how it's implemented. When server's are managed automatically, anything becomes serverless. So functions whose infrastructure is managed automatically becomes serverless function.
- china has better infra hardware than google, so what level is it? is it better than ASIC?
- ASIC is the final evolution possible. They have better ASIC hardwares.
- So which hardware systems are used as storage in Google Cloud, SSD or something else?
- It is implemented in background. It could be anything
- Does Google CLoud storage has nfs provision. As far as I know Azure Blob doesn't have nfs implicitly but use smb?
- Google Cloud Storage is hadoop file system. It's not nfs
- How does skill badges compare to certifications? Can they be showcased like certifications or merely a progress point towards certfication?
- They can be showcased. But Certification is better than badge and badge is better than No badge.
- How to troubleshoot the various firewall network issues when connecting to the cloud. Each PaaS needs a separate VM and separate network configuration?
- There is a networking specialist course which probably goes into details of this
- What is the reason behind understanding the storage space, is it to gauge as Data Engineer what would be the space required when a new pipeline is going to be built or is it to give a general idea of cloud store or both
- It's to get general idea of the portofolio of google cloud products
- Please let me know what are GCP credits.Is it required for training or required even for normal production projects
what does mean tensor flow and its benefit

what is the procedure and process to get GCP credit

How can we start learning things ie what path one should follow if he/she want to work as a devops?