Done: Big Data & Machine Learning Fundamentals - 12 April 2021

# Done: Big Data & Machine Learning Fundamentals - 12 April 2021 > This Note: https://is.gd/owemah or https://hackmd.io/@ajinkyakolhe112/ryHxerZLO > Slides: https://1drv.ms/b/s!Aq6hYeVV5o6DhstxiE85WHGWV5bISQ?e=yHpkKp Learn better by doing these 3 Things 1. Relate what’s being taught to what you know already - Make it engaging by relating it movies 2. Make detailed Notes to understand deeply 3. Revise via Notes to increase the retention - After revising for 5 to 10 times, the information is permanent in the long term memory --- ## Notes General GCP Resources - Check the learning paths: https://cloud.google.com/training#learning-paths - Details of Certification: https://www.evernote.com/shard/s295/sh/ab8acf7b-98b0-46b3-afbd-3756b46a825e/ffb53c4f70d0fe7fb85f56a9a80bad2f - No recording allowed, but Revist today’s course online (4 different ways): 1. https://www.coursera.org/learn/gcp-big-data-ml-fundamentals (Audit the course then you can do it for free. https://www.classcentral.com/report/coursera-signup-for-free/) or on pluralsight 2. https://cloudonair.withgoogle.com/events/apac-gcp-fundamentals-series 3. https://cloudonair.withgoogle.com/events/cloud-onboard-data-fundamentals 4. https://www.youtube.com/playlist?list=PLY7sQ59Bufns3VafkhnHpbdbGBrTxSXwi - https://github.com/gregsramblings/google-cloud-4-words - https://www.gcpweekly.com/gcp-resources/ - Free Google Cloud Access: https://go.qwiklabs.com/qwiklabs-free Key Summary of Important Terms - What is Cloud Computing - Cloud computing is abstraction on infrastructure for compute and storage. - Cloud is a place like amazon, where you shop for required compute and storage - You rent the hardware for compute like CPUs and hardware for storage like HDD, SSD or Databases. - Pay per use. When you are done with using something, you simply stop it. And then you no longer pay for it. - If I need 5000 machines for 2 minutes, then I start them, use them, and after using, delete them. - Cloud Service provider maintains all the hardwares, you simply rent infrastructure as and when needed. - Cloud Offers easy way to scale, is more secure and is cost effective (cost effective if done right and if maintained and monitored properly) - Using cloud is simple, but not necessarily easy - Can focus just on developing software and let the cloud provider take care of the infrastructure… OnPrem vs Cloud vs Serverless Cloud - OnPrem - User configured, user managed and user maintained - Cloud - User configured, provider managed and provider maintained - Different Ways of Using Cloud - (on prem is user bought, user configured & user maintained) - Infrastructure as a Service: User configured, user maintained & Provider provided - Managed Product (Partially Managed by us and Partially by the Provider): User configured, provider managed & maintained but partial work still is needed by user - Fully Managed / Serverless: Everything is done by the provider. User just codes - Example restaurant - IaaS: You cook in the restaurant. - PaaS: Buffet self service - Serverless: Waiter serves you prepared food - https://www.episerver.com/articles/pizza-as-a-service & https://www.bmc.com/blogs/saas-vs-paas-vs-iaas-whats-the-difference-and-how-to-choose/ - Serverless Cloud - Fully automated and no configuration required. Different roles in Data Team Hypothetical Scenario - 1x or 2x Infra (Physical, IaaS or Cloud) - 1x DevOps (Stack automation, Containers and Platform as a service) - 2x or 3x Data Engineer (Data Pipelines, Data Automation, Data as a Service, Data Ingestion) - 2x Analytics (1x Batch Analytics, 1x Real-Time Analytics and Predictive APIs) - 1x AI and ML Data Scienst (Machine Learning and AI algorithms) - 1x Front-End Dev (Web and Js developer, web and mobile apps) - Specialized Roles - 1x Network Architect - 1x Security Engineer - 1x Data Viz Developer Types of Jobs on Cloud 1. Migration. We move the original codebase from on prem to on cloud - Google Cloud has partner ecosystem and parterns are heavily involved in migration. - Migration Jobs 1. Step 1: Move application as is (Lift & Shift). We are simply running the same application but at different place 1. Most demand for Cloud Enabled people 2. Step 2: Improve the application to take advantage of Cloud Products. 3. Step 3: (Cloud Native / Serverless) Rewrite everything for cloud Native. No maintainence required 4. Can find many guide here: https://cloud.google.com/docs/tutorials/ 2. Cloud Native Development or New application: Generally startups do this - Cloud Architect is Design & Develop the system - Design consideration for Scale for security for future growth all these are part of architect’s job - Cloud Developer just Develop the system - Architect vs Civil Engineer - Civil Engineer Builds the building according to the blueprint. - Architect designs the blueprint considering the feasibility of the building. - Movies Reference: Inception Dream Designer: Architect 3. Feature Addition on prexisting cloud hosted product 4. Maintainance of the cloud hosted Product Evolution to Cloud 1. First On Prem either server or data center - My Infra, I buy it, I manage it 2. Then On Cloud. (First migration called lift & shift) - Someone else’s infra - Renting not Buying. But renting gives flexibility - House Rent vs Buy? - Renting gives us abilrity to change 3. Then Cloud Native / Serverless / Fully Managed - Someone else’s Infra, but completely automatically managed - Iron Man? Mark I vs Mark 85 in End Game. nano tech. Migration to Cloud 1. Lift & Shift. Move as is 2. Optimize for Cloud. Tiny rewrite to improve the performance 3. Complete Rewrite for Cloud Native or Containerize for Hybrid cloud via Container ### Module 1 Big Data - data analysed is just 1% of all data produced, thus huge potential for growth in this field - Cloud helps analyze such a large scale data easier compared to on prem systems Data Scientist vs Data Engineer - Overlapping skillset but different specialisation - Data Science is overcrowded,if you are really good then go for it. Otherwise choose Data Engineering if interested. - Demand Ratio is 3:1 , where as supply ratio is 1:3. - Concentrate and focus less crowded areas otherwise be the best in crowd. Google Cloud different products - Compute - Custom Hardware TPU, best choice for ML workloads - ASIC -> Used by Red Hat OpenShift Container Platform(OCP) & IBM CP4A as well is nothing but a TPU. - Storage - Networking - Networking in Google Cloud is powerful - Private fiber optic cables for better networking with high bandwidth in Google Cloud - Security. - Security is a shared responsibilityl - IBM Security Services product a.k.a Managed Security Services (M.S.S) is a Global Market Leader in Web and Cyber Security space worldwide - Big Data & ML Product Summary: - Explosion or Growth of Data, - Cloud playing an important role in Big Data, - Data Scientist v/s Data Engineers (Demand and Supply), - Big Data & Analytics + Cloud Migration/Modernisation - With advent of technologies like Kubernetes, Docker containers and Red Hat OpenShift powered with IBM CP4A — High Availability is achievable. ### Module 2: - Scale - Autoscale - Change according to demand automatically either increases or decreases without user intervention - Eg: room of requirements from Harry Potter - Scale - You have to execute the command which can resize the resources either up or down. This is not automatic - eg: Ant Man. Red is shrinking and blue for big. Scale not auto scale. It needs to be called. Doesn’t happen automatically - Two types of scaling - Vertical Scaling - Increased demand is met by increasing the power of the same machine - Single machine computing - Horizontal Scaling - Increased demand is met by increasing the number of machines working together - All Distributed computing products are horizontal Scaling - Dataproc - SQL for write is not horizontal Scale because it’s not distributed computing - SQL for write can only do vertical scale - Spanner is distributed computing, hence it’s horizontal scale - This is a sustainable way because there is a upper limit to vertical scaling and it’s very expensive to build more powerful machines Machine Learning Recommendation via DataProc Code - https://github.com/GoogleCloudPlatform/training-data-analyst/blob/1d9e223185c26d85357ad77f0053616e4b23e52b/CPB100/lab3b/sparkml/train_and_apply.py#L26-L32 - https://github.com/GoogleCloudPlatform/training-data-analyst/blob/1d9e223185c26d85357ad77f0053616e4b23e52b/CPB100/lab3b/sparkml/train_and_apply.py#L50 - Other algorithms using Apache ML Lib: https://spark.apache.org/docs/latest/ml-classification-regression.html Storage: Apache HDFS/Amazon S3/Google Cloud Storage Processing: Hadoop/MapReduce/Spark/YARN/HiveQL/Pig Latin Recommendation is Part of AI Any AI needs 3 things 1. Data - Can be saved in databases sql or no sql - Can be saved in data lake or data warehouse - Data needs to be processed to bring it in format needed for ML model 2. Model - Many programming languages can be used to write ML model. Tensorflow, Pytorch, MLLib, Scikit learn, xgboost, BigqueryML, R, … 3. Infrastructure to run Model Products 1. DataProc: Managed Hadoop. Nothing but Hadoop on GCP. (Called EMR/Elastic Mapreduce in aws and HDinsight in Azure) - Cluster: Group of machines who work together parallely. It’s divide the work and do it parallel. That is the basis of big Data - Migrate on prem Hadoop to On cloud Dataproc - https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-jobs - https://cloud.google.com/solutions/migration/hadoop/migrating-apache-spark-jobs-to-cloud-dataproc - https://cloud.google.com/solutions/migration/hadoop/hadoop-gcp-migration-overview 2. Cloud SQL: Managed RDBMS (MySql, SQL server and PostgreSQL) - OLTP - Migrate on prem sql to Cloud SQL - Migrate oracle to cloud sql https://cloud.google.com/solutions/migrating-data-from-oracle-to-cloud-sql-for-mysql or https://cloud.google.com/solutions/migrating-mysql-to-cloudsql-concept - Migrate others #### BigQuery - DataBase: R&W also called as OLTP. Lots R & W - SQL - Structured Data - Not a Big Data, Not a distributed System - NoSQL - Structured or Unstructed Data - Big Data - NewSQL - Structured but Big Data - Combines Distributed computing computing of NOSQL with easy querying of SQL - DataWarehouse: OLAP. 80% times you are doing read queries on a huge Data - Querying should be easy & should query large volume of Data - Data Lake: No Database no data analytics Bigquery - https://cloud.google.com/blog/products/bigquery/anatomy-of-a-bigquery-query - SQL query select language, sum(views) views from bigquery-samples.wikipedia_benchmark.Wiki100B where REGEXP_CONTAINS(title,".*o.*o.o.") group by language order by views desc - https://cloud.google.com/blog/products/data-analytics/new-blog-series-bigquery-explained-overview - BigQuery ML - https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create - BigQuery - https://www.qwiklabs.com/quests/147?parent=catalog - https://www.qwiklabs.com/focuses/3460?parent=catalog - BigQuery ML - https://www.qwiklabs.com/focuses/1797?parent=catalog - https://www.qwiklabs.com/focuses/16547?parent=catalog - https://www.qwiklabs.com/focuses/14294?parent=catalog - Other competitions where BigQuery ML can be used 1. https://www.kaggle.com/c/zillow-prize-1 (1.2 million dollars prize) - Predicting price of the house 2. https://www.kaggle.com/c/two-sigma-financial-modeling 1 million dollars prize - Predicting direction of the share movement. Up or Down 3. https://www.kaggle.com/c/deloitte-western-australia-rental-prices 1 million dollars prize - Predicting Rental Price 4. https://www.kaggle.com/c/home-credit-default-risk 70k dollars prize - Predicting Default or Not 5. https://www.kaggle.com/c/santander-customer-transaction-prediction 65k dollars prize - Predicting Whether will buy or not AI(Artificial Intelligence): learning from Data (Superset) DL(Deep Learning): learning from Semi-structured or Un-structured Data like No/SQL. Data is generally big data, but it need not be. - All about data from human sensors (machine sensors might have tabular data so ML) ML(Machine Learning): learning from Tabular data or Structured data or RDBMS. Data could be big data or not. DS(Data Science): Not just limited to data modelling, but also data storage, data processing etc. Cloud computing too - collect Data, organize data in DW. Not only creating Models in compute engine - Data warehouse is a central place to store/save all types of data which might be needed in future BigQuery Data Warehouse: Best suited for big data analytics queries. Very easy product to use and it’ serverless. All components of AI System 1. AI or Artificial Intelligence - AI is superset - ML is subset of AI - DL is subset of ML - Data Science is ML or DL on Cloud using all related tools for the job 2. Data Science 3. ML or Machine Learning - Tabular Data 4. DL or Deep Learning - Unstructured Data. Images, Video, Audio, Text etc Every AI is made up of 5. Data 6. Model 7. Infra to run the Model Types of AI Problem 8. Recommendation AI 9. Value Prediction AI = Regression 10. Class Prediciton AI = Classification 11. Analmoly Detection AI https://cloud.google.com/bigquery-ml/docs/reference/standard-sql/bigqueryml-syntax-create https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml --- ## Questions - [Hi Ajinkya...please let me know potential career path for Data Analytics Manager? ] - I'm very new to cloud platforms, which certification should i opt for to deeply understand how all this works? - [ How to get a job at google gcp?] test question - Could you please explain how analytics career is diferent from data science or is it similar? - underwater cables/data centers seems interesting. How Google is maintaining/monitoring this? - Is there any good google cloud data engieneering course which you can suggest. - [Can we write our own ML alogrithm without using the default ones?] - [So will that not make the results constant?] - What are the steps Google taking to become Market leader in cloud?