# Lecture 1: Machine learning project development process

Part of mini-course of [Apache Submarine: Design and Implementation of a Machine Learning Platform](https://hackmd.io/@submarine/B17x8LhAH). Day 1, [Lecture 1](https://cloudera2008-my.sharepoint.com/:p:/g/personal/weichiu_cloudera2008_onmicrosoft_com/ERz2lyduexlHtqwadR2dvMABl7XHuJ3TOGH0ewL8n9620w?e=mLHHbj)

* 1hr
* [AI PaaS is at the peak of Garner’s Hype Cycle chart](https://www.apriorit.com/dev-blog/635-ai-ai-paas)

[Hype Cycle for Emerging Technologies, 2018](https://www.gartner.com/en/newsroom/press-releases/2018-08-20-gartner-identifies-five-emerging-technology-trends-that-will-blur-the-lines-between-human-and-machine)


*Democratized AI*

*[AI technologies](https://www.gartner.com/smarterwithgartner/how-to-get-artificial-intelligence-right/) will be virtually everywhere over the next 10 years. While these technologies enable early adopters to adapt to new situations and solve problems that have not been encountered previously, these technologies will become available to the masses — democratized. Movements and trends like cloud computing, the “maker” community and open source will eventually propel AI into everyone’s hands.*

*This trend is enabled by the following technologies: AI Platform as a Service (PaaS), Artificial General Intelligence, Autonomous Driving (Levels 4 and 5), Autonomous Mobile Robots, Conversational AI Platform, Deep Neural Nets, Flying [Autonomous Vehicles](https://www.gartner.com/it-glossary/autonomous-vehicles/), Smart Robots, and [Virtual Assistants](https://www.gartner.com/smarterwithgartner/how-to-listen-to-the-voice-of-things-in-the-iot/).*

*“Technologies representing democratized AI populate three out of five sections on the Hype Cycle, and some of them, such as deep neural nets and virtual assistants, will reach mainstream adoption in the next two to five years,” said Mr. Walker. “Other emerging technologies of that category, such as smart robots or AI PaaS, are also moving rapidly through the Hype Cycle approaching the peak and will soon have crossed it.”*

[Gartner Hype Cycle for Emerging Technologies, 2019](https://www.gartner.com/smarterwithgartner/5-trends-appear-on-the-gartner-hype-cycle-for-emerging-technologies-2019/)


[What's New In Gartner's Hype Cycle For AI, 2019](https://www.forbes.com/sites/louiscolumbus/2019/09/25/whats-new-in-gartners-hype-cycle-for-ai-2019/#27f1008e547b)


Beyond hypes, we see real difficulties when applying ML techniques, many of them are unrelated to the core ML algorithms:

1. a lack of infrastructure to support ML applications to make the development easier, faster, scalable, more maintainable.
2. No well understood best practices.
3. Additionally, data plays a great role. You need high quality, high volume data to make good predictions. (Garbage in, garbage out)

An ivory tower style ML research is not going to work. In practice, there is a general consensus about the workflow of atypical ML application, that is composed of the following stages:

data ingestion, data preparation, feature extraction, model development, model training, model deployment and model serving

* Data collection
* TBD
* Data preparation
* TBD
* Feature extraction
* TBD
* Data collection, preparation, feature extraction are all called data engineering.
* Model development
* TBD
* Model training
* The process of creating machine learning algorithms using machine learning frameworks and training data.
* Model serving / machine learning inference
* The process of using trained machine learning algorithm (ML model) to make a prediction.

A real ML system is a dance between data scientists, data engineers and dataops. This course essentially focuses on the latter 2 roles.

ML applications are deployed at scale at several tech companies. I’ll use the following case studies

* Case studies: ML development process at large tech companies (1hr)
* This is an area of fast development and excitement. Here I am sharing a few publicly referenceable MLP case studies. They are all tackling similar problems with different approaches and visions.
* [How to build the Next-Gen ML/AI Platform](https://www.slideshare.net/JoshYeh/nextgen-mlai-platform)
* [Google: Hidden Technical Debt in Machine Learning Systems](https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf)
* [Case study](https://docs.google.com/document/d/1amvyU-E0OkmOg-chzenD4wONg7uuba035eaz2adynew/edit?usp=sharing)
* [Facebook FBLearner Flow](https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/)
* [Case Study](https://docs.google.com/document/d/1m7z4Ve2ZYy911jhVWxx0T0vC-4IE3ItMYrWn8zyWBVA/edit?usp=sharing)
* [LinkedIn Productive ML initiative and Tensorflow on YARN](https://www.slideshare.net/xkrogen/hadoop-meetup-jan-2019-tony-tensorflow-on-yarn-and-beyond)
* Will cover it in lecture 6.
* Twitter Cortext [Case study](https://docs.google.com/document/d/1ET7fZpUMlewmKSKNef8BDoLOpT3MoZ6YTnxRt0CYsZ8/edit?usp=sharing)
* [Airbnb Bighead](https://databricks.com/session/bighead-airbnbs-end-to-end-machine-learning-platform)
* [Case Study](https://docs.google.com/document/d/1hDnQfjEI69S3Jfmg0bK98fSRcV21GsywEfekMnVwDQY/edit?usp=sharing)
* [Netflix Machine Learning Infra](https://www.slideshare.net/FaisalZakariaSiddiqi/ml-infra-for-netflix-recommendations-ai-nextcon-talk) / [Machine Learning Infrastructure for Netflix Recommendations
](https://www.youtube.com/watch?v=oS5-qEX5LC0)
* Case study
* Netflix Metaflow
* [https://medium.com/netflix-techblog/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9](https://medium.com/netflix-techblog/open-sourcing-metaflow-a-human-centric-framework-for-data-science-fa72e04a5d9)
* [Case study](https://docs.google.com/document/d/1E7QneJspRv8ejBjsqX1sKX4sAXiv7ii8vB_iueL3kj4/edit?usp=sharing)
* [Uber Michelangelo](https://eng.uber.com/michelangelo/)
* [Case study](https://docs.google.com/document/d/1g3LAnECSlHBVZd-2PI5xKNzFp92CeM-H-jLhHOOh9lA/edit?usp=sharing)
* Flyte
* [Open Source Cloud Native Machine
Learning and Data Processing Platform](https://static.sched.com/hosted_files/kccncna19/7f/Flyte%20Kubecon.pdf?fbclid=IwAR3wOcjbn88lM7LEnNWaYb-Bt0SmeS1WA4G0q4nN1J-JrMv5tAfCAJq7nU8)
* [lyft/flyte](https://github.com/lyft/flyte)
* [Case study](https://docs.google.com/document/d/1cCcUCwx9XzgR51dYBPDqIrmjmS_apmtlnDDfannIKOs/edit?usp=sharing)
* Spotify
* [https://labs.spotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/](https://labs.spotify.com/2019/12/13/the-winding-road-to-better-machine-learning-infrastructure-through-tensorflow-extended-and-kubeflow/)
* Case study
* Bloomberg

- [Rules of Machine Learning: Best Practices for ML Engineering](http://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf)

- [Applied Machine Learning at Facebook:A Datacenter Infrastructure Perspective](https://research.fb.com/wp-content/uploads/2017/12/hpca-2018-facebook.pdf)

- [FBLearner Flow](https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/)





[How do Data Professionals Spend their Time on Data Science Projects?](https://businessoverbroadway.com/2019/02/19/how-do-data-professionals-spend-their-time-on-data-science-projects/)

[Data Science Survey - 2018](https://www.kaggle.com/sudhirnl7/data-science-survey-2018/comments)


###### tags: `2019-minicourse-submarine`, `Machine Learning'