# Lecture 10: Model deployment, model serving and monitoring

Part of mini-course of [Apache Submarine: Design and Implementation of a Machine Learning Platform](https://hackmd.io/@submarine/B17x8LhAH). Day 3, Lecture 10

* 1 hr
* Model server
* [The Rise of the Model Servers](https://medium.com/@vikati/the-rise-of-the-model-servers-9395522b6c58)
* [Serving of ML models in Kubeflow
](https://www.kubeflow.org/docs/components/serving/)
* TensorFlow Serving: Flexible, High-Performance ML Serving
* Paper: [TensorFlow-Serving: Flexible, High-Performance ML Serving](http://learningsys.org/nips17/assets/papers/paper_1.pdf)
* [TensorFlow Serving
](https://www.kubeflow.org/docs/components/serving/tfserving_new/)
* Online training, where model being served is updated in-place is not encouraged at Google.
* <TODO>
* Clipper: A Low-Latency Online Prediction Serving System
* [Clipper: A Low-Latency Online Prediction Serving System](https://www.usenix.org/system/files/conference/nsdi17/nsdi17-crankshaw.pdf)
* Open source model servers:
* [Seldon Serving](https://www.kubeflow.org/docs/components/serving/seldon/) / [SeldonIO/seldon-core](https://github.com/SeldonIO/seldon-core)
* [KFServing](https://www.kubeflow.org/docs/components/serving/kfserving/)
* References:
* [Lecture 12: Model Serving CSE599W: Spring 2018](http://dlsys.cs.washington.edu/pdf/lecture12.pdf)

## Monitoring

* purpose: evaluate model performance, compare prediction versus actual.

Workflow manager: Azkban, Oozie, Airflow, Luigi

* [Open Source Data Pipeline – Luigi vs Azkaban vs Oozie vs Airflow
](https://www.bizety.com/2017/06/05/open-source-data-pipeline-luigi-vs-azkaban-vs-oozie-vs-airflow/)
* Oozie
* the standard workflow manager for Hadoop. Old but not obsolete. Workflow is specified by Java or XML configuration file.
* Airflow
* Most popular WM these days. Runs on Hadoop and Kubernetes.
* Many more: [LF AI Foundation Interactive Landscape](https://landscape.lfai.foundation/category=workflow&format=card-mode&grouping=category)
* Data preprocessing, ML job, model serving

## Benchmarks

* [Powering AI: The explosion of new AI hardware accelerators](https://bereadycontenthub.com/beready/psg/art/powering-ai-the-explosion-of-new-ai-hardware-accelerators/)
* [MLPerf](https://mlperf.org/)
* EEMBC Adasmark benchmarking framework
* DAWNBench
* [DAWNBench: An End-to-End Deep Learning Benchmark and Competition](https://dawn.cs.stanford.edu/benchmark/papers/nips17-dawnbench.pdf)
* [Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark](https://cs.stanford.edu/~matei/papers/2019/sigops_osr_dawnbench_analysis.pdf)

###### tags: `2019-minicourse-submarine` `Machine Learning`