# Telos
## Vision
Ability to create and implement domain specific reusable machine learning blue prints.

reduce time taken for model to production incrementally.

Example - current use case implementation of JioMart can be applied to Ajio, Tiara, Hamleys, Reliance Digital OR other e-commerce platform like shopify. We only need to write connectors.


## Benefits

1. Reduce number of data engineers needed in a project
2. Decreased time to market of machine learning model
3. Reusability of features and model - again decrease ML efforts

## Data & Statistics
### Telos
* 200K line of codes (260K including empty lines)
* 2500 files (Java, JS, config and python)
* 50+ microservices
* 23 contributors
* 7646 commits (git log --oneline | wc -l)
* First commit - April 2022

Source - https://devops.jio.com/AICOE/TelOS/_git/telos

### Comparison - MLFlow
* 443K line of codes
* 5485 commits
* 25 active contributors in last 6 months
* First commit - June 2018

Source - https://openhub.net/p/mlflow

## What problem telos is solving?

ML Project life cycle need multiple tools to execute.

https://neptune.ai/blog/mlops-tools-platforms-landscape

No single platform solve all problems. Data scientists and data engineers work in silos. It result in effort duplication and take longer time to productionize model.

## What is solution?
Single platform to automate ML tasks and solve most common problems related to ML project execution.

Save efforts (in person month) of data engineers, MlOps engineer and Data scientist.

### Platform Capabilities
1. **Feature Store -** Derived features (Automated), time series features, profile features
2. **Knowledge Graph -** Configurable schema based graph database to store entities, attributes (and its qualifier), predicates (and its qualifier), multiple names and multiole Identifiers
3. **Dynamic Data Pipelines -** Ability to create configuration drive data pipeline supporting map-reduce and other important functions
5. **Training and model deployment -** automated model training and deployment pipeline
6. **Real time inferences -** Trigger based realtime inference for delta changes on feature or other dependent inference
7. **Batch inference -** scheduler based inference
8. **Repositories -** Model store, Inference store, Label store, Metric Store, Decision stores
9. **Other Pipelines -** Feature to inference to Decision to Output
10. **Non ML Model Support -** Support for Rule based and heuristic models with expression engine
11. **UI workbench -** Everything configurable and domain agnostic
12. **Input Connectors -** kafka, csv, api
12. **Output Connectors -** kafka, csv, api

Stage URL : http://10.167.165.90:30974/

### Who are the users of platform
Domain expert, data scientists and data engineers

### Dollar Benefit
Each additional model onboarding will only have incremental cost associated using telos.
#### Jio mart personalisation use cases
1. Buy Again - Quick access to frequently purchased products.
2. Suggested Stores - Quick access to categories of interest and products within those categories based on past shopping journeys/interactions
3. Journey Continuity Widget - Need seamless shopping experience to effortlessly pick up from where users left off
4. Suggested Products - Easily discover products within categories users are actually interested in, based on past shopping journeys/interactions.
5. Similar Products - Discover similar products to the hero product
6. Bought Together Products - Discover complementary products to the hero product
7. Items you may have missed - Discover missed products
8. Recommendations based on cart items - Discover more items during cart checkout
9. Products/Categories related to your purchase - Discover what to purchase next
10. Product Notifications - Get alerts on personalized product offerings (outside of platform)

#### Timelines.
1. First model/use case will need 8-10 weeks of efforts to onboard. It include creation of connectors, system integration, production/sit enviornment creation, data onboarding, infosec approval.
2. Next model will take about half of its time for onboarding.
3. After 4th-5th use case puting model to production should only take 2-3 weeks.
4. Telos will make ML experimets fast for data scientist because availability of pre defined features and knowledge.
5. Improvement in design of previous deployed model should take less than week to put in production
6. Expecting to have all personalization use case of Jio Mart by end of June, with at least 5 models with better version in production

## Shailesh Vision
Extendable operating systems (e.g. retail-os, agriculture-os, healthcare-os, factory-os) based on telos with domain specific predefine model recepies, knowledge, feature and data pipelines.

Plug and play system that only needs knowledge onboarding and connector integration to bootstrap complete system.

## Prior to April 2022
Created two versions of platform. They never made into production.
### Telos
1. Jio Fabric - Until Feb 2021
2. Brain OS - Until Nove 2021
3. Telos - April 2022

### Cortex
1. AIX
2. Cortex
3. Scrapped

## WOW
https://devops.jio.com/AICOE/telos/_workitems/recentlyupdated/
### Internal releases
1. Shailesh defined scope in meeting
2. We create tasks in azure devops - Product Backlog
3. Give demo to shailesh every three months

| Functionality | Sage Maker| Azure ML| H2O 3 |Telos| Telos Futue| Notes|
|----- |-----|-----|-----|-----|-----|------|
|Domain standardization| NA |NA |NA |Yes | | Example - once features, models are defined for JioMart same can be applied to Ajio, Reliance Digital, Tiara, Hamleys or Shopify. We only need to add connectors.|
|Feature Store |Yes |Yes |Yes |Yes* | | Support for time context, spatial context and binned context while defining keys. Example week day vs weekend model|
|Decision Store |No |No |No |Yes | |
|Inference Store |No |No |No| Yes | | |
|Real time data pipeline for feature generation| No| No |No |Yes | | Based on Kafka|
|Batch pipeline for big data support| No |No |No| Partial| Yes |Based on Spark - Dropin replacement for Airflow, Ray|
|Model Development (ML Experiment Capability)| Yes |Yes |Yes| No |No| Telos keep experiment out of scope (Auto ML)|
|Data Labeling | Yes |Yes| Yes| Partial | Yes*| Partial - Decisions will be converted to Lables|
|Model management |Yes| Yes| Yes| Yes | |
|Model deployment |Yes |Yes| Yes |Yes*| Yes| Intgration with open source - MLFlow|
|Model training| Yes |Yes |Yes| Yes| Yes |Intgration with open source - MLFlow (H2O integration pendingn to support distributed and parallel training)|
|Batch inferencing |Yes |Yes |Yes |Partial| Yes |
|Real time inferencing| Yes |Yes |Yes| Yes |Yes | |
|Model Observability |Yes| Yes |Yes |Partial| | Via metrics|
|Knowledge Graph| NA |NA |NA |Yes| | Support for advance search|
|Heuristic Function Support| NA |NA| NA |Yes| | Support complex expression|
|Rule engine support| NA |NA| NA| Partial Yes| | Integration with Drools rule engine in progress|
|Optimization Engine Support| NA |NA| NA| No| Yes| Constraint resolver - Integration with Optaplanner pending|
|Casacade inference - Automerging of DAG| No |No| No| Yes | | Model inference of one use case result inference pipeline of other model.|

* Supported only in Telos
* Supported by most ML platform
* Not supported by telos
* Implemetation in progress

## Links
* Admin UI - http://10.167.165.5:30974/auth/login
* User Stories - https://devops.jio.com/AICOE/telos/_workitems/recentlyupdated/
* Source Code Repo - https://devops.jio.com/AICOE/telos
* Observability - http://10.167.165.80:3301/

## What gap does it fill for technology?

### Workbench and Schema
Telos enable use of schema to define all its ML construct. Once defined they can be reused across domain.

Use Case : Once schema of **dynamic price** models is defined along with its feature and inference, it can be reuses. Reusability can happen across JioMart, Ajio.

### Knowledge Graph
Rich support for Taxonmy and Ontology to support any kind of knowledge graph at scale. Support for multiple names and business ids.

Use Case : Support cross sell, upsell, recommendation and semantic search use cases.

### Dynamic data pipelines
Ability to create dynamic pipeline using UI to generate complex features using map reduce and beyond.

Use Case : For demand forecasting of certain SKU during weekend in north india, data scientist need to crunch sales data. Telos support this using workbench.

### Feature Store
Extensible support for simple, derived and profile features. Support to missing value using back fill, forward fill and mean values.

Use Case : Given weight and height as feature, telos can calculate BMI using UI configuration.

### Non Machine Learning Models
Telos support rule engine and optimzation engine via integration with Drool and Optaplanner

Use Case : Heuristic, Route optimzation and bin packing

### Model trigger infrastructure
Telos USP is to trigger model inference as result of execution of other model inference, rule or optimization thus creating dynamic workflows.

Use Case : Increase in churn score should result in dynamic coupon code.

### Comprehensible Repositories Set
Telos support inference store, decision store, metric store, model store and lable store.

Use case : Long term memory for inference, decision and metrics helps in root cause analysis.

### Dynamic workflows
Automated work flow to support inference to decision conversion, feature/decision to label mapping and calculate moving averages of features.

Use Case : This week aggregated sales will be label for demand forecasting of previous week model inference.

### Automated training and deployment of models
Automated training infrastructure triggered via scheduler. Support for latest data for model building via fetch service.

### Connector Architecture
Pre built suit of connector to ingress and engress data into and out of telos.

Use Case : Kafka, CSV, Excel, Redis and API connector