# 以平台開發模式為大型 VOD 系統建置 metadata 處理系統 - Sam Tu (DevOps, Cloud)

{%hackmd @mopcon/BkMIqw_76 %}

> 從這開始

## Background Introduction
### Importance of metadata
- Search & Filter
- localization
- monetization
- etc

> Metadata 的不正確性在日本相當嚴重

### Processing of metadata
* Pick(content)
* Pack(encoding)
* Deliver(CDN)
* Engage(storefront)
* Consume(player)

> Content may be king.
> But metadata holds the keys to the kingdom

### Challenge
1. 需消化 17 種 csv（包在 zip），解 zip 時得顧及關聯性
2. 資料量大，需平行處理
3. 跨專案，不只複製 logic，也會因客戶需求分別客製化

## Platform Solution
- 建立 `Landing` 與 `DaaS`
- `Daas`
- Input: dedup, diff, ...etc
- Service: meta upload API
- `Landing` 僅處理 data relations
- Format Validate
- Dependency Validate（資料往往不照預期順序上傳）
- Lock
- Version check
- Sink

## Meta Processing System Introduction
1. Input files to `DaaS`, process videos, meta and images.
2. Share metadata across images in `DaaS`
3. Transfer "DaaS Final Data" to `Landing`
4. Pending State Design and Retry Strategy

> Extended blog (之前看到覺得很有趣 :D ) [AWS Retry strategy](https://aws.amazon.com/tw/blogs/architecture/exponential-backoff-and-jitter/)

## Operation Experience
1. Dashboard
2. Report email
3. Logging (ELK)
4. AWS Fargate Restart(gracefully shutdown)

## Conclusion
1. 暸解客戶的特性
2. 開發初期就先規劃監控及 loging，不建議全丟給維運人員處理
3. 沒聽到 orz