PyCon TW 2017 Collaborative Talk Notes
Day 1 - R1
Quick Links
How to update this note?
- Everyone can freely update this note. 任何人都能自由地更新內容。
- Please respect all the participants and follow our code of conduct during discussion. 討論、記錄時,請遵守大會的行為準則
- Slider:
- Speaker: Renaldi Gondosubroto
作者所用的語言
VOEZ Current Status
- Profile, save data, achievement, leaderboard
- Based on GCP
- Teaser
HTTP protocol
Request
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
Response
Image Not Showing
Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
Genuine & Purchase verification
leaderboard -> fake score
- verify request: send request & attach signature
- RSA
game play -> pirated app & server
- server授權:RSA request hash 增加 nonce (random number)
Stability
publish order and request order (Racer condition)
- request order
- client read game info A, then B, then C
- publish order
- publish game info C, then B, then A
- remove game info A, then B, then C
database cache mechanism
- database in HDD and SSD: slow
- Redis and mem in memory: fast, as cache
reliable inter-server request
- 由於server之間的溝通導致玩家扣錢失敗
- A: Game server B: 金流伺服器
- 扣款失敗:rollback
- 嘗試三次,失敗後要求client重新連線
- 可以讓玩家排隊沒關係!資料庫不能掛!
- Python is so SLOW
- "skip Python code execution if you can" - HT Kuo 06, 09, 2017
- 支援百萬人上線需要的反應時間: nanosec
- 用Google BigQuery取得公開資料,並把結果存成靜態檔案放流,迴避Python執行
- 使用CDN
- 全球的玩家連線到各地的CDN
- 節省流量及運算資源
- asset files
- 可以準確在換日的時候讓 cache expired
- service downtime: 503
- service update: 舊的版本丟410
Timezone
- 為了準確地在本地換日時間(ex:3 a.m.)更新活動
- TZ = "Asia/Taipei"
- 用UTC去計算會碰到日光節約時間的問題
Q: Why not use AWS?
AWS跟中華電信間連線超慢
Q: 應對百萬使用者的方式?
設定動態開設資源
Q: Header content type為空?
手動刪除的XD
Q: 阿里雲和GCP共通方式?
可以做data sync但中國比較偏向自己有伺服器但是Code共通
Q:歌曲放上CDN是否會被竊取?
歌曲資料有經過加密
自我介紹:從趨勢獨立成立的子公司 soocii
平台提供給手遊玩家的價值
費時<1年
backend工程師*5
推薦大師:Martin Fowler、Sam Newmer
使用 microservice 的好處:
曾經面對的挑戰
系統配置
混合雲:AWS + GCP (因為AWS在tokyo,速度不夠快)
使用技術:Python + C
micro service 問題:開發環境的準備
-
外部相依性 & 內部相依性
- 全部在本地
- 只有microservice跟business logic在本地
- 只有microservice在本地
- 都在雲端開發
-
soocii 採用混合方案
- 本地環境:docker-compose, mock data, 假認證
- 整合環境:docker-compose
參考:Development environments for microservices (https://www.datawire.io/guide/deployment/development-environments-microservices/)
-
跨服務的溝通
- 程式溝通方式
- micro service 要能時刻提供服務,不會影響主要功能
- 理想上:不應該存在部分元件異常導致全部損毀(如:Redis 異常,應該要可以再存取 RDB)
- 實際上:很難,像是 Auth 是全部共用模組,難以取代,不過另外一個想法是也不容易壞掉
-
自動化測試(CI)
- 理想:修補後可以馬上上線
- 測試種類:(測越多信心越高)
- unit test: functions logics(最快)
- service test: classes + mock class
- requests.mock, mock.patch
- UI test: 整合性測試(最慢)
- consumer-driven contract (CDC)
- 非同步測試:
- celery: task_always_eager
- event-driven (good to have)
- sleep (bad)
-
自動化部署(CD)
- 盡可能延後
- 多版本API同時並存(透過blueprint找到對應版本)
導入 microservice 的關鍵
- 所有成員要有DevOps mindset
- 至少要有CI(自動)/CD(到處都可部署)的雛形
- 每個service有獨立的CI
- 監控機制
Q. Instance實際溝通的行為,開發中如何處理?
AWS: (application load balancer)
每個docker註冊在ALB後面 => 確保服務活著,但不清楚是跟哪個docker溝通
Q. 跨伺服器溝通如果逼不得已要交換資料,怎麼處理?
只可能盡量避免。
Q. 為什麼要做混合雲?發生什麼事情?
直播的frame rate掉太多,內部做profiling本地端最快,最後才採用本地端。
-
Slide: https://speakerdeck.com/dawny33/understanding-serverless-architectures
-
Speaker: Jalem Raj Rohit
-
what is serverless?
function as a service, no servers
AWS: Lambda > functions > …
-
function container
- function executed in container
- We are not running or maintaining servers
-
advantages
- less time maintaining server
- lots of server cost saved
- e.g. deep learning
-
dis(ad-)vantages
- allow to run in limit time (AWS: timout setting)
- no control over containers
- monitoring is very very very difficult
- especially, scale out as distributed
-
lessons learned and pitfalls
- limit running time problem
- Ansiable to rescue
- help provisioning workers
- running Ansible in
nohup
mode
- no control over container problem
- monitoring distributed system is difficult
-
How to scale?
-
Slider: Python module in Rust
-
Speaker: 許邱翔 (Chiu-Hsiang Hsu) (dv)
-
Rust community love Python most
-
Rust cares memory safety without GC ; Rust is high performance like C
benchmark
-
PyPI of Rust
-
pyvenv + pip -> Cargo
-
PEP of Rust -> RFC
Q: why the Rust source code need something like "Extend C"
A: that is for avoiding default behavior "name mangling" like C++
Q: Rust can share memory buffer with Python?
A: yes