Pomelo on k8s - HackMD

# Pomelo on k8s 這篇共筆主要是記錄將 Pomelo 分散式部署在 k8s 上的研究(踩雷)過程 [pomelo](https://github.com/NetEase/pomelo/wiki/Home-in-Chinese) 是網易開發的分散式遊戲 server 引擎，目前已經停止維護 pomelo 的設計理念類似 k8s，差別在 pomelo 是以實體機器(包括虛擬機)為對象，不像 k8s 是以容器來設計 --- ## 部署相關問題 ### SSH 連線 between Pods 要將 pomelo 跑在 k8s 上，有個最大困難要先克服: pomelo 的分散式部署需要透過 ssh 連線，由 `master node` 連到其他的 `worker node`，將 pomelo process 獨立跑在各個 node，並透過 rpc 通訊。因此要把 pomelo 放上 k8s，必須要先讓每個 pod 可以透過 ssh 連線解決方法就是在 Dockerfile 上動手腳 ```dockerfile= FROM node:6.9.5 # Install plugins RUN apt-get update RUN apt-get install -y openssh-server vim net-tools # SSH config init RUN mkdir /var/run/sshd && chmod 0755 /var/run/sshd WORKDIR /root RUN mkdir .ssh && chown root .ssh && chmod 700 .ssh RUN ssh-keygen -t rsa -N "" -f ~/.ssh/id_rsa RUN touch /root/.ssh/authorized_keys # Create pomeloapp directory RUN mkdir -p /usr/src/pomelo WORKDIR /usr/src/pomelo # Install pomeloapp dependencies COPY . /usr/src/pomelo/ RUN npm install pomelo@2.2.5 -g RUN npm install pomelo-cli -g # Copy ssh key to each node WORKDIR /usr/src/pomelo/game-server/ssh RUN cp id_rsa.pub /root/.ssh/id_rsa.pub RUN cp id_rsa /root/.ssh/id_rsa RUN cp id_rsa.pub /root/.ssh/authorized_keys RUN cp sshd_config /etc/ssh/sshd_config #Replace pomelo and pomelo-cli with custom version COPY pomelo-custom/pomelo /usr/local/lib/node_modules/pomelo COPY pomelo-cli-custom/pomelo /usr/local/lib/node_modules/pomelo-cli WORKDIR /usr/src/pomelo/game-server/ CMD ["pomelo", "start"] ``` 先在本機利用 `sshkey-gen` 產生一組公/私鑰放到 pomelo 專案底下，再將這組 key 複製到 image 中，讓 container 執行時能透過這組 key 連線 >安全起見，請在本機產生一組新的 key ，不要使用舊有的另外，因為 node js image 本身沒有提供 sshd 的功能，需要另外安裝 `openssh-server` 套件並作相關的連線設定 --- ### SSH public key certification 另外一個問題是：ssh 在首次連線時會將機器的 public key 記錄在 `$HOME/.ssh/known_hosts`，當下次訪問相同機器時，OpenSSH 會核對公鑰，如果公鑰不同，OpenSSH 會發出警告，避免受到 [DNS Hijack](https://www.puritys.me/docs-blog/article-236-DNS-HIJACKING-%E8%AA%AA%E6%98%8E%E8%88%87%E9%98%B2%E8%AD%B7.html) 之類的攻擊但是 pomelo 在執行時是預設檢查已經通過的前提，所以 `master node` 若是直接執行 `pomelo start`，會遇到 public key 驗證失敗的訊息，導致 ssh 連線失敗解決方法是，建立一個開機腳本 `start.sh` ```shell= #!/bin/bash ssh -o "StrictHostKeyChecking no" game-gate ssh -o "StrictHostKeyChecking no" game-conn ssh -o "StrictHostKeyChecking no" game-svc ssh -o "StrictHostKeyChecking no" game-logic ... /usr/sbin/sshd service ssh start tail -f /dev/null ``` `master node` 會先連到每一台 worker 紀錄連線資訊，之後 pomelo start 就不會遇到因為 public key 驗證失敗的錯誤訊息 --- ### Pomelo cli command failed 還有一個很雷的問題： pomelo cli 的部分功能，在分散式部署的情形下會失效？？這個問題真的很雷，發生的原因要歸咎於 pomelo 的 lib `lib/modules/console.js` ```javascript=390 var start = function(server) { return (function() { checkPort(server, function(status) { if(status === 'busy') { fails.push(server); latch.done(); } else { starter.run(app, server, function(err) { if(err) { fails.push(server); latch.done(); } }); process.nextTick(function() { successFlag = true; latch.done(); }); } }); })(); }; ``` 執行 pomelo cli，例如 `pomelo add ...`，會透過 ssh 連線到指定 host 的機器，將要下的指令與參數帶入，在 cli 利用 `child_process` 上執行在執行指令之前，為避免 port 被佔用，會先執行`checkPort` function ```javascript=285 var checkPort = function(server, cb) { if (!server.port && !server.clientPort) { utils.invokeCallback(cb, 'leisure'); return; } var p = server.port || server.clientPort; var host = server.host; var cmd = 'netstat -tln | grep '; if (!utils.isLocal(host)) { cmd = 'ssh ' + host + ' ' + cmd; } exec(cmd + p, function(err, stdout, stderr) { if (stdout || stderr) { utils.invokeCallback(cb, 'busy'); } else { p = server.clientPort; exec(cmd + p, function(err, stdout, stderr) { if (stdout || stderr) { utils.invokeCallback(cb, 'busy'); } else { utils.invokeCallback(cb, 'leisure'); } }); } }); }; ``` 關鍵點在293行的 `netstat`，pomelo 會透過它來檢查 port 是否被佔用問題來了，如果機器沒有安裝 `net-tools` 套件，那會發生什麼事？ 299行告訴我們，如果 `stderr` 發生，就當作 `busy` 回傳，導致 start 失敗解決方法很簡單，把 image 加上 `net-tools` 這個套件就好 --- ### Persistent Pod 除了上述問題外，還有一個特別的需求：我們希望 pomelo 可以執行在**有狀態**的 node 上，即便 pod 重啟或更新，重要的資料不會消失(例如： log sqlite.db 等) k8s 提供了一個解決方案： [StatefulSets](https://kubernetes.io/zh/docs/concepts/workloads/controllers/statefulset/) >StatefulSet 用来管理某 Pod 集合的部署和扩缩，并为这些 Pod 提供持久存储和持久标识符。 > >和 Deployment 类似， StatefulSet 管理基于相同容器规约的一组 Pod。但和 Deployment 不同的是， StatefulSet 为它们的每个 Pod 维护了一个有粘性的 ID。这些 Pod 是基于相同的规约来创建的，但是不能相互替换：无论怎么调度，==每个 Pod 都有一个永久不变的 ID==。 > >如果希望使用存储卷为工作负载提供持久存储，可以使用 StatefulSet 作为解决方案的一部分。尽管 StatefulSet 中的单个 Pod 仍可能出现故障，但持久的 Pod 标识符使得将现有卷与替换已失败 Pod 的新 Pod 相匹配变得更加容易。 > >请注意，当 Pod 或者 StatefulSet 被删除时，与 PersistentVolumeClaims 相关联的 ==PersistentVolume 并不会被删除==。要删除它必须通过手动方式来完成。 StatefulSet 的 yaml 設定，基本上和 Deployment 差不多，只有 kind 的參數由 `Deployment` 改為`StatefulSet` 透過 `StatefulSet` ，我們可以將 pod 當作是一台輕量的 vm，而且更方便管理/部署，而且資料得儲存也更可靠 --- ## 部署範例這邊先以 `master node` 為例 ```yaml= apiVersion: v1 kind: Service metadata: name: master labels: app: master spec: selector: app: master clusterIP: None ports: - protocol: TCP name: "master" port: 3005 targetPort: 3005 - protocol: TCP name: "ssh" port: 22 targetPort: 22 --- apiVersion: apps/v1 kind: StatefulSet metadata: name: master spec: serviceName: "master" selector: matchLabels: app: master template: metadata: labels: app: master spec: containers: - name: master image: pomelo/gs:0.0.1 imagePullPolicy: Never command: ["sh"] args: ["./start_master.sh"] resources: requests: cpu: 100m memory: 150Mi limits: cpu: 500m memory: 300Mi volumeMounts: - name: game-master-log mountPath: /usr/src/pomelo/logs - name: servers mountPath: /usr/src/pomelo/game-server/config/servers.json subPath: "servers.json" readOnly: true - name: master mountPath: /usr/src/pomelo/game-server/config/master.json subPath: "master.json" readOnly: true - name: certs mountPath: /usr/src/pomelo/game-server/config_bak/certsSetting.json subPath: "certsSetting.json" readOnly: true - name: wagers mountPath: /usr/src/pomelo/game-server/config/server/wagers/WagersDomainSetting.json subPath: "WagersDomainSetting.json" readOnly: true - name: mysql mountPath: /usr/src/pomelo/game-server/config/server/dba/mysqlSetting.json subPath: "mysqlSetting.json" readOnly: true - name: api mountPath: /usr/src/pomelo/game-server/config/server/api/apiSetting.json subPath: "apiSetting.json" readOnly: true - name: block mountPath: /usr/src/pomelo/game-server/config/server/api/blockSetting.json subPath: "blockSetting.json" readOnly: true volumes: - name: servers configMap: name: config items: - key: servers.json path: servers.json - name: master configMap: name: config items: - key: master.json path: master.json - name: certs configMap: name: config items: - key: certsSetting.json path: certsSetting.json - name: wagers configMap: name: config items: - key: WagersDomainSetting.json path: WagersDomainSetting.json - name: mysql configMap: name: config items: - key: mysqlSetting.json path: mysqlSetting.json - name: api configMap: name: config items: - key: apiSetting.json path: apiSetting.json - name: block configMap: name: config items: - key: blockSetting.json path: blockSetting.json restartPolicy: Always volumeClaimTemplates: - metadata: name: game-master-log spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 500Mi ``` 比較麻煩的部分在於，為了開發部署方便，我們希望將 pomelo 的設定檔利用 k8s 提供的 `ConfigMap` 處理，但是 pomelo 相關的所需的設定檔相當多且分散，設定上很費時這邊為了方便起見，採用了 k8s 的[`kustomize`](https://kubernetes.io/zh/docs/tasks/manage-kubernetes-objects/kustomization/) api 來部署首先將檔案結構如下放置： ```shell= ./ ├── StatefulSet │ ├── conn.yaml │ ├── gate.yaml │ ├── logic.yaml │ ├── master.yaml │ └── service.yaml ├── configMap │ ├── WagersDomainSetting.json │ ├── apiSetting.json │ ├── blockSetting.json │ ├── certsSetting.json │ ├── master.json │ ├── mysqlSetting.json │ └── servers.json └── kustomization.yaml ``` `kustomization.yaml` ```yaml= namePrefix: game- configMapGenerator: - name: config files: - configMap/servers.json - configMap/master.json - configMap/certsSetting.json - configMap/WagersDomainSetting.json - configMap/mysqlSetting.json - configMap/apiSetting.json - configMap/blockSetting.json generatorOptions: disableNameSuffixHash: true resources: - StatefulSet/gate.yaml - StatefulSet/conn.yaml - StatefulSet/logic.yaml - StatefulSet/service.yaml - StatefulSet/master.yaml ``` `configMapGenerator` 可以根據指定的檔案建立設定檔，結果如下： ```yaml= kind: ConfigMap apiVersion: v1 metadata: name: config data: WagersDomainSetting.json: | { "k8s": "https://d-bga.inslotx.com/BetRecord/client/detail.php" } apiSetting.json: | { "k8s": { "mobileService": { "host": "d-bapi.inslotx.com/", "path": "elibomApi/BBBattleAPI/" }, "cashService": { "host": "d-bapi.inslotx.com/", "path": "elibomApi/BBBattleAPI/" } } } blockSetting.json: | { "k8s": { "host": "block", "port": [30001, 30002] } } certsSetting.json: | { "k8s": { "certs": [] } } master.json: | { "k8s": { "id": "master-server-1", "host": "game-master", "port": 3005 } } mysqlSetting.json: | { "k8s": { "inDBService": { "host": "192.168.50.129", "account": "root", "password": "root", "database": "InBattle", "frontName": "" }, "wagersDBService": { "host": "192.168.50.129", "account": "root", "password": "root", "database": "InWagersDB10", "frontName": "" } } } servers.json: | { "k8s": { "gate": [ { "id": "gate-server-1", "publichost": "192.168.64.6", "host": "game-gate", "port": 4041, "clientPort": 30014, "sslPort": 30015, "webPort": 30050, "keyPath": "../key/ryans-key.pem", "certPath": "../key/ryans-cert.pem", "frontend": true, "restart-force": true } ], "connector": [ { "host": "game-conn", "publichost": "192.168.64.6", "port": "4061++", "clientPort": "30053++", "sslPort": "30063++", "keyPath": "../key/ryans-key.pem", "certPath": "../key/ryans-cert.pem", "clusterCount": 4, "frontend": true, "restart-force": true } ], "game": [ { "host": "game-logic", "port": "6065++", "clusterCount": 1, "restart-force": true }, { "host": "game-logic", "port": "6070++", "clusterCount": 1, "restart-force": true, "gameIDs": [ 11001, 11002, 11003, 11004, 11005, 10012, 10013, 12006, 12008, 12009, 12010, 12011, 12012, 12013, 12014, 12015, 13001, 14001, 14002, 14003, 15001 ] } ], "lobby": [ { "id": "lobby-server-1", "host": "game-svc", "port": 4001, "restart-force": true } ], "dba": [ { "id": "dba-server-1", "host": "game-svc", "port": 8001, "restart-force": true } ], "api": [ { "id": "api-server-1", "host": "game-svc", "port": 2001, "restart-force": true } ], "pool": [ { "id": "pool-server-1", "host": "game-svc", "port": 6001, "restart-force": true } ], "wager": [ { "id": "wager-server-1", "host": "game-svc", "port": 7001, "restart-force": true } ] } } ``` `resources` 會根據 yaml 檔的內容進行部署執行時只需下 `kubectl apply -k <kustomization.yaml 路徑>` 指令就可部署完成 :::success :mag:更多 kustomize 用法: * [使用 Kustomize 管理 Kubernetes 配置檔](https://ellis-wu.github.io/2018/07/26/kustomize-introduction/) * [Oh My Kustomize ! 部署應用的神兵利器](https://ithelp.ithome.com.tw/articles/10218646) * [github](https://github.com/kubernetes-sigs/kustomize) ::: --- ## 實際演練 ### [安裝 kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/) ```shell= curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl" chmod +x ./kubectl sudo mv ./kubectl /usr/local/bin/kubectl sudo chown root: /usr/local/bin/kubectl ``` or ```shell= brew install kubectl ``` 測試安裝結果 ```shell= kubectl version --client ``` ### 安裝 minikube ```shell= curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-amd64 sudo install minikube-darwin-amd64 /usr/local/bin/minikube ``` or ```shell= brew install minikube ``` 測試安裝結果 ```shell= minikube version ``` ### 執行 k8s ```shell= minikube start --driver=hyperkit --cpus=6 --memory='8g' ``` >minikube start -h 可以看到很多 option parameters，例如 `--cpus` `--memory` 等，這邊因為是要跑 pomelo，所以規格開高一點測試 cluster 建立是否成功 ```shell= kubectl cluster-info ``` ![](https://i.imgur.com/yZ1wj2s.png) :::warning :musical_note: 請記下這個 IP (`192.168.64.6` ，實際可能會和我的不同)，這是 minikube 在你的本機的 IP，之後 pomelo 的對外連線需要靠它 ::: --- ### [建立一個 hello-world app](https://kubernetes.io/docs/tutorials/hello-minikube/) ```shell= kubectl create deployment hello-node --image=k8s.gcr.io/echoserver:1.4 ``` 觀察結果 ```shell= kubectl get deployments ``` ![](https://i.imgur.com/tmyAr3R.png) ```shell= kubectl get pods ``` ![](https://i.imgur.com/6VreMRY.png) 將 `hello-world` 暴露到外部 ```shell= kubectl expose deployment hello-node --type=LoadBalancer --port=8080 ``` 觀察結果 ```shell= kubectl get services ``` ![](https://i.imgur.com/h7ps2Xe.png) 因為我們的 cluster 是建立在 minikube 上，所以可以透過它來訪問這個服務 ```shell= minikube service list ``` ![](https://i.imgur.com/0enWUh5.png) 用瀏覽器開啟 url 或透過 `curl` 觀察結果 ![](https://i.imgur.com/9fXdjOF.png) ![](https://i.imgur.com/xqjrLXd.png) clean up 清理善後 ```shell= kubectl delete service hello-node kubectl delete deployment hello-node ``` --- ### 執行 pomelo on minikube 首先到[Cloud Source Repository](https://source.cloud.google.com/gcp-20201229-002/BattlePomeloKustomize)下載 `kustomization` 專案 >Cloud Source Repo 是 GCP 提供的代碼管理倉庫(類似 github Gitlab) 在 clone 之前要先註冊 ssh 金鑰產生一組新的 key ```shell= ssh-keygen -t ecdsa -C "<你的公司 e-mail>" ``` 複製公鑰 ```shell= cat ~/.ssh/id_ecdsa.pub ``` 到[這邊](https://source.cloud.google.com/user/ssh_keys?register=true)註冊金鑰 ![](https://i.imgur.com/fPOPSXW.png) 金鑰名稱取一個不重複容易識別的就好(例如 : 公司 ID) ![](https://i.imgur.com/FOttDLM.png) 複製 clone url ![](https://i.imgur.com/qlvNQUq.png) >要是 clone 失敗無法解決的話，可以暫時先到[這裡](https://storage.cloud.google.com/bbcg-server/BattlePomeloKustomize.zip)下載專案架構 ![](https://i.imgur.com/BYPM4iq.png) 這邊的 `mysqlSetting.json` 建議先改成連 local DB >我本機有自架 mysql on docker，所以都連自己的 DB ```json= { "k8s": { "inDBService": { "host": "192.168.50.129", "account": "root", "password": "root", "database": "InBattle", "frontName": "" }, "wagersDBService": { "host": "192.168.50.129", "account": "root", "password": "root", "database": "InWagersDB10", "frontName": "" } } } ``` `connector` 和 `gate` 的 `publichost` 請改成你的 minikube IP >忘記 IP 的可以下 `minikube ip` 指令查詢 ```json= "gate": [ { "id": "gate-server-1", "publichost": "192.168.64.6", "host": "game-gate", "port": 4041, "clientPort": 30014, "sslPort": 30015, "webPort": 30050, "keyPath": "../key/ryans-key.pem", "certPath": "../key/ryans-cert.pem", "frontend": true, "restart-force": true } ], "connector": [ { "host": "game-conn", "publichost": "192.168.64.6", "port": "4061++", "clientPort": "30053++", "sslPort": "30063++", "keyPath": "../key/ryans-key.pem", "certPath": "../key/ryans-cert.pem", "clusterCount": 4, "frontend": true, "restart-force": true } ], ``` 接下來要為 minikube 建立 pomelo image 首先到 `BattlePomelo` 專案根目錄 ```shell= # 切換到 `dev-docker` 分支 git checkout dev-docker # 將 shell session 的 docker env 切換到 minikube eval $(minikube -p minikube docker-env) # build image for minikube docker build -t pomelo/gs:latest -f ./Dockerfile_k8s . ``` 成功的話，下 `docker images | grep pomelo` 指令應該可以看到下圖 ![](https://i.imgur.com/GhU3dbb.png) >注意，這邊的環境是在 minikube 裡，不是你的本機，所以 image list 會不同準備完畢後，到`BattlePomeloKustomize` 專案根目錄下，執行 `kustomize` ``` kubectl apply -k ./overlays/develop/ ``` ![](https://i.imgur.com/28V2Lyb.png) 可以用 Lens 觀察（安裝說明請參考 [Docker & k8s 開發部署共筆](https://hackmd.io/QeMstz1cTFWfPy8dStr1Vw?view#Lens)) ![](https://i.imgur.com/5oYXozs.png) ![](https://i.imgur.com/8XIL9vI.png) ![](https://i.imgur.com/4JPihKA.png) ![](https://i.imgur.com/kcIxdcV.png) 接下來連到 pomelo `master node` 測試可以透過 cli ```shell= kubectl exec -it game-master-0 -- bash ``` 或 Lens ![](https://i.imgur.com/sbMtryO.png) 進到 `master node` 之後，直接執行 `pomelo start -e k8s` 第一次執行可能會遇到下圖的錯誤 ![](https://i.imgur.com/vGZpo6j.png) 因為 poemlo 剛跑起來的時候資源使用會飆高，導致部分 rpc check 可能會 timeout，這部分還在 tune 目前就先關掉再重開一次就可解決 ![](https://i.imgur.com/lgF0m1Q.png) 跑起來了再來就可以用 bot 新增一個 `k8s_local` 環境連過去測試 ![](https://i.imgur.com/6ZH0sqd.png) 連上了 ![](https://i.imgur.com/uvaFm3R.png) --- ## Q&A :::success :question: ::: --- ###### tags: `k8s` `docker` `container` `tutorials`

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.