# 把 K8S ETCD Leader 那台 VM 給砍了，在從虛擬化那層重新 Restore VM 回來會發生什麼事
## 測試環境
- K8S: RKE2 (3M2W)
- OS: RHEL 9
- 虛擬化平台: Proxmox
## 先備知識
### Etcd 名詞
* Raft : 強一致性的算法。
* Node : 一個 Raft State Machine 實例。
* Member : 一個 etcd 實例。它管理著一個 Node，並且可以為客戶端請求提供服務。
* Peer : 對同一個 etcd 集群中另外一個 Member 的稱呼。
* WAL : 預寫式日誌，etcd 用於持久化儲存的日誌格式。
* snapshot : etcd防止WAL文件過多而設置的快照，儲存 etcd 資料狀態。
* Proxy : etcd 的一種模式，為 etcd 集群提供反向代理服務。
* Leader : Raft 算法中通過競選而產生的處理所有資料提交的節點。
* Follower : 競選失敗的節點作為 Raft 中的附屬節點，為算法提供強一致性保證。
* Candidate : 當 Follower 超過一定時間接收不到 Leader 的心跳時轉變為 Candidate 並且開始競選 Leader。
* Term : 某個節點成為 Leader 到下一次競選時間，稱為一個 Term，可以理解為一個任期。
* Index : 資料編號。 Raft 中通過 Term 和 Index 來定位資料。
### 選舉方法
1. 初始啟動時，節點處於 Follower 狀態並且會被設定一個 election timeout ，如果在這一個時間周期內沒有收到 Leader 的 heartbeat ，節點將會發起選舉，將自己切換為 candidate 之後，並向集群中其他 Follower 節點發送，詢問其是否選舉自己為 Leader 。
2. 當收到來自集群中超過半數節點的接受頭票後，節點成為 Leader ，開始接收保存 client 的數據，並向其他的 Follower 節點同步日誌。如果沒有達成一致，則 candidate 隨機選擇等待間隔 (150ms ~ 300ms) 再次發起投票，得到集群中的半數以上 Follower 接受的 candidate 將成為 Leader 。
3. Leader 節點依靠定時向 Follower 發送 heartbeat 來保持其地位。
4. 任何時候如果其他 Follower 在 election timeout 期間都沒有收到來自 Leader 的 heartbeat，同樣會將自己的狀態切換為 candidate 並發起選舉。每成功選舉一次，新 Leader 的任期 (Term) 都會比之前 Leader 的任期大 1。
5. Leader 完整性 (Leader Completeness) ： 指 Leader 日誌的完整性，當 Log 在任期 Term1 被 commit 後，那麼以後的任期，Leader 都必須包含該 Log ，Raft 在選舉階段就使用 Term 的判斷用於保證完整性，當請求投票的該 Candidate 的 Term 較大或 Term 相同 Index 較大則投票，否則拒絕該請求。
## 先確認誰是 etcd leader
```bash!
$ kubectl -n kube-system exec -it etcd-k1m1 -- bash
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379,https://192.168.11.181:2379,https://192.168.11.182:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status -w table
```
螢幕輸出 :
```bash=
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://127.0.0.1:2379 | 18feb806d6d7ca6c | 3.5.9 | 13 MB | true | false | 15 | 9443274 | 9443274 | |
| https://192.168.11.181:2379 | 7a7b4c1993f92918 | 3.5.9 | 12 MB | false | false | 15 | 9443274 | 9443274 | |
| https://192.168.11.182:2379 | b79561c9abbe9615 | 3.5.9 | 13 MB | false | false | 15 | 9443274 | 9443274 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```
> k1m1 是 ETCD Leader，將 k1m1 那台 Node 關機，並建立硬碟的備份
> 
> 再從 Proxmox node Storgae -> Backups -> Restore 一台新的 VM 出來，並開機。
## 檢查誰是新的 leader
```bash!
$ ETCDCTL_ENDPOINTS='https://192.168.11.180:2379,https://192.168.11.181:2379,https://192.168.11.182:2379' ETCDCTL_CACERT='/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/rke2/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/rke2/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status -w table
```
螢幕輸出 :
```bash=
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.11.180:2379 | 18feb806d6d7ca6c | 3.5.9 | 15 MB | false | false | 17 | 9457565 | 9457565 | |
| https://192.168.11.181:2379 | 7a7b4c1993f92918 | 3.5.9 | 15 MB | false | false | 17 | 9457565 | 9457565 | |
| https://192.168.11.182:2379 | b79561c9abbe9615 | 3.5.9 | 15 MB | true | false | 17 | 9457565 | 9457565 | |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```
> k1m3 被重新選舉為新的 node
## 檢視 k1m1 ETCD pod logs
```bash!
$ kubectl -n kube-system logs etcd-k1m1 | less
```
螢幕輸出 :
```bash!
{"level":"info","ts":"2024-04-02T02:01:19.822099Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"18feb806d6d7ca6c [term: 15] received a MsgHeartbeat message with higher term from b79561c9abbe9615 [term: 17]"}
{"level":"info","ts":"2024-04-02T02:01:19.82212Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"18feb806d6d7ca6c became follower at term 17"}
{"level":"info","ts":"2024-04-02T02:01:19.822137Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 18feb806d6d7ca6c elected leader b79561c9abbe9615 at term 17"}
```
> Term : 某個節點成為 Leader 到下一次競選時間，稱為一個 Term，可以理解為一個任期。
可以看到在第 17 任期，這台 node 變成 follower。
## 檢視 k1m3 ETCD pod logs
```bash!
$ kubectl -n kube-system logs etcd-k1m3 | less
```
螢幕輸出 :
```bash!
{"msg":"b79561c9abbe9615 became candidate at term 17"}
{"msg":"b79561c9abbe9615 received MsgVoteResp from b79561c9abbe9615 at term 17"}
{"msg":"b79561c9abbe9615 [logterm: 16, index: 9454172] sent MsgVote request to 18feb806d6d7ca6c at term 17"}
{"msg":"b79561c9abbe9615 [logterm: 16, index: 9454172] sent MsgVote request to 7a7b4c1993f92918 at term 17"}
{"msg":"waiting for ReadIndex response took too long, retrying","sent-request-id":10814706598670506357,"retry-timeout":"500ms"}
{"msg":"b79561c9abbe9615 received MsgVoteResp from 7a7b4c1993f92918 at term 17"}
{"msg":"b79561c9abbe9615 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"msg":"b79561c9abbe9615 became leader at term 17"}
{"msg":"raft.node: b79561c9abbe9615 elected leader b79561c9abbe9615 at term 17"}
```
> 可以看到在第 17 任期，這台 k1m3 變成 leader。
## 參考文章
- [Kubernetes 核心介紹 Etcd](https://alanzhan.dev/post/2022-02-28-kubetnetes-etcd/)
Sign in with Wallet
Connect another wallet