# kubeadm k8s 備份與升級
## 目標
* 將 k8s `1.32.6` 升級至 `1.33.2`
* 升級前需先將 k8s 備份
### 升級路徑
* 順序是 Master => Worker
* 每次升級最大不允許跨超過一個次版本（minor version），例如 `1.30.z` 升到 `1.33.z` 版本，要照 `1.30.z` -> `1.31.z` -> `1.32.z` -> `1.33.z` 升級
## 環境資訊
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane 6d18h v1.32.6
w1 Ready worker 6d18h v1.32.6
w2 Ready worker 6d18h v1.32.6
w3 Ready worker 3d v1.32.6
w4 Ready worker 3d v1.32.6
```
## 備份 k8s
* 安裝 etcdctl
```
$ ETCD_RELEASE=$(curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest|grep tag_name | cut -d '"' -f 4)
$ wget https://github.com/etcd-io/etcd/releases/download/${ETCD_RELEASE}/etcd-${ETCD_RELEASE}-linux-amd64.tar.gz
$ tar zxvf etcd-${ETCD_RELEASE}-linux-amd64.tar.gz
$ sudo cp -rp etcd-${ETCD_RELEASE}-linux-amd64/etcdctl /usr/local/bin
$ etcdctl version
etcdctl version: 3.6.1
API version: 3.6
```
* 設定與測試 etcdctl
```
$ alias etcdctl="ETCDCTL_API=3 sudo /usr/local/bin/etcdctl \
--endpoints=127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key"
$ etcdctl member list
75369154dc9822ab, started, m1, https://10.10.7.30:2380, https://10.10.7.30:2379, false
```
### 對 etcd snapshot
```
$ mkdir ~/etcd
$ etcdctl snapshot save "$HOME"/etcd/etcd-snapshot.db
```
* 檢查備份
```
$ ls -l "$HOME"/etcd
total 25556
-rw------- 1 root root 26165280 Jun 26 15:24 etcd-snapshot.db
```
## 開始升級 k8s
### 升級 Master node
* 更新套件清單到 `1.33` 版本
```
$ sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
$ sudo apt update
```
* 檢查可以升級到的版本有哪些
```
$ sudo apt-cache madison kubeadm kubelet kubectl
kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubectl | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubectl | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubectl | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
```
* 升級 kubeadm 套件
```
$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.2-1.1' && \
sudo apt-mark hold kubeadm
```
* 確認 kubeadm 套件版本
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```
* 驅逐 master 上的 workload
```
$ kubectl drain m1 --ignore-daemonsets --delete-emptydir-data --force
```
* 測試升級計畫
```
$ sudo kubeadm upgrade plan
```
螢幕輸出
```
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.6
[upgrade/versions] kubeadm version: v1.33.2
[upgrade/versions] Target version: v1.33.2
[upgrade/versions] Latest version in the v1.32 series: v1.32.6
Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT NODE CURRENT TARGET
kubelet m1 v1.32.6 v1.33.2
kubelet w1 v1.32.6 v1.33.2
kubelet w2 v1.32.6 v1.33.2
kubelet w3 v1.32.6 v1.33.2
kubelet w4 v1.32.6 v1.33.2
Upgrade to the latest stable version:
COMPONENT NODE CURRENT TARGET
kube-apiserver m1 v1.32.6 v1.33.2
kube-controller-manager m1 v1.32.6 v1.33.2
kube-scheduler m1 v1.32.6 v1.33.2
kube-proxy 1.32.6 v1.33.2
CoreDNS v1.11.3 v1.12.0
etcd m1 3.5.16-0 3.5.21-0
You can now apply the upgrade by executing the following command:
kubeadm upgrade apply v1.33.2
_____________________________________________________________________
The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.
API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io v1alpha1 v1alpha1 no
kubelet.config.k8s.io v1beta1 v1beta1 no
_____________________________________________________________________
```
* master node 測試沒問題後就，正式升級到 1.33.2
```
$ sudo kubeadm upgrade apply v1.33.2
```
螢幕輸出
```
.......
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.2".
[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
```
* 如果要升級其他 master node 時，是使用 `upgrade node` 而不是 `upgrade apply`
```
$ sudo kubeadm upgrade node
```
* 升級 kubelet 和 kubectl
```
$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.2-1.1' kubectl='1.33.2-1.1' && \
sudo apt-mark hold kubelet kubectl
```
* 重新啟動 kubelet
```
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet
```
* uncordon master node
```
$ kubectl uncordon m1
```
* 確認 master node 已升級完畢
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane 6d19h v1.33.2
w1 Ready worker 6d19h v1.32.6
w2 Ready worker 6d19h v1.32.6
w3 Ready worker 3d v1.32.6
w4 Ready worker 3d v1.32.6
```
* 升級時會自動備份到 `/etc/kubernetes/tmp` 再升級後可以考慮只保留最新的一份即可
```
$ sudo ls -l /etc/kubernetes/tmp
total 84
drwx------ 2 root root 4096 Jun 27 09:49 kubeadm-backup-etcd-2025-06-27-09-49-05
drwx------ 3 root root 4096 Jun 27 09:51 kubeadm-backup-etcd-2025-06-27-09-51-24
```
### 升級 Worker node，更新 worker node 要逐一更新
* 更新套件清單到 `1.33` 版本
```
$ sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
$ sudo apt update
```
* 檢查可以升級到的版本有哪些
```
$ sudo apt-cache madison kubeadm kubelet
kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
kubelet | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb Packages
```
* 升級 kubeadm 套件
```
$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.2-1.1' && \
sudo apt-mark hold kubeadm
```
* 確認 kubeadm 套件版本
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```
* 驅逐 worker 上的 workload
```
# master 上執行
$ kubectl drain w1 --ignore-daemonsets --delete-emptydir-data --force
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane 6d19h v1.33.2
w1 Ready,SchedulingDisabled worker 6d19h v1.32.6
w2 Ready worker 6d19h v1.32.6
w3 Ready worker 3d v1.32.6
w4 Ready worker 3d v1.32.6
```
* 開始升級 worker node
```
$ sudo kubeadm upgrade node
```
* 升級 kubelet 套件
```
$ sudo apt-mark unhold kubelet && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.2-1.1' && \
sudo apt-mark hold kubelet
```
* 重新啟動 kubelet
```
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet
```
* uncordon worker node
```
$ kubectl uncordon w1
```
* 確認 w1 worker 升級成功
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane 6d19h v1.33.2
w1 Ready worker 6d19h v1.33.2
w2 Ready worker 6d19h v1.32.6
w3 Ready worker 3d v1.32.6
w4 Ready worker 3d v1.32.6
```
## 以上步驟重複執行升級所有 worker
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready control-plane 6d19h v1.33.2
w1 Ready worker 6d19h v1.33.2
w2 Ready worker 6d19h v1.33.2
w3 Ready worker 3d v1.33.2
w4 Ready worker 3d v1.33.2
```
## 故障排除
### 問題一
* 升級時遇到以下報錯
```
[upgrade/staticpods] This can take up to 5m0s
[upgrade/etcd] Failed to upgrade etcd: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: failed to obtain static Pod hash for component etcd on Node m1: pods "etcd-m1" is forbidden: User "kubernetes-admin" cannot get resource "pods" in API group "" in the namespace "kube-system"
```
* 解決辦法
* 原因是因為這個 etcd 之前有做過 restore，有改過 `etcd-data` 的位置，升級時一定要將 `etcd-data` 位置指向 `/var/lib/etcd`
```
$ sudo nano /etc/kubernetes/manifests/etcd.yaml
......
- hostPath:
path: /var/lib/etcd # 修改此行
type: DirectoryOrCreate
name: etcd-data
```
## 參考
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/
Sign in with Wallet
Connect another wallet