---
# System prepended metadata

title: kubeadm k8s 備份與升級
tags: [K8S]

---

# kubeadm k8s 備份與升級

## 目標
* 將 k8s `1.32.6` 升級至 `1.33.2`
* 升級前需先將 k8s 備份

### 升級路徑

* 順序是 Master => Worker
* 每次升級最大不允許跨超過一個次版本（minor version），例如 `1.30.z` 升到 `1.33.z` 版本，要照 `1.30.z` -> `1.31.z` -> `1.32.z` -> `1.33.z` 升級

## 環境資訊
```
$ kubectl get no
NAME   STATUS   ROLES           AGE     VERSION
m1     Ready    control-plane   6d18h   v1.32.6
w1     Ready    worker          6d18h   v1.32.6
w2     Ready    worker          6d18h   v1.32.6
w3     Ready    worker          3d      v1.32.6
w4     Ready    worker          3d      v1.32.6
```
## 備份 k8s
* 安裝 etcdctl

```
$ ETCD_RELEASE=$(curl -s https://api.github.com/repos/etcd-io/etcd/releases/latest|grep tag_name | cut -d '"' -f 4)

$ wget https://github.com/etcd-io/etcd/releases/download/${ETCD_RELEASE}/etcd-${ETCD_RELEASE}-linux-amd64.tar.gz

$ tar zxvf etcd-${ETCD_RELEASE}-linux-amd64.tar.gz

$ sudo cp -rp etcd-${ETCD_RELEASE}-linux-amd64/etcdctl /usr/local/bin

$ etcdctl version
etcdctl version: 3.6.1
API version: 3.6
```
* 設定與測試 etcdctl
```
$ alias etcdctl="ETCDCTL_API=3 sudo /usr/local/bin/etcdctl \
--endpoints=127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key"

$ etcdctl member list
75369154dc9822ab, started, m1, https://10.10.7.30:2380, https://10.10.7.30:2379, false
```

### 對 etcd snapshot
```
$ mkdir ~/etcd

$ etcdctl snapshot save "$HOME"/etcd/etcd-snapshot.db
```
* 檢查備份
```
$ ls -l "$HOME"/etcd
total 25556
-rw------- 1 root root 26165280 Jun 26 15:24 etcd-snapshot.db
```

## 開始升級 k8s

### 升級 Master node

* 更新套件清單到 `1.33` 版本
```
$ sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt update
```
* 檢查可以升級到的版本有哪些
```
$ sudo apt-cache madison kubeadm kubelet kubectl
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubectl | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubectl | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubectl | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
```
* 升級 kubeadm 套件
```
$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.2-1.1' && \
sudo apt-mark hold kubeadm
```
* 確認 kubeadm 套件版本
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```

* 驅逐 master 上的 workload
```
$ kubectl drain m1 --ignore-daemonsets --delete-emptydir-data --force
```


* 測試升級計畫
```
$ sudo kubeadm upgrade plan
```
螢幕輸出
```
[preflight] Running pre-flight checks.
[upgrade/config] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"...
[upgrade/config] Use 'kubeadm init phase upload-config --config your-config-file' to re-upload it.
[upgrade] Running cluster health checks
[upgrade] Fetching available versions to upgrade to
[upgrade/versions] Cluster version: 1.32.6
[upgrade/versions] kubeadm version: v1.33.2
[upgrade/versions] Target version: v1.33.2
[upgrade/versions] Latest version in the v1.32 series: v1.32.6

Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   NODE      CURRENT   TARGET
kubelet     m1        v1.32.6   v1.33.2
kubelet     w1        v1.32.6   v1.33.2
kubelet     w2        v1.32.6   v1.33.2
kubelet     w3        v1.32.6   v1.33.2
kubelet     w4        v1.32.6   v1.33.2

Upgrade to the latest stable version:

COMPONENT                 NODE      CURRENT    TARGET
kube-apiserver            m1        v1.32.6    v1.33.2
kube-controller-manager   m1        v1.32.6    v1.33.2
kube-scheduler            m1        v1.32.6    v1.33.2
kube-proxy                          1.32.6     v1.33.2
CoreDNS                             v1.11.3    v1.12.0
etcd                      m1        3.5.16-0   3.5.21-0

You can now apply the upgrade by executing the following command:

        kubeadm upgrade apply v1.33.2

_____________________________________________________________________


The table below shows the current state of component configs as understood by this version of kubeadm.
Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
upgrade to is denoted in the "PREFERRED VERSION" column.

API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
kubelet.config.k8s.io     v1beta1           v1beta1             no
_____________________________________________________________________
```


* master node 測試沒問題後就，正式升級到 1.33.2
```
$ sudo kubeadm upgrade apply v1.33.2
```
螢幕輸出

```
.......
[upgrade] SUCCESS! A control plane node of your cluster was upgraded to "v1.33.2".

[upgrade] Now please proceed with upgrading the rest of the nodes by following the right order.
```

* 如果要升級其他 master node 時，是使用 `upgrade node` 而不是 `upgrade apply`
```
$ sudo kubeadm upgrade node
```

* 升級 kubelet 和 kubectl

```
$ sudo apt-mark unhold kubelet kubectl && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.2-1.1' kubectl='1.33.2-1.1' && \
sudo apt-mark hold kubelet kubectl
```
* 重新啟動 kubelet
```
$ sudo systemctl daemon-reload

$ sudo systemctl restart kubelet
```
* uncordon master node
```
$ kubectl uncordon m1
```
* 確認 master node 已升級完畢
```
$ kubectl get no
NAME   STATUS   ROLES           AGE     VERSION
m1     Ready    control-plane   6d19h   v1.33.2
w1     Ready    worker          6d19h   v1.32.6
w2     Ready    worker          6d19h   v1.32.6
w3     Ready    worker          3d      v1.32.6
w4     Ready    worker          3d      v1.32.6
```

* 升級時會自動備份到 `/etc/kubernetes/tmp` 再升級後可以考慮只保留最新的一份即可
```
$ sudo ls -l /etc/kubernetes/tmp
total 84
drwx------ 2 root root 4096 Jun 27 09:49 kubeadm-backup-etcd-2025-06-27-09-49-05
drwx------ 3 root root 4096 Jun 27 09:51 kubeadm-backup-etcd-2025-06-27-09-51-24
```

### 升級 Worker node，更新 worker node 要逐一更新

* 更新套件清單到 `1.33` 版本
```
$ sudo curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.33/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg

$ echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.33/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

$ sudo apt update
```
* 檢查可以升級到的版本有哪些
```
$ sudo apt-cache madison kubeadm kubelet
   kubeadm | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubeadm | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.2-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.1-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
   kubelet | 1.33.0-1.1 | https://pkgs.k8s.io/core:/stable:/v1.33/deb  Packages
```
* 升級 kubeadm 套件
```
$ sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm='1.33.2-1.1' && \
sudo apt-mark hold kubeadm
```
* 確認 kubeadm 套件版本
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```

* 驅逐 worker 上的 workload
```
# master 上執行
$ kubectl drain w1 --ignore-daemonsets --delete-emptydir-data --force

$ kubectl get no
NAME   STATUS                     ROLES           AGE     VERSION
m1     Ready                      control-plane   6d19h   v1.33.2
w1     Ready,SchedulingDisabled   worker          6d19h   v1.32.6
w2     Ready                      worker          6d19h   v1.32.6
w3     Ready                      worker          3d      v1.32.6
w4     Ready                      worker          3d      v1.32.6
```
* 開始升級 worker node
```
$ sudo kubeadm upgrade node
```
* 升級 kubelet 套件
```
$ sudo apt-mark unhold kubelet && \
sudo apt-get update && sudo apt-get install -y kubelet='1.33.2-1.1' && \
sudo apt-mark hold kubelet
```


* 重新啟動 kubelet
```
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet
```
* uncordon worker node
```
$ kubectl uncordon w1
```

* 確認 w1 worker 升級成功
```
$ kubectl get no
NAME   STATUS   ROLES           AGE     VERSION
m1     Ready    control-plane   6d19h   v1.33.2
w1     Ready    worker          6d19h   v1.33.2
w2     Ready    worker          6d19h   v1.32.6
w3     Ready    worker          3d      v1.32.6
w4     Ready    worker          3d      v1.32.6
```


## 以上步驟重複執行升級所有 worker

```
$ kubectl get no
NAME   STATUS   ROLES           AGE     VERSION
m1     Ready    control-plane   6d19h   v1.33.2
w1     Ready    worker          6d19h   v1.33.2
w2     Ready    worker          6d19h   v1.33.2
w3     Ready    worker          3d      v1.33.2
w4     Ready    worker          3d      v1.33.2
```




## 故障排除

### 問題一
* 升級時遇到以下報錯
```
[upgrade/staticpods] This can take up to 5m0s
[upgrade/etcd] Failed to upgrade etcd: couldn't upgrade control plane. kubeadm has tried to recover everything into the earlier state. Errors faced: failed to obtain static Pod hash for component etcd on Node m1: pods "etcd-m1" is forbidden: User "kubernetes-admin" cannot get resource "pods" in API group "" in the namespace "kube-system"
```
* 解決辦法
* 原因是因為這個 etcd 之前有做過 restore，有改過 `etcd-data` 的位置，升級時一定要將 `etcd-data` 位置指向 `/var/lib/etcd`
```
$ sudo nano /etc/kubernetes/manifests/etcd.yaml
......
  - hostPath:
      path: /var/lib/etcd      # 修改此行
      type: DirectoryOrCreate
    name: etcd-data
```


## 參考


https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/