# 全離線安裝 Kubeadm K8s
## 目標
在離線環境上安裝 3m2w 的 K8s，詳細資訊如下
1. k8s 版本 `1.33.2`
2. containerd 版本 `2.1.3`
3. runc 版本 `1.3.0`
4. CNI: calico `3.29.4`
注意: 本版資訊會隨時間更動
## 事前檔案準備
在一台可上網環境的機器，且有安裝 `docker` 或 `podman`
```
$ mkdir -p work/{harbor,k8s}
```
### 準備 harbor(如果客戶已有 harbor 可以跳過此步驟)
```!
$ cd ~/work/harbor
$ wget https://github.com/goharbor/harbor/releases/download/v2.7.0/harbor-offline-installer-v2.7.0.tgz
```
```!
# 準備 docker
$ sudo wget https://download.docker.com/linux/static/stable/x86_64/docker-28.3.0.tgz
# 準備 docker-compose
$ sudo wget https://github.com/docker/compose/releases/download/1.18.0/docker-compose-Linux-x86_64
```
* 下載自簽憑證 script
```
$ wget https://raw.githubusercontent.com/cooloo9871/SelfSigned-RootCA/master/mk
$ chmod +x mk
```
* 做成壓縮包
```
$ cd ..
$ ls -l harbor
total 1716872
-rw-r--r-- 1 root root 81874097 Jun 24 23:55 docker-28.3.0.tgz
-rw-r--r-- 1 root root 8479184 Dec 7 2021 docker-compose-Linux-x86_64
-rw-rw-r-- 1 bigred bigred 789527572 Dec 19 2022 harbor-offline-installer-v2.7.0.tgz
-rw-rw-r-- 1 bigred bigred 878174181 Jun 25 06:17 harbor.tar.zst
-rwxrwxr-x 1 bigred bigred 1515 Jun 25 06:33 mk
$ tar --zstd -cvf harbor.tar.zst -C ~/work harbor
```
```
$ ls -l ~/work
total 857316
drwxrwxr-x 2 bigred bigred 4096 Jun 25 11:19 harbor
-rw-rw-r-- 1 bigred bigred 877875781 Jun 25 14:08 harbor.tar.zst
drwxrwxr-x 2 bigred bigred 4096 Jun 25 14:03 k8s
```
### 準備 k8s
```
$ cd ~/work/k8s
```
* 下載 master 套件
```
$ wget https://dl.k8s.io/v1.33.2/kubernetes-server-linux-amd64.tar.gz
```
* 下載 worker 套件
```
$ wget https://dl.k8s.io/v1.33.2/kubernetes-node-linux-amd64.tar.gz
```
* 下載 containerd
```
$ wget https://github.com/containerd/containerd/releases/download/v2.1.3/containerd-2.1.3-linux-amd64.tar.gz
$ wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service
```
* 下載 runC
```
$ wget https://github.com/opencontainers/runc/releases/download/v1.3.0/runc.amd64
```
* 下載 cni
```
$ curl -sL "$(curl -sL https://api.github.com/repos/containernetworking/plugins/releases/latest | \
jq -r '.assets[].browser_download_url' | grep 'linux-amd64.*.tgz$')" -o cni-plugins.tgz
```
* 下載 crictl
```
# 根據 k8s 版本指定
$ VERSION="v1.33.0"
$ wget https://github.com/kubernetes-sigs/cri-tools/releases/download/$VERSION/crictl-$VERSION-linux-amd64.tar.gz
```
* 下載 `calico 3.29.4` yaml
```
$ curl -sL https://raw.githubusercontent.com/projectcalico/calico/v3.29.4/manifests/calico.yaml -o calico.yaml
```
* 下載 `metrics-server` yaml
```
$ wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
##- --kubelet-insecure-tls 加這行是告訴 kubelet 不要安全模式，告訴kubelet封包不要加密。
$ nano components.yaml
..........
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --metric-resolution=15s
- --kubelet-insecure-tls # 新增這行
image: k8s.gcr.io/metrics-server/metrics-server:v0.7.2
imagePullPolicy: IfNotPresent
```
* 取得 `kube-vip` 版本代號
```
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1
# 因此 kube-vip image 就是 ghcr.io/kube-vip/kube-vip:v0.9.2
```
* 安裝 kubeadm 查看所需要的 image
```
$ wget https://dl.k8s.io/v1.33.2/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes/server/bin
$ sudo cp kubeadm /usr/bin/
```
* 透過 kubeadm 可以先查看 `1.33.2` 以及其他版本所需的 image
```
$ kubeadm config images list --kubernetes-version=v1.33.2
registry.k8s.io/kube-apiserver:v1.33.2
registry.k8s.io/kube-controller-manager:v1.33.2
registry.k8s.io/kube-scheduler:v1.33.2
registry.k8s.io/kube-proxy:v1.33.2
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.21-0
$ kubeadm config images list --kubernetes-version=v1.33.1
registry.k8s.io/kube-apiserver:v1.33.1
registry.k8s.io/kube-controller-manager:v1.33.1
registry.k8s.io/kube-scheduler:v1.33.1
registry.k8s.io/kube-proxy:v1.33.1
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.21-0
$ kubeadm config images list --kubernetes-version=v1.33.0
registry.k8s.io/kube-apiserver:v1.33.0
registry.k8s.io/kube-controller-manager:v1.33.0
registry.k8s.io/kube-scheduler:v1.33.0
registry.k8s.io/kube-proxy:v1.33.0
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.21-0
```
* 編寫下載 image script
```
$ nano pull.sh
#!/bin/bash
#set -x
RED='\033[1;31m' # alarm
GRN='\033[1;32m' # notice
YEL='\033[1;33m' # warning
NC='\033[0m' # No Color
[[ ! -f ./images.txt ]] && printf "${RED}images.txt file not found${NC}\n" && exit 1
image_name=$(paste -sd' ' images.txt)
docker_command() {
while read line
do
sudo docker pull $line &>/dev/null
if [[ "$?" == 0 ]]; then
printf "${GRN}download $line success${NC}\n"
else
printf "${RED}download $line fail${NC}\n"
fi
done < images.txt
sudo docker save $image_name > k8s_images.tar
tar --zstd -cf images.tar.zst k8s_images.tar
rm -rf k8s_images.tar
printf "${GRN}All images have been saved as images.tar.zst${NC}\n"
}
podman_command() {
while read line
do
sudo podman pull $line &>/dev/null
if [[ "$?" == 0 ]]; then
printf "${GRN}download $line success${NC}\n"
else
printf "${RED}download $line fail${NC}\n"
fi
done < images.txt
sudo podman save -m $image_name > k8s_images.tar
tar --zstd -cf images.tar.zst k8s_images.tar
rm -rf k8s_images.tar
printf "${GRN}All images have been saved as images.tar.zst${NC}\n"
}
if ! which docker &>/dev/null && ! which podman &>/dev/null; then
printf "${RED}docker and podman command not found${NC}\n" && exit 1
fi
sudo -n true &>/dev/null
if [[ "$?" != "0" ]]; then
printf "${RED}Passwordless sudo is NOT enabled${NC}\n" && exit 1
fi
if which docker &>/dev/null; then
docker_command
else
podman_command
fi
```
* 以下 image 是根據本次測試所需，需依照不同環境準備
```
$ nano images.txt
registry.k8s.io/kube-apiserver:v1.33.0
registry.k8s.io/kube-controller-manager:v1.33.0
registry.k8s.io/kube-scheduler:v1.33.0
registry.k8s.io/kube-proxy:v1.33.0
registry.k8s.io/kube-apiserver:v1.33.1
registry.k8s.io/kube-controller-manager:v1.33.1
registry.k8s.io/kube-scheduler:v1.33.1
registry.k8s.io/kube-proxy:v1.33.1
registry.k8s.io/kube-apiserver:v1.33.2
registry.k8s.io/kube-controller-manager:v1.33.2
registry.k8s.io/kube-scheduler:v1.33.2
registry.k8s.io/kube-proxy:v1.33.2
registry.k8s.io/coredns/coredns:v1.12.0
registry.k8s.io/pause:3.10
registry.k8s.io/etcd:3.5.21-0
docker.io/calico/cni:v3.29.4
docker.io/calico/node:v3.29.4
docker.io/calico/kube-controllers:v3.29.4
registry.k8s.io/metrics-server/metrics-server:v0.7.2
ghcr.io/kube-vip/kube-vip:v0.9.2
```
* 開始下載 image，並自動打包
```
$ chmod +x pull.sh
$ ./pull.sh
```
做成 `k8s.tar.zst` 打包檔
```
$ cd ..
$ ls -l k8s/
total 1244756
-rw-rw-r-- 1 bigred bigred 325176 Jun 25 06:19 calico.yaml
-rw-rw-r-- 1 bigred bigred 55901748 Jun 25 08:51 cni-plugins.tgz
-rw-rw-r-- 1 bigred bigred 4340 Jun 25 06:20 components.yaml
-rw-rw-r-- 1 bigred bigred 33200838 Jun 19 22:39 containerd-2.1.3-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 1248 Jun 25 06:19 containerd.service
-rw-rw-r-- 1 bigred bigred 20370694 Apr 22 07:51 crictl-v1.33.0-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 637987968 Jun 26 01:38 images.tar.zst
-rw-rw-r-- 1 bigred bigred 767 Jun 26 01:37 images.txt
-rw-rw-r-- 1 bigred bigred 135110634 Jun 18 17:53 kubernetes-node-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 379831997 Jun 18 17:53 kubernetes-server-linux-amd64.tar.gz
-rwxrwxr-x 1 bigred bigred 1247 Jun 25 07:59 pull.sh
-rw-rw-r-- 1 bigred bigred 11862624 Apr 29 04:43 runc.amd64
$ tar --zstd -cvf k8s.tar.zst -C ~/work k8s
```
#### 將 `harbor.tar.zst` 和 `k8s.tar.zst` 帶至客戶環境做安裝
```
$ ls -l
total 2901648
drwxrwxr-x 2 bigred bigred 4096 Jun 25 06:17 harbor
-rw-rw-r-- 1 bigred bigred 1755574653 Jun 25 06:17 harbor.tar.zst
drwxrwxr-x 2 bigred bigred 4096 Jun 25 06:25 k8s
-rw-rw-r-- 1 bigred bigred 1215692757 Jun 25 06:30 k8s.tar.zst
```
## 開始離線安裝
### 安裝 harbor
* 將 `harbor.tar.zst` 上傳到 harbor 主機，然後解壓縮
```
$ tar --zstd -xf harbor.tar.zst
$ ls -l harbor
total 1716868
-rw-r--r-- 1 bigred bigred 81874097 Jun 25 07:55 docker-28.3.0.tgz
-rw-r--r-- 1 bigred bigred 8479184 Dec 7 2021 docker-compose-Linux-x86_64
-rw-rw-r-- 1 bigred bigred 789527572 Dec 19 2022 harbor-offline-installer-v2.7.0.tgz
-rw-rw-r-- 1 bigred bigred 878174181 Jun 25 14:17 harbor.tar.zst
-rwxrwxr-x 1 bigred bigred 1515 Jun 25 14:33 mk
```
* 安裝 docker
```
$ cd harbor/
$ sudo tar -xvf docker-28.3.0.tgz
$ sudo cp docker/* /usr/bin/
$ sudo nano /etc/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd
ExecReload=/bin/kill -s HUP $MAINPID
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s
[Install]
WantedBy=multi-user.target
```
* 啟動 docker
```
$ sudo chmod +x /etc/systemd/system/docker.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now docker
$ sudo docker version
Client:
Version: 28.3.0
API version: 1.51
Go version: go1.24.4
Git commit: 38b7060
Built: Tue Jun 24 15:43:00 2025
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 28.3.0
API version: 1.51 (minimum version 1.24)
Go version: go1.24.4
Git commit: 265f709
Built: Tue Jun 24 15:44:17 2025
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.27
GitCommit: 05044ec0a9a75232cad458027ca83437aae3f4da
runc:
Version: 1.2.6
GitCommit: v1.2.6-0-ge89a299
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
* 安裝 docker-compose
```
$ sudo mv docker-compose-Linux-x86_64 /usr/bin/docker-compose
$ sudo chmod 755 /usr/bin/docker-compose
```
* 將 harbor 解壓縮，並自簽憑證
```
$ tar xvf harbor-offline-installer-v2.7.0.tgz;cd harbor/
$ cp harbor.yml.tmpl harbor.yml
$ mkdir ssl;mv ../mk ssl;cd ssl
# 確認名稱解析正確
$ host harbor.example.com
harbor.example.com has address 172.20.7.89
# 產生憑證
$ ./mk create harbor.example.com 172.20.7.89
Certificate request self-signature ok
subject=CN = example
$ ls -l
total 32
-rw------- 1 bigred bigred 3434 Jun 25 15:32 ca-key.pem
-rw-rw-r-- 1 bigred bigred 2009 Jun 25 15:32 ca.pem
-rw-rw-r-- 1 bigred bigred 41 Jun 25 15:32 ca.srl
-rw-rw-r-- 1 bigred bigred 1582 Jun 25 15:32 cert.csr
-rw------- 1 bigred bigred 3272 Jun 25 15:32 cert-key.pem
-rw-rw-r-- 1 bigred bigred 1960 Jun 25 15:32 cert.pem
-rw-rw-r-- 1 bigred bigred 83 Jun 25 15:32 extfile.cnf
-rwxrwxr-x 1 bigred bigred 1515 Jun 25 14:33 mk
$ pwd
/home/bigred/harbor/harbor/ssl
```
* 設定 `hostname`
* 設定憑證位置
```
$ cd ..
$ nano harbor.yml
# Configuration file of Harbor
# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: harbor.example.com
# http related config
http:
# port for http, default is 80. If https enabled, this port will redirect to https port
port: 80
# https related config
https:
# https port for harbor, default is 443
port: 443
# The path of cert and key files for nginx
certificate: /home/bigred/harbor/harbor/ssl/cert.pem # 設定憑證絕對路徑
private_key: /home/bigred/harbor/harbor/ssl/cert-key.pem # 設定憑證絕對路徑
......
```
* 設定 docker 可以免信任登入 harbor
```
$ sudo mkdir /etc/docker/
$ sudo nano /etc/docker/daemon.json
{
"log-level": "warn",
"log-driver": "json-file",
"insecure-registries": ["172.22.7.89","harbor.example.com"],
"log-opts": {
"max-size": "10m",
"max-file": "5"
}
}
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
```
* 啟動 harbor
```
$ sudo ./install.sh --with-trivy
```
```
$ sudo docker-compose ps
Name Command State Ports
-------------------------------------------------------------------------------------------------------------
harbor-core /harbor/entrypoint.sh Up
harbor-db /docker-entrypoint.sh 13 Up
harbor-jobservice /harbor/entrypoint.sh Restarting
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up
nginx nginx -g daemon off; Up 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp
redis redis-server /etc/redis.conf Up
registry /home/harbor/entrypoint.sh Up
registryctl /home/harbor/start.sh Up
trivy-adapter /home/scanner/entrypoint.sh Up
```
### 設定 harbor 開機自動啟用
* 須注意 docker-compose 與 docker-compose.yml 絕對路徑是否正確
```
$ which docker-compose
/usr/bin/docker-compose
$ pwd docker-compose.yml
/home/bigred/harbor/harbor
```
```
$ cat <<EOF | sudo tee /usr/lib/systemd/system/harbor.service
[Unit]
Description=Harbor start at boot
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/docker-compose -f /home/bigred/harbor/harbor/docker-compose.yml up
ExecStop=/usr/bin/docker-compose -f /home/bigred/harbor/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
EOF
```
```
$ sudo systemctl enable --now harbor.service
```
* 登入 harbor
> 帳號: admin 密碼: Harbor12345
## 上傳 image
* 將 `k8s.tar.zst` 上傳到有 docker 或 podman 指令的主機上，並解壓縮
```
$ tar --zstd -xf k8s.tar.zst
$ ls -l k8s
total 1244756
-rw-rw-r-- 1 bigred bigred 325176 Jun 25 06:19 calico.yaml
-rw-rw-r-- 1 bigred bigred 55901748 Jun 25 08:51 cni-plugins.tgz
-rw-rw-r-- 1 bigred bigred 4340 Jun 25 06:20 components.yaml
-rw-rw-r-- 1 bigred bigred 33200838 Jun 19 22:39 containerd-2.1.3-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 1248 Jun 25 06:19 containerd.service
-rw-rw-r-- 1 bigred bigred 20370694 Apr 22 07:51 crictl-v1.33.0-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 637987968 Jun 26 01:38 images.tar.zst
-rw-rw-r-- 1 bigred bigred 767 Jun 26 01:37 images.txt
-rw-rw-r-- 1 bigred bigred 135110634 Jun 18 17:53 kubernetes-node-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 379831997 Jun 18 17:53 kubernetes-server-linux-amd64.tar.gz
-rwxrwxr-x 1 bigred bigred 1247 Jun 25 07:59 pull.sh
-rw-rw-r-- 1 bigred bigred 11862624 Apr 29 04:43 runc.amd64
$ cd k8s
```
* 先登入 harbor
```
$ sudo docker login harbor.example.com
Username: admin
Password:
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
Login Succeeded
```
* 編寫 push script
```
$ nano push.sh
#!/bin/bash
# set -x
RED='\033[1;31m' # alarm
GRN='\033[1;32m' # notice
YEL='\033[1;33m' # warning
NC='\033[0m' # No Color
registry="$1"
[[ ! -f ./images.txt ]] && printf "${RED}images.txt file not found${NC}\n" && exit 1
image_name=$(paste -sd' ' images.txt)
docker_command() {
tar --zstd -xf images.tar.zst
sudo docker load < k8s_images.tar &>/dev/null
while read line
do
img="${line##*/}"
sudo docker tag $line $registry/library/$img &>/dev/null
sudo docker push $registry/library/$img &>/dev/null
if [[ "$?" == 0 ]]; then
printf "${GRN}push $line success${NC}\n"
else
printf "${RED}push $line fail${NC}\n"
fi
done < images.txt
printf "${GRN}All images have been successfully uploaded to Harbor.${NC}\n"
}
podman_command() {
tar --zstd -xf images.tar.zst
sudo podman load < k8s_images.tar &>/dev/null
while read line
do
img="${line##*/}"
sudo podman tag $line $registry/library/$img &>/dev/null
sudo podman push $registry/library/$img &>/dev/null
if [[ "$?" == 0 ]]; then
printf "${GRN}push $line success${NC}\n"
else
printf "${RED}push $line fail${NC}\n"
fi
done < images.txt
printf "${GRN}All images have been successfully uploaded to Harbor.${NC}\n"
}
help() {
cat <<EOF
Usage: push.sh [harbor domain]
for example: push.sh harbor.example.com
EOF
exit
}
if ! which docker &>/dev/null && ! which podman &>/dev/null; then
printf "${RED}docker and podman command not found${NC}\n" && exit 1
fi
sudo -n true &>/dev/null
if [[ "$?" != "0" ]]; then
printf "${RED}Passwordless sudo is NOT enabled${NC}\n" && exit 1
fi
if [[ "$#" < 1 ]]; then
help
fi
if which docker &>/dev/null; then
docker_command
else
podman_command
fi
```
* 開始上傳 image 到 harbor
```
$ chmod +x push.sh
$ ./push.sh harbor.example.com
```
* 在 harbor 上查看 library repo 已成功上傳 image
## 安裝 k8s
### 在每個節點先安裝 containerd、runc、cni、crictl 套件
* 將 `k8s.tar.zst` 上傳到每個節點，並解壓縮
```
$ tar --zstd -xf k8s.tar.zst
$ cd k8s
$ ls -l
total 1244756
-rw-rw-r-- 1 bigred bigred 325176 Jun 25 06:19 calico.yaml
-rw-rw-r-- 1 bigred bigred 55901748 Jun 25 08:51 cni-plugins.tgz
-rw-rw-r-- 1 bigred bigred 4340 Jun 25 06:20 components.yaml
-rw-rw-r-- 1 bigred bigred 33200838 Jun 19 22:39 containerd-2.1.3-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 1248 Jun 25 06:19 containerd.service
-rw-rw-r-- 1 bigred bigred 20370694 Apr 22 07:51 crictl-v1.33.0-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 637987968 Jun 26 01:38 images.tar.zst
-rw-rw-r-- 1 bigred bigred 767 Jun 26 01:37 images.txt
-rw-rw-r-- 1 bigred bigred 135110634 Jun 18 17:53 kubernetes-node-linux-amd64.tar.gz
-rw-rw-r-- 1 bigred bigred 379831997 Jun 18 17:53 kubernetes-server-linux-amd64.tar.gz
-rwxrwxr-x 1 bigred bigred 1247 Jun 25 07:59 pull.sh
-rw-rw-r-- 1 bigred bigred 11862624 Apr 29 04:43 runc.amd64
```
* 確認每個節點都可以解析到 harbor
```
$ host harbor.example.com
harbor.example.com has address 172.20.7.89
```
### 安裝 containerd
```
$ sudo tar Cxzvf /usr/local containerd-2.1.3-linux-amd64.tar.gz
$ sudo mkdir -p /usr/local/lib/systemd/system
$ sudo mv containerd.service /usr/local/lib/systemd/system/containerd.service
$ sudo systemctl daemon-reload
$ sudo systemctl enable --now containerd
```
```
$ sudo systemctl status containerd.service
● containerd.service - containerd container runtime
Loaded: loaded (/usr/local/lib/systemd/system/containerd.service; enabled; preset: enabled)
Active: active (running) since Tue 2025-06-10 06:17:14 UTC; 9s ago
Docs: https://containerd.io
Process: 1162 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
```
### 安裝 runc
```
$ sudo install -m 755 runc.amd64 /usr/local/sbin/runc
```
```
$ runc -v
runc version 1.3.0
commit: v1.3.0-0-g4ca628d1
spec: 1.2.1
go: go1.23.8
libseccomp: 2.5.6
```
### 安裝 cni 套件
```
$ sudo mkdir -p /opt/cni/bin
$ sudo tar xf cni-plugins.tgz -C /opt/cni/bin
```
```
$ sudo ls -l /opt/cni/bin
total 96188
-rwxr-xr-x 1 root root 5033580 Apr 25 20:58 bandwidth
-rwxr-xr-x 1 root root 5694447 Apr 25 20:58 bridge
-rwxr-xr-x 1 root root 13924938 Apr 25 20:58 dhcp
-rwxr-xr-x 1 root root 5247557 Apr 25 20:58 dummy
-rwxr-xr-x 1 root root 5749447 Apr 25 20:58 firewall
-rwxr-xr-x 1 root root 5163089 Apr 25 20:58 host-device
-rwxr-xr-x 1 root root 4364143 Apr 25 20:58 host-local
-rwxr-xr-x 1 root root 5269812 Apr 25 20:58 ipvlan
-rw-r--r-- 1 root root 11357 Apr 25 20:58 LICENSE
-rwxr-xr-x 1 root root 4263979 Apr 25 20:58 loopback
-rwxr-xr-x 1 root root 5305057 Apr 25 20:58 macvlan
-rwxr-xr-x 1 root root 5125860 Apr 25 20:58 portmap
-rwxr-xr-x 1 root root 5477120 Apr 25 20:58 ptp
-rw-r--r-- 1 root root 2343 Apr 25 20:58 README.md
-rwxr-xr-x 1 root root 4488703 Apr 25 20:58 sbr
-rwxr-xr-x 1 root root 3736370 Apr 25 20:58 static
-rwxr-xr-x 1 root root 5332257 Apr 25 20:58 tap
-rwxr-xr-x 1 root root 4352498 Apr 25 20:58 tuning
-rwxr-xr-x 1 root root 5267833 Apr 25 20:58 vlan
-rwxr-xr-x 1 root root 4644777 Apr 25 20:58 vrf
```
### 安裝 crictl
```
# 根據 k8s 版本指定
$ VERSION="v1.33.0"
$ sudo tar zxvf crictl-$VERSION-linux-amd64.tar.gz -C /usr/local/bin
$ rm -f crictl-$VERSION-linux-amd64.tar.gz
```
* 設定 crictl
```
$ sudo nano /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
timeout: 10
```
### 生成 containerd 設定檔
```
$ sudo mkdir /etc/containerd
$ sudo containerd config default | sudo tee /etc/containerd/config.toml
```
* 設定 containerd 設定連接 Harbor Bypass TLS
* 建立目錄，`harbor.example.com` 是 harbor 的 domain，此目錄可以任意取名
* 在 `hosts.toml` 設定 `skip_verify = true` 跳過 tls 驗證
```
$ sudo mkdir -p /etc/containerd/certs.d/harbor.example.com
$ sudo nano /etc/containerd/certs.d/harbor.example.com/hosts.toml
server = "https://harbor.example.com"
[host."https://harbor.example.com"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true
```
* 修改 `config.toml`
* 新增 `SystemdCgroup = true`
* 新增 `config_path = '/etc/containerd/certs.d'`
* 修改預設要拉 pause image 的位置 `harbor.example.com/library/pause:3.10`
```
$ sudo nano /etc/containerd/config.toml
......
[plugins.'io.containerd.cri.v1.images'.pinned_images]
sandbox = 'harbor.example.com/library/pause:3.10'
[plugins.'io.containerd.cri.v1.images'.registry]
config_path = '/etc/containerd/certs.d'
......
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
......
[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
SystemdCgroup = true
......
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
```
* 重啟 containerd
```
$ sudo systemctl restart containerd.service
```
### master 節點安裝 kubelet kubeadm kubectl
```
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes/server/bin
$ sudo cp kubeadm kubelet kubectl /usr/bin/
```
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```
### 設定 kubelet service
* 如果有自己更換 cri 需要再更改
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=containerd.service
Requires=containerd.service
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
$ sudo mkdir /etc/systemd/system/kubelet.service.d
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
```
$ sudo systemctl enable --now kubelet.service
```
### worker 節點安裝 kubelet kubeadm
```
$ tar -xzvf kubernetes-node-linux-amd64.tar.gz
$ cd kubernetes/node/bin
$ sudo cp kubeadm kubelet /usr/bin/
```
```
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"33", EmulationMajor:"", EmulationMinor:"", MinCompatibilityMajor:"", MinCompatibilityMinor:"", GitVersion:"v1.33.2", GitCommit:"a57b6f7709f6c2722b92f07b8b4c48210a51fc40", GitTreeState:"clean", BuildDate:"2025-06-17T18:39:42Z", GoVersion:"go1.24.4", Compiler:"gc", Platform:"linux/amd64"}
```
### 設定 kubelet service
* 如果有自己更換 cri 需要再更改
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=containerd.service
Requires=containerd.service
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
$ sudo mkdir /etc/systemd/system/kubelet.service.d
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
```
$ sudo systemctl enable --now kubelet.service
```
## 初始化 k8s
* 設定 kube-vip
```
$ cd ~/k8s
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.7.100
# 宣告網卡名稱
$ export INTERFACE=ens18
# 注意這裡要更換 kube-vip image 位置
$ alias kube-vip="sudo ctr -n k8s.io image pull --hosts-dir "/etc/containerd/certs.d" harbor.example.com/library/kube-vip:v0.9.2;sudo ctr -n k8s.io run --rm --net-host harbor.example.com/library/kube-vip:v0.9.2 vip /kube-vip"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 更換 `kube-vip.yaml` yaml 的 image 位置
```
$ sudo sed -i 's|ghcr.io/kube-vip/kube-vip:v0.9.2|harbor.example.com/library/kube-vip:v0.9.2|g' /etc/kubernetes/manifests/kube-vip.yaml
```
* 確認更改
```
$ cat /etc/kubernetes/manifests/kube-vip.yaml | grep image:
image: harbor.example.com/library/kube-vip:v0.9.2
```
* 1.29 以後需要調整權限，只有第一台 master 需要修改
```
$ sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
```
* `advertiseAddress` 需更換為自己的 master ip
* `controlPlaneEndpoint` 要指定 vip 的位置
```
$ nano init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.20.7.90 # change from Master node IP
bindPort: 6443
nodeRegistration:
criSocket: unix:///run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
name: m1 # change from Master node hsotname
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.33.2
controlPlaneEndpoint: 172.20.7.100:6443
apiServer:
timeoutForControlPlane: 4m0s
certificatesDir: /etc/kubernetes/pki
clusterName: topgun # set your clusterName
controllerManager:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10257"
scheduler:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10259"
etcd:
local:
dataDir: /var/lib/etcd
# imageRepository: harbor.example.com/library
# imageTag: 3.5.15-0
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"
dns: {}
#imageRepository: harbor.example.com/library
#imageTag: v1.11.3
imageRepository: harbor.example.com/library # 更換為內部 harbor 位置
networking:
dnsDomain: cluster.local # DNS domain used by Kubernetes Services.
podSubnet: 10.244.0.0/16 # the subnet used by Pods.
serviceSubnet: 10.96.0.0/16 # subnet used by Kubernetes Services.
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: "0.0.0.0:10249"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
systemReserved:
memory: "1Gi"
kubeReserved:
memory: "2Gi"
```
開始安裝
* `--upload-certs` 將 `control-plane` 節點所需的金鑰和憑證上傳到 `kubeadm-certs Secret` 中，以供其他 `control-plane` 節點下載使用。
```
$ sudo kubeadm init --upload-certs --config=init-config.yaml
```
輸出結果並記錄註冊指令:
```
......
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join 172.20.7.100:6443 --token ma5zm7.1rc9i2qjybt5uve5 \
--discovery-token-ca-cert-hash sha256:cefd410656e767c60f84b2e394c9837326ca03380cb6d07bad9a0fd03d971044 \
--control-plane --certificate-key 126e9fc8daeb3e304187383a4d00999ae2f6cd09e196b662d81755a66a2a0415
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.20.7.100:6443 --token ma5zm7.1rc9i2qjybt5uve5 \
--discovery-token-ca-cert-hash sha256:cefd410656e767c60f84b2e394c9837326ca03380cb6d07bad9a0fd03d971044
```
* 設定 kubeconfig
```
$ mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
```
### 部屬 calico 3.29.4
* 將 calico.yaml 改為 harbor 位置
```
$ sed -i \
-e 's|docker.io/calico/cni:v3.29.4|harbor.example.com/library/cni:v3.29.4|g' \
-e 's|docker.io/calico/node:v3.29.4|harbor.example.com/library/node:v3.29.4|g' \
-e 's|docker.io/calico/kube-controllers:v3.29.4|harbor.example.com/library/kube-controllers:v3.29.4|g' \
calico.yaml
```
* 檢查 image 位置都已更換
```
$ cat calico.yaml|grep image
image: harbor.example.com/library/cni:v3.29.4
imagePullPolicy: IfNotPresent
image: harbor.example.com/library/cni:v3.29.4
imagePullPolicy: IfNotPresent
image: harbor.example.com/library/node:v3.29.4
imagePullPolicy: IfNotPresent
image: harbor.example.com/library/node:v3.29.4
imagePullPolicy: IfNotPresent
image: harbor.example.com/library/kube-controllers:v3.29.4
imagePullPolicy: IfNotPresent
```
```
$ kubectl apply -f calico.yaml
```
### 環境檢查
```
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-7b67c89ff4-cx42g 1/1 Running 0 27s
kube-system calico-node-45fjd 1/1 Running 0 27s
kube-system coredns-6b8f4f6974-6685s 1/1 Running 0 10m
kube-system coredns-6b8f4f6974-jxt4b 1/1 Running 0 10m
kube-system etcd-m1 1/1 Running 0 10m
kube-system kube-apiserver-m1 1/1 Running 0 10m
kube-system kube-controller-manager-m1 1/1 Running 0 10m
kube-system kube-proxy-k8s8s 1/1 Running 0 10m
kube-system kube-scheduler-m1 1/1 Running 0 10m
kube-system kube-vip-m1 1/1 Running 0 10m
```
* k8s 安裝好後再將 `kube-vip` 權限調整回來
```
$ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet
```
## 加入 m2 master node
#### 設定 kube-vip
* 在 m2 執行以下命令
```
$ cd ~/k8s
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.7.100
# 宣告網卡名稱
$ export INTERFACE=ens18
# 注意這裡要更換 kube-vip image 名稱位置
$ alias kube-vip="sudo ctr -n k8s.io image pull --hosts-dir "/etc/containerd/certs.d" harbor.example.com/library/kube-vip:v0.9.2;sudo ctr -n k8s.io run --rm --net-host harbor.example.com/library/kube-vip:v0.9.2 vip /kube-vip"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 更換 `kube-vip.yaml` yaml 的 image 位置
```
$ sudo sed -i 's|ghcr.io/kube-vip/kube-vip:v0.9.2|harbor.example.com/library/kube-vip:v0.9.2|g' /etc/kubernetes/manifests/kube-vip.yaml
```
* 確認更改
```
$ cat /etc/kubernetes/manifests/kube-vip.yaml | grep image:
image: harbor.example.com/library/kube-vip:v0.9.2
```
* 在 m2 開始安裝 k8s
```
$ sudo kubeadm join 172.20.7.100:6443 --token ma5zm7.1rc9i2qjybt5uve5 \
--discovery-token-ca-cert-hash sha256:cefd410656e767c60f84b2e394c9837326ca03380cb6d07bad9a0fd03d971044 \
--control-plane --certificate-key 126e9fc8daeb3e304187383a4d00999ae2f6cd09e196b662d81755a66a2a0415
```
## 加入 m3 master node
#### 設定 kube-vip
* 在 m3 執行以下命令
```
$ cd ~/k8s
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.7.100
# 宣告網卡名稱
$ export INTERFACE=ens18
# 注意這裡要更換 kube-vip image 名稱位置
$ alias kube-vip="sudo ctr -n k8s.io image pull --hosts-dir "/etc/containerd/certs.d" harbor.example.com/library/kube-vip:v0.9.2;sudo ctr -n k8s.io run --rm --net-host harbor.example.com/library/kube-vip:v0.9.2 vip /kube-vip"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 更換 `kube-vip.yaml` yaml 的 image 位置
```
$ sudo sed -i 's|ghcr.io/kube-vip/kube-vip:v0.9.2|harbor.example.com/library/kube-vip:v0.9.2|g' /etc/kubernetes/manifests/kube-vip.yaml
```
* 確認更改
```
$ cat /etc/kubernetes/manifests/kube-vip.yaml | grep image:
image: harbor.example.com/library/kube-vip:v0.9.2
```
* 在 m3 開始安裝 k8s
```
$ sudo kubeadm join 172.20.7.100:6443 --token ma5zm7.1rc9i2qjybt5uve5 \
--discovery-token-ca-cert-hash sha256:cefd410656e767c60f84b2e394c9837326ca03380cb6d07bad9a0fd03d971044 \
--control-plane --certificate-key 126e9fc8daeb3e304187383a4d00999ae2f6cd09e196b662d81755a66a2a0415
```
## 加入 w1、w2 worker node
如果沒記錄到指令，可以在 m1 使用以下指令產出 `worker` 註冊指令
```
$ sudo kubeadm token create --print-join-command
```
```
$ sudo kubeadm join 172.20.7.100:6443 --token ma5zm7.1rc9i2qjybt5uve5 \
--discovery-token-ca-cert-hash sha256:cefd410656e767c60f84b2e394c9837326ca03380cb6d07bad9a0fd03d971044
```
w1、w2 貼上標籤，都是叫 worker
```
$ kubectl label node w1 node-role.kubernetes.io/worker=; kubectl label node w2 node-role.kubernetes.io/worker=
```
## 環境檢查
```
$ kubectl get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
m1 Ready control-plane 151m v1.33.2 172.20.7.90 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.1.3
m2 Ready control-plane 5m12s v1.33.2 172.20.7.91 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.1.3
m3 Ready control-plane 2m8s v1.33.2 172.20.7.92 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.1.3
w1 Ready worker 42s v1.33.2 172.20.7.93 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.1.3
w2 Ready worker 39s v1.33.2 172.20.7.94 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.1.3
```
## 安裝 metrics-server
更換 `metrics-server` yaml 的 image 位置
```
$ sudo sed -i 's|registry.k8s.io/metrics-server/metrics-server:v0.7.2|harbor.example.com/library/metrics-server:v0.7.2|g' components.yaml
```
```
$ kubectl apply -f components.yaml
```
驗證功能
```
$ kubectl -n kube-system get po -l k8s-app=metrics-server
NAME READY STATUS RESTARTS AGE
metrics-server-7c9dbfcd55-th6b2 1/1 Running 0 74s
$ kubectl top no
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
m1 50m 1% 1534Mi 17%
m2 54m 1% 890Mi 10%
m3 39m 0% 833Mi 9%
w1 14m 0% 480Mi 5%
w2 16m 0% 505Mi 5%
```
## 驗證 Apiserver HA 容錯
查看 `plndr-cp-lock` 目前是 m1 提供當 vip
```
$ kubectl -n kube-system get lease
NAME HOLDER AGE
apiserver-276wpfzo55z6xlhwmv7sebbpdm apiserver-276wpfzo55z6xlhwmv7sebbpdm_f501be5a-c4c4-474f-9879-df21f6a12370 152m
apiserver-adkfkrghmfikf7izpjatkklyyi apiserver-adkfkrghmfikf7izpjatkklyyi_bb70aa21-49bb-4cf6-b3c7-7f5060a1badc 6m3s
apiserver-ormoeicu2rhguypboutgrrqply apiserver-ormoeicu2rhguypboutgrrqply_e1755620-b109-4212-9d1d-56e98ef38c01 3m3s
kube-controller-manager m1_dfb51065-5982-421d-86f6-20284b80a1b2 152m
kube-scheduler m1_e1eb91aa-d316-4dc5-8e53-91ce3316847e 152m
plndr-cp-lock m1 152m
```
將 m1 關機
```
bigred@m1:~$ sudo poweroff
```
在外部管理主機還是可以控制 k8s，並且確認 vip 轉移到 m2 上
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
m1 NotReady control-plane 154m v1.33.2
m2 Ready control-plane 7m25s v1.33.2
m3 Ready control-plane 4m21s v1.33.2
w1 Ready worker 2m55s v1.33.2
w2 Ready worker 2m52s v1.33.2
$ kubectl -n kube-system get lease
NAME HOLDER AGE
apiserver-276wpfzo55z6xlhwmv7sebbpdm apiserver-276wpfzo55z6xlhwmv7sebbpdm_f501be5a-c4c4-474f-9879-df21f6a12370 154m
apiserver-adkfkrghmfikf7izpjatkklyyi apiserver-adkfkrghmfikf7izpjatkklyyi_bb70aa21-49bb-4cf6-b3c7-7f5060a1badc 7m51s
apiserver-ormoeicu2rhguypboutgrrqply apiserver-ormoeicu2rhguypboutgrrqply_e1755620-b109-4212-9d1d-56e98ef38c01 4m51s
kube-controller-manager m2_70fa2776-6126-4884-b0b9-e29e4e180ed2 154m
kube-scheduler m3_f814c532-52b9-48bb-bbc5-33048cae7070 154m
plndr-cp-lock m2 154m
```
Sign in with Wallet
Connect another wallet