# Kubeadm 在 ubuntu server（24.04） 安裝 k8s 3m1w HA 架構 (v1.33) 使用 CRI-O
## 建置 HA k8s 前的設定動作
### 下載 podman (每一個節點都需安裝)
```
$ curl -fsSL -o podman-linux-amd64.tar.gz https://github.com/mgoltzsche/podman-static/releases/latest/download/podman-linux-amd64.tar.gz
$ tar -zxvf podman-linux-amd64.tar.gz;cd podman-linux-amd64
```
* 使用 rsync 複製檔案，並且會把原本會被覆寫的檔案移到備份目錄
```
$ sudo rsync -aHAX --numeric-ids --info=progress2 \
--backup --backup-dir="/root/usr-backup-$(date +%F_%H%M%S)" \
./usr/ /usr/
```
```
$ sudo mkdir -p /etc/containers
$ sudo rsync -aHAX --no-o --no-g --info=progress2 \
--backup --backup-dir="/root/etc-containers-backup-$(date +%F_%H%M%S)}" \
./etc/containers/ /etc/containers/
```
```
$ sudo podman version
Client: Podman Engine
Version: 5.6.0
API Version: 5.6.0
Go Version: go1.24.6
Built: Thu Jan 1 08:00:00 1970
OS/Arch: linux/amd64
```
* 如果 podman 需要使用 rootless ，還要安裝以下套件
```
$ sudo apt install -y uidmap
```
### 下載與安裝 CRI-O (每一個節點都需安裝)
```
$ K8SVERSION=v1.33.4 # 更改指定 k8s 版本
$ wget https://storage.googleapis.com/cri-o/artifacts/cri-o.amd64.${K8SVERSION}.tar.gz
$ tar -zxvf cri-o.amd64.${K8SVERSION}.tar.gz; cd cri-o/
$ sudo cp bin/* /usr/local/bin/
$ sudo cp contrib/crio.service /etc/systemd/system/crio.service
$ sudo systemctl daemon-reload && sudo systemctl enable --now crio
```
```
$ systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
Loaded: loaded (/etc/systemd/system/crio.service; enabled; preset: enabled)
Active: active (running) since Fri 2025-09-05 14:14:19 CST; 5s ago
```
* 已安裝 crun
```
$ crun -v
crun version 1.22
commit: 4de19b63a85efd9ea8503452179c371181750130
rundir: /run/user/1001/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
```
* 使用 podman 登入 Harbor，並將產生的認證 token 儲存到 CRI-O 指定的 auth.json 檔案中，這樣可以透過這個 account pull 不公開的 image(可選，這是每個節點都要產生)
```
$ sudo podman login harbor.k8sexample.com --authfile /etc/crio/registries.d/auth.json
$ sudo cat /etc/crio/registries.d/auth.json
{
"auths": {
"harbor.k8sexample.com": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}
```
* 設定 `crio.conf` 配置檔
```
$ sudo nano /etc/crio/crio.conf
[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "systemd"
default_runtime = "crun"
default_capabilities = [
"CHOWN",
"DAC_OVERRIDE",
"FSETID",
"FOWNER",
"SETGID",
"SETUID",
"SETPCAP",
"NET_BIND_SERVICE",
"AUDIT_WRITE",
"SYS_CHROOT",
"KILL"
]
[crio.runtime.runtimes.crun]
runtime_path = "/usr/local/bin/crun"
runtime_type = "oci"
runtime_root = ""
[crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dir = "/opt/cni/bin"
[crio.image]
pause_image="registry.k8s.io/pause:3.10"
```
```
$ sudo nano /etc/crictl.yaml
runtime-endpoint: unix:///var/run/crio/crio.sock
image-endpoint: unix:///var/run/crio/crio.sock
timeout: 2
```
* 設定內部 image registry 憑證 insecure
```
$ sudo mkdir -p /etc/containers/registries.conf.d/
$ sudo nano /etc/containers/registries.conf.d/insecure-registry.conf
[[registry]]
prefix = "harbor.example.com"
location = "harbor.example.com"
insecure = true
```
```
$ sudo systemctl daemon-reload
$ sudo systemctl restart crio.service
```
### control plane 節點安裝 kubelet kubeadm kubectl
* control plane 節點下載套件
```
$ cd ~
$ wget https://dl.k8s.io/${K8SVERSION}/kubernetes-server-linux-amd64.tar.gz
$ tar -xzvf kubernetes-server-linux-amd64.tar.gz
$ cd kubernetes/server/bin && sudo cp kubeadm kubelet kubectl /usr/bin/
```
#### 設定 kubelet service
* cri 設定為 crio
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=crio.service
Requires=crio.service
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
$ sudo mkdir /etc/systemd/system/kubelet.service.d
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
* 設定節點 internal ip
```
$ sudo mkdir -p /etc/default
$ INTERFACE=ens18;IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)
$ echo "KUBELET_EXTRA_ARGS=\"--node-ip=$IPV4_IP\"" | envsubst | sudo tee /etc/default/kubelet
```
```
$ sudo systemctl enable --now kubelet.service
```
### worker 節點安裝 kubelet kubeadm
* Worker 下載套件
```
$ cd ~
$ wget https://dl.k8s.io/${K8SVERSION}/kubernetes-node-linux-amd64.tar.gz
$ tar -xzvf kubernetes-node-linux-amd64.tar.gz
$ cd kubernetes/node/bin && sudo cp kubeadm kubelet /usr/bin/
```
#### 設定 kubelet service
* cri 設定為 crio
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=crio.service
Requires=crio.service
[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
$ sudo mkdir /etc/systemd/system/kubelet.service.d
$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
* 設定節點 internal ip
```
$ sudo mkdir -p /etc/default
$ INTERFACE=ens18;IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)
$ echo "KUBELET_EXTRA_ARGS=\"--node-ip=$IPV4_IP\"" | envsubst | sudo tee /etc/default/kubelet
```
```
$ sudo systemctl enable --now kubelet.service
```
## 透過 kubeadm 建立 HA k8s
### 設定 m1
#### 設定 kube-vip
* 此時 k8s 還沒有建置，先產生 `kube-vip` yaml 檔
```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9
# 宣告網卡名稱
$ export INTERFACE=ens19
# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1
$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
# 1.29 以後需要調整權限，只有第一台 master 需要修改
$ sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
```
* `advertiseAddress` 需更換為自己的 master ip
* `controlPlaneEndpoint` 要指定 vip 的位置
```
$ nano init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 172.20.2.10 # change from Master node IP
bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/crio/crio.sock # change from crio Unix Socket
imagePullPolicy: IfNotPresent
name: crio-m1 # change from Master node hsotname
taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.33.4
controlPlaneEndpoint: 172.20.2.9:6443 # kube-vip 位置
apiServer:
timeoutForControlPlane: 4m0s
certSANs:
- 127.0.0.1
certificatesDir: /etc/kubernetes/pki
clusterName: topgun # set your clusterName
controllerManager:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10257"
extraVolumes:
- name: tz-config
hostPath: /etc/localtime
mountPath: /etc/localtime
readOnly: true
scheduler:
extraArgs:
bind-address: "0.0.0.0"
secure-port: "10259"
etcd:
local:
dataDir: /var/lib/etcd
extraArgs:
listen-metrics-urls: "http://0.0.0.0:2381"
dns: {}
imageRepository: registry.k8s.io
networking:
dnsDomain: cluster.local # DNS domain used by Kubernetes Services.
podSubnet: 10.244.0.0/16 # the subnet used by Pods.
serviceSubnet: 10.96.0.0/16 # subnet used by Kubernetes Services.
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: "0.0.0.0:10249"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
imageMinimumGCAge: "2m0s" # 至少要保留 image 2 分鐘
imageMaximumGCAge: "168h" # 設為 1 週 (168 hours) → 超過 1 週未使用的 image 可回收
systemReserved:
memory: "1Gi"
kubeReserved:
memory: "2Gi"
```
開始安裝
* `--upload-certs` 將 `control-plane` 節點所需的金鑰和憑證上傳到 `kubeadm-certs Secret` 中，以供其他 `control-plane` 節點下載使用。
```
$ sudo kubeadm init --upload-certs --config=init-config.yaml
```
輸出結果並記錄註冊指令:
```
......
You can now join any number of control-plane nodes running the following command on each as root:
kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c
Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f
```
* 設定 kubeconfig
```
$ mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
$ echo -e "source /usr/share/bash-completion/bash_completion\nsource <(kubectl completion bash)" | sudo tee -a /etc/profile
```
### 部屬 calico 3.29.4
```
$ curl -sL https://raw.githubusercontent.com/projectcalico/calico/v3.29.4/manifests/calico.yaml | kubectl apply -f -
```
### 環境檢查
* k8s 安裝好後再將 kube-vip 權限調整回來
```
$ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
$ sudo systemctl daemon-reload
$ sudo systemctl restart kubelet
```
```
$ kubectl get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-cbd56df6f-n6ltq 1/1 Running 0 7m53s
kube-system calico-node-kklbf 1/1 Running 0 7m53s
kube-system coredns-674b8bbfcf-8844m 1/1 Running 0 10m
kube-system coredns-674b8bbfcf-dzww6 1/1 Running 0 10m
kube-system etcd-crio-m1 1/1 Running 0 10m
kube-system kube-apiserver-crio-m1 1/1 Running 0 10m
kube-system kube-controller-manager-crio-m1 1/1 Running 0 10m
kube-system kube-proxy-jg7z6 1/1 Running 0 10m
kube-system kube-scheduler-crio-m1 1/1 Running 0 10m
kube-system kube-vip-crio-m1 1/1 Running 0 10m
```
### 加入 m2 master node
#### 設定 kube-vip
```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9
# 宣告網卡名稱
$ export INTERFACE=ens19
# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1
$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 在 m2 開始安裝 k8s
```
$ sudo kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c
```
### 加入 m3 master node
#### 設定 kube-vip
```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9
# 宣告網卡名稱
$ export INTERFACE=ens19
# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1
$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"
$ sudo mkdir -p /etc/kubernetes/manifests/
$ kube-vip manifest pod \
--address $VIP \
--interface $INTERFACE \
--controlplane \
--arp \
--leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 在 m3 開始安裝 k8s
```
$ sudo kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c
```
### 加入 worker node
如果沒記錄到指令，可以在 m1 使用以下指令產出 `worker` 註冊指令
```
$ sudo kubeadm token create --print-join-command
```
worker 貼上標籤
```
$ kubectl label node crio-w1 node-role.kubernetes.io/worker=
```
## 環境檢查
```
$ kubectl get no -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
crio-m1 Ready control-plane 20m v1.33.4 172.20.2.10 <none> Ubuntu 24.04.2 LTS 6.8.0-59-generic cri-o://1.33.4
crio-m2 Ready control-plane 5m49s v1.33.4 172.20.2.11 <none> Ubuntu 24.04.2 LTS 6.8.0-59-generic cri-o://1.33.4
crio-m3 Ready control-plane 4m26s v1.33.4 172.20.2.12 <none> Ubuntu 24.04.2 LTS 6.8.0-59-generic cri-o://1.33.4
crio-w1 Ready worker 3m16s v1.33.4 172.20.2.15 <none> Ubuntu 24.04.2 LTS 6.8.0-59-generic cri-o://1.33.4
```
Sign in with Wallet
Connect another wallet