---
# System prepended metadata

title: Kubeadm 在 ubuntu server（24.04）安裝 k8s 3m1w  HA 架構 (v1.33) 使用 CRI-O
tags: [K8S]

---

# Kubeadm 在 ubuntu server（24.04） 安裝 k8s 3m1w HA 架構 (v1.33) 使用 CRI-O

## 建置 HA k8s 前的設定動作
### 下載 podman (每一個節點都需安裝)

```
$ curl -fsSL -o podman-linux-amd64.tar.gz https://github.com/mgoltzsche/podman-static/releases/latest/download/podman-linux-amd64.tar.gz

$ tar -zxvf podman-linux-amd64.tar.gz;cd podman-linux-amd64
```
* 使用 rsync 複製檔案，並且會把原本會被覆寫的檔案移到備份目錄
```
$ sudo rsync -aHAX --numeric-ids --info=progress2 \
  --backup --backup-dir="/root/usr-backup-$(date +%F_%H%M%S)" \
  ./usr/ /usr/
```

```
$ sudo mkdir -p /etc/containers

$ sudo rsync -aHAX --no-o --no-g --info=progress2 \
  --backup --backup-dir="/root/etc-containers-backup-$(date +%F_%H%M%S)}" \
  ./etc/containers/ /etc/containers/
```

```
$ sudo podman version
Client:       Podman Engine
Version:      5.6.0
API Version:  5.6.0
Go Version:   go1.24.6
Built:        Thu Jan  1 08:00:00 1970
OS/Arch:      linux/amd64
```

* 如果 podman 需要使用 rootless ，還要安裝以下套件
```
$ sudo apt install -y uidmap
```

### 下載與安裝 CRI-O (每一個節點都需安裝)

```
$ K8SVERSION=v1.33.4     # 更改指定 k8s 版本

$ wget https://storage.googleapis.com/cri-o/artifacts/cri-o.amd64.${K8SVERSION}.tar.gz

$ tar -zxvf cri-o.amd64.${K8SVERSION}.tar.gz; cd cri-o/

$ sudo cp bin/* /usr/local/bin/

$ sudo cp contrib/crio.service /etc/systemd/system/crio.service

$ sudo systemctl daemon-reload && sudo systemctl enable --now crio
```

```
$ systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
     Loaded: loaded (/etc/systemd/system/crio.service; enabled; preset: enabled)
     Active: active (running) since Fri 2025-09-05 14:14:19 CST; 5s ago
```
* 已安裝 crun
```
$ crun -v
crun version 1.22
commit: 4de19b63a85efd9ea8503452179c371181750130
rundir: /run/user/1001/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
```
* 使用 podman 登入 Harbor，並將產生的認證 token 儲存到 CRI-O 指定的 auth.json 檔案中，這樣可以透過這個 account pull 不公開的 image(可選，這是每個節點都要產生)
```
$ sudo podman login harbor.k8sexample.com --authfile /etc/crio/registries.d/auth.json

$ sudo cat /etc/crio/registries.d/auth.json
{
	"auths": {
		"harbor.k8sexample.com": {
			"auth": "YWRtaW46SGFyYm9yMTIzNDU="
		}
	}
}
```


* 設定 `crio.conf` 配置檔
```
$ sudo nano /etc/crio/crio.conf
[crio.runtime]
conmon_cgroup = "pod"
cgroup_manager = "systemd"
default_runtime = "crun"

default_capabilities = [
  "CHOWN",
  "DAC_OVERRIDE",
  "FSETID",
  "FOWNER",
  "SETGID",
  "SETUID",
  "SETPCAP",
  "NET_BIND_SERVICE",
  "AUDIT_WRITE",
  "SYS_CHROOT",
  "KILL"
]

[crio.runtime.runtimes.crun]
runtime_path = "/usr/local/bin/crun"
runtime_type = "oci"
runtime_root = ""

[crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dir = "/opt/cni/bin"

[crio.image]
pause_image="registry.k8s.io/pause:3.10"
```

```
$ sudo nano /etc/crictl.yaml 
runtime-endpoint: unix:///var/run/crio/crio.sock
image-endpoint: unix:///var/run/crio/crio.sock
timeout: 2
```
* 設定內部 image registry 憑證 insecure
```
$ sudo mkdir -p /etc/containers/registries.conf.d/

$ sudo nano /etc/containers/registries.conf.d/insecure-registry.conf
[[registry]]
prefix = "harbor.example.com"
location = "harbor.example.com"
insecure = true
```
```
$ sudo systemctl daemon-reload

$ sudo systemctl restart crio.service
```

### control plane 節點安裝 kubelet kubeadm kubectl


* control plane 節點下載套件
```
$ cd ~

$ wget https://dl.k8s.io/${K8SVERSION}/kubernetes-server-linux-amd64.tar.gz

$ tar -xzvf kubernetes-server-linux-amd64.tar.gz

$ cd kubernetes/server/bin && sudo cp kubeadm kubelet kubectl /usr/bin/
```
#### 設定 kubelet service

* cri 設定為 crio
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=crio.service
Requires=crio.service

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

$ sudo mkdir /etc/systemd/system/kubelet.service.d

$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
* 設定節點 internal ip
```
$ sudo mkdir -p /etc/default

$ INTERFACE=ens18;IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)

$ echo "KUBELET_EXTRA_ARGS=\"--node-ip=$IPV4_IP\"" | envsubst | sudo tee /etc/default/kubelet
```
```
$ sudo systemctl enable --now kubelet.service
```
### worker 節點安裝 kubelet kubeadm


* Worker 下載套件
```
$ cd ~

$ wget https://dl.k8s.io/${K8SVERSION}/kubernetes-node-linux-amd64.tar.gz

$ tar -xzvf kubernetes-node-linux-amd64.tar.gz

$ cd kubernetes/node/bin && sudo cp kubeadm kubelet /usr/bin/
```

#### 設定 kubelet service

* cri 設定為 crio
```
$ cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/
After=crio.service
Requires=crio.service

[Service]
ExecStart=/usr/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

$ sudo mkdir /etc/systemd/system/kubelet.service.d

$ cat << EOF | sudo tee /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating
# the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet \$KUBELET_KUBECONFIG_ARGS \$KUBELET_CONFIG_ARGS \$KUBELET_KUBEADM_ARGS \$KUBELET_EXTRA_ARGS
EOF
```
* 設定節點 internal ip
```
$ sudo mkdir -p /etc/default

$ INTERFACE=ens18;IPV4_IP=$(ip -4 a s $INTERFACE | awk '/inet / {print $2}' | cut -d'/' -f1)

$ echo "KUBELET_EXTRA_ARGS=\"--node-ip=$IPV4_IP\"" | envsubst | sudo tee /etc/default/kubelet
```
```
$ sudo systemctl enable --now kubelet.service
```
## 透過 kubeadm 建立 HA k8s

### 設定 m1

#### 設定 kube-vip
* 此時 k8s 還沒有建置，先產生 `kube-vip` yaml 檔
```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9

# 宣告網卡名稱
$ export INTERFACE=ens19

# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")

# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1

$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"

$ sudo mkdir -p /etc/kubernetes/manifests/

$ kube-vip manifest pod \
    --address $VIP \
    --interface $INTERFACE \
    --controlplane \
    --arp \
    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml

# 1.29 以後需要調整權限，只有第一台 master 需要修改
$ sudo sed -i 's|path: /etc/kubernetes/admin.conf|path: /etc/kubernetes/super-admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml
```

* `advertiseAddress` 需更換為自己的 master ip
* `controlPlaneEndpoint` 要指定 vip 的位置
```
$ nano init-config.yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.20.2.10                                 # change from Master node IP
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/crio/crio.sock           # change from crio Unix Socket
  imagePullPolicy: IfNotPresent
  name: crio-m1                                                     # change from Master node hsotname
  taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: 1.33.4
controlPlaneEndpoint: 172.20.2.9:6443                   # kube-vip 位置
apiServer:
  timeoutForControlPlane: 4m0s
  certSANs:
   - 127.0.0.1
certificatesDir: /etc/kubernetes/pki
clusterName: topgun                                        # set your clusterName
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"
    secure-port: "10257"
  extraVolumes:
    - name: tz-config
      hostPath: /etc/localtime
      mountPath: /etc/localtime
      readOnly: true
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"
    secure-port: "10259"
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      listen-metrics-urls: "http://0.0.0.0:2381"
dns: {}
imageRepository: registry.k8s.io
networking:
  dnsDomain: cluster.local                                        # DNS domain used by Kubernetes Services.
  podSubnet: 10.244.0.0/16                              # the subnet used by Pods.
  serviceSubnet: 10.96.0.0/16                           # subnet used by Kubernetes Services.
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
metricsBindAddress: "0.0.0.0:10249"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
shutdownGracePeriod: 30s
shutdownGracePeriodCriticalPods: 10s
imageMinimumGCAge: "2m0s"         # 至少要保留 image 2 分鐘
imageMaximumGCAge: "168h"         # 設為 1 週 (168 hours) → 超過 1 週未使用的 image 可回收
systemReserved:
  memory: "1Gi"
kubeReserved:
  memory: "2Gi"
```
開始安裝

* `--upload-certs` 將 `control-plane` 節點所需的金鑰和憑證上傳到 `kubeadm-certs Secret` 中，以供其他 `control-plane` 節點下載使用。
```
$ sudo kubeadm init --upload-certs --config=init-config.yaml
```
輸出結果並記錄註冊指令:

```
......
You can now join any number of control-plane nodes running the following command on each as root:

  kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
	--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
	--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
	--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f
```

* 設定 kubeconfig 
```
$ mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config

$ echo -e "source /usr/share/bash-completion/bash_completion\nsource <(kubectl completion bash)" | sudo tee -a /etc/profile
```


### 部屬 calico 3.29.4

```
$ curl -sL https://raw.githubusercontent.com/projectcalico/calico/v3.29.4/manifests/calico.yaml | kubectl apply -f -
```


### 環境檢查

* k8s 安裝好後再將 kube-vip 權限調整回來
```
$ sudo sed -i 's|path: /etc/kubernetes/super-admin.conf|path: /etc/kubernetes/admin.conf|g' /etc/kubernetes/manifests/kube-vip.yaml

$ sudo systemctl daemon-reload

$ sudo systemctl restart kubelet
```
```
$ kubectl get po -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-cbd56df6f-n6ltq   1/1     Running   0          7m53s
kube-system   calico-node-kklbf                         1/1     Running   0          7m53s
kube-system   coredns-674b8bbfcf-8844m                  1/1     Running   0          10m
kube-system   coredns-674b8bbfcf-dzww6                  1/1     Running   0          10m
kube-system   etcd-crio-m1                              1/1     Running   0          10m
kube-system   kube-apiserver-crio-m1                    1/1     Running   0          10m
kube-system   kube-controller-manager-crio-m1           1/1     Running   0          10m
kube-system   kube-proxy-jg7z6                          1/1     Running   0          10m
kube-system   kube-scheduler-crio-m1                    1/1     Running   0          10m
kube-system   kube-vip-crio-m1                          1/1     Running   0          10m
```
### 加入 m2 master node

#### 設定 kube-vip

```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9

# 宣告網卡名稱
$ export INTERFACE=ens19

# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")

# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1

$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"

$ sudo mkdir -p /etc/kubernetes/manifests/

$ kube-vip manifest pod \
    --address $VIP \
    --interface $INTERFACE \
    --controlplane \
    --arp \
    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 在 m2 開始安裝 k8s
```
$ sudo kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
	--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
	--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c
```


### 加入 m3 master node

#### 設定 kube-vip

```
# 安裝 kube-vip Set configuration details
# IP 要改成叢集外的新 IP
$ export VIP=172.20.2.9

# 宣告網卡名稱
$ export INTERFACE=ens19

# 取得 kube-vip 版本代號
$ export KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")

# 檢查 kube-vip 版本
$ echo $KVVERSION
v0.9.1

$ alias kube-vip="sudo podman run --network host --rm ghcr.io/kube-vip/kube-vip:$KVVERSION"

$ sudo mkdir -p /etc/kubernetes/manifests/

$ kube-vip manifest pod \
    --address $VIP \
    --interface $INTERFACE \
    --controlplane \
    --arp \
    --leaderElection | sudo tee /etc/kubernetes/manifests/kube-vip.yaml
```
* 在 m3 開始安裝 k8s
```
$ sudo kubeadm join 172.20.2.9:6443 --token qcz23n.itof4ltm3bca48pa \
	--discovery-token-ca-cert-hash sha256:0ce6e36d6a6a4f9ae3f25ed5e69051b4fa61e94f2ad3bb77b06dbda30220e33f \
	--control-plane --certificate-key 26589356b1554c14c916570e2eb6de69dd0cdaa4b38ed2a758c2d1a26b56ad4c
```

### 加入 worker node

如果沒記錄到指令，可以在 m1 使用以下指令產出 `worker` 註冊指令
```
$ sudo kubeadm token create --print-join-command
```
worker 貼上標籤
```
$ kubectl label node crio-w1 node-role.kubernetes.io/worker=
```

## 環境檢查
```
$ kubectl get no -owide
NAME      STATUS   ROLES           AGE     VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
crio-m1   Ready    control-plane   20m     v1.33.4   172.20.2.10   <none>        Ubuntu 24.04.2 LTS   6.8.0-59-generic   cri-o://1.33.4
crio-m2   Ready    control-plane   5m49s   v1.33.4   172.20.2.11   <none>        Ubuntu 24.04.2 LTS   6.8.0-59-generic   cri-o://1.33.4
crio-m3   Ready    control-plane   4m26s   v1.33.4   172.20.2.12   <none>        Ubuntu 24.04.2 LTS   6.8.0-59-generic   cri-o://1.33.4
crio-w1   Ready    worker          3m16s   v1.33.4   172.20.2.15   <none>        Ubuntu 24.04.2 LTS   6.8.0-59-generic   cri-o://1.33.4
```