kubespray - HackMD

# kubespray ## 什麼是 kubespray kubespray 它是一種基於 Ansible 的 IaC（Infrastructure as Code）方案，支援在裸機、雲端或虛擬機環境中建立 K8s 叢集。 ![image](https://hackmd.io/_uploads/S1irD3mSxx.png) ## 應用背景透過 kubeadm 的方式部署事前需準備很多設定，也會有很多情況導致 k8s 設定不一致（例如節點之間的 oci,cri 版本不一致），而透過 Kubespray 自動化安裝，實務上來說安裝 k8s 會較穩定，也比較方便 ## 事前環境準備目標做出 3m1w 架構的 k8s，需先準備好 vm 相關設定 1. 準備一台遠端管理主機，負責執行 kubespray 來建立 k8s(類似 ocp 的 Bastion 主機) 2. 所有節點的網路路之間可以互相通訊 3. 遠端管理主機對其他節點不需要 SSH 密碼即可登入 4. 所有節點指定的使用者都擁有 sudo 權限，且不需要輸入密碼 5. 所有節點需要安裝 Python 6. 所有節點需要設定 `/etc/hosts` 解析到所有主機 k8s node vm 相關資訊： > OS: ubuntu server 24.04 > user: bigred > vip: 172.20.7.90 > master1: 172.20.7.91 > master2: 172.20.7.92 > master3: 172.20.7.93 > worker1: 172.20.7.94 ### vm 事前設定 #### 遠端管理主機設定 * 在遠端管理主機上產生 ssh 私鑰，並將公鑰丟到 `authorized_keys` ，然後再把 ssh 目錄 scp 到每一台節點上，這樣每個節點之間就可以免密碼 ssh ``` $ ssh-keygen -t rsa -P "" $ cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys $ scp -r ~/.ssh/ 172.20.7.91:~/ $ scp -r ~/.ssh/ 172.20.7.92:~/ $ scp -r ~/.ssh/ 172.20.7.93:~/ $ scp -r ~/.ssh/ 172.20.7.94:~/ ``` #### k8s node 事前設定 * 所有節點需要設定 `/etc/hosts` 解析到所有主機 ``` $ sudo nano /etc/hosts ...... 172.20.7.91 master1 172.20.7.92 master2 172.20.7.93 master3 172.20.7.94 worker1 ``` * 確認每個節點都安裝 python ``` $ python3 --version Python 3.12.3 ``` ### 設定台灣時區 ``` $ sudo timedatectl set-timezone Asia/Taipei ``` ``` $ timedatectl Local time: Tue 2025-06-10 17:53:24 CST Universal time: Tue 2025-06-10 09:53:24 UTC RTC time: Tue 2025-06-10 09:53:24 Time zone: Asia/Taipei (CST, +0800) System clock synchronized: yes NTP service: active RTC in local TZ: no ``` ### 設定校時 ``` $ sudo apt -y install chrony $ sudo systemctl enable chrony # 設定使用 google ntp $ sudo nano /etc/chrony/chrony.conf ...... #pool ntp.ubuntu.com iburst maxsources 4 #pool 0.ubuntu.pool.ntp.org iburst maxsources 1 #pool 1.ubuntu.pool.ntp.org iburst maxsources 1 #pool 2.ubuntu.pool.ntp.org iburst maxsources 2 pool time.google.com iburst maxsources 4 $ sudo systemctl daemon-reload $ sudo systemctl restart chrony ``` ``` $ chronyc sources MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^? time2.google.com 1 6 1 51 +70us[ +70us] +/- 3425us ``` ### 關閉 SWAP 交換分區 ``` $ sudo swapoff -a $ sudo sed -i '/swap/s/^/#/' /etc/fstab ``` ### 設定 kernel modules ``` $ sudo apt install -y ipvsadm ipset $ cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf overlay br_netfilter ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 EOF $ sudo cat <<EOF |sudo tee /etc/sysctl.d/99-kubernetes-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF $ sudo sysctl --system $ sudo reboot ``` ## 安裝 Kubespray 以下操作都在遠端管理主機執行 * 下載 Kubespray 專案 ``` $ git clone https://github.com/kubernetes-sigs/kubespray.git;cd kubespray ``` * 遠端管理主機安裝 podman 或 docker 啟動 kubespray container，並進到 container 部署 ``` $ sudo podman run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \ --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \ quay.io/kubespray/kubespray:v2.28.0 bash ``` * 修改 `inventory.ini` 節點清單設定檔，設定 k8s 節點相關資訊 - `node_taints` 拿掉 master 上預設的 taint。 - `node_labels` 幫 worker 貼上標籤，但這裡會把 `k8s-cluster.yml` 添加的 label 覆蓋，所以在這裡要指定。 ``` $ vim /inventory/inventory.ini [all] master1 ansible_ssh_host=172.20.7.91 node_taints="['node-role.kubernetes.io/control-plane:NoSchedule-']" master2 ansible_ssh_host=172.20.7.92 node_taints="['node-role.kubernetes.io/control-plane:NoSchedule-']" master3 ansible_ssh_host=172.20.7.93 node_taints="['node-role.kubernetes.io/control-plane:NoSchedule-']" worker1 ansible_ssh_host=172.20.7.94 node_labels="{'node-role.kubernetes.io/worker':'', 'my':'test'}" [kube_control_plane] master1 master2 master3 [etcd] master1 master2 master3 [kube_node] worker1 [k8s-cluster:children] kube_control_plane kube_node ``` * 編輯 `k8s-cluster.yml` 設定 - k8s calico CNI - pod,service CIDR - 叢集名稱 - 設定 CRI(預設就是 containerd) - 啟用 ipvs ``` $ vim /inventory/group_vars/k8s_cluster/k8s-cluster.yml kube_version: 1.32.2 # 在每個節點貼上自己要的 label node_labels: my: test ...... kube_network_plugin: calico ...... kube_service_addresses: 10.98.0.0/16 ...... kube_pods_subnet: 10.244.0.0/16 ...... kube_proxy_mode: ipvs ...... kube_proxy_strict_arp: true ...... cluster_name: cluster.local ...... ## Default: containerd container_manager: containerd ``` * 編輯 `addons.yml` 設定要在 k8s 上安裝哪些應用，以下設定 - 啟用 `metrics_server` - 啟用 `local_path_provisioner` - 啟用 `ingress_nginx` - 啟用 `metallb`，需注意分配自己規劃的網段 - 啟用 `kube_vip`，設定自己的 vip，與自己的網卡名稱 ``` $ vim /inventory/group_vars/k8s_cluster/addons.yml ...... metrics_server_enabled: true ...... local_path_provisioner_enabled: true local_path_provisioner_namespace: "local-path-storage" local_path_provisioner_storage_class: "local-path" local_path_provisioner_reclaim_policy: Delete local_path_provisioner_claim_root: /opt/local-path-provisioner/ ...... ingress_nginx_enabled: true ...... metallb_enabled: true metallb_speaker_enabled: "{{ metallb_enabled }}" metallb_namespace: "metallb-system" metallb_config: speaker: nodeselector: kubernetes.io/os: "linux" tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" value: "" effect: "NoSchedule" controller: nodeselector: kubernetes.io/os: "linux" tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Equal" value: "" effect: "NoSchedule" address_pools: primary: ip_range: - 172.20.7.100-172.20.7.110 auto_assign: true ...... kube_vip_enabled: true kube_vip_arp_enabled: true kube_vip_controlplane_enabled: true kube_vip_address: 172.20.7.90 loadbalancer_apiserver: address: "{{ kube_vip_address }}" port: 6443 kube_vip_interface: ens18 ``` ### 開始部署 k8s 透過 ansible 部署 k8s，`-e ansible_user=bigred --become --become-user=root` 這邊宣告使用 bigred 部署，並且 bigred 使用者可以使用 sudo ``` $ ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa ./cluster.yml -e ansible_user=bigred --become --become-user=root ``` 部署完成螢幕輸出： ``` ...... PLAY RECAP *************************************************************************************************** master1 : ok=719 changed=153 unreachable=0 failed=0 skipped=1017 rescued=0 ignored=6 master2 : ok=614 changed=132 unreachable=0 failed=0 skipped=939 rescued=0 ignored=3 master3 : ok=616 changed=133 unreachable=0 failed=0 skipped=937 rescued=0 ignored=3 worker1 : ok=444 changed=83 unreachable=0 failed=0 skipped=636 rescued=0 ignored=1 Friday 04 July 2025 05:51:31 +0000 (0:00:00.055) 0:10:11.707 *********** =============================================================================== kubernetes-apps/metallb : Kubernetes Apps | Wait for MetalLB controller to be running ---------------- 26.12s download : Download_file | Download item ------------------------------------------------------------- 23.95s kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try) ------------------- 22.87s kubernetes/control-plane : Joining control plane node to the cluster. -------------------------------- 18.72s download : Download_container | Download image if required ------------------------------------------- 15.98s kubernetes/kubeadm : Join to cluster if needed ------------------------------------------------------- 15.98s download : Download_container | Download image if required ------------------------------------------- 15.39s kubernetes-apps/ansible : Kubernetes Apps | CoreDNS -------------------------------------------------- 14.69s download : Download_file | Download item ------------------------------------------------------------- 12.78s download : Download_container | Download image if required ------------------------------------------- 12.02s download : Download_container | Download image if required ------------------------------------------- 10.13s download : Download_container | Download image if required -------------------------------------------- 9.94s download : Download_container | Download image if required -------------------------------------------- 9.21s download : Download_file | Download item -------------------------------------------------------------- 8.82s system_packages : Update package management cache (APT) ----------------------------------------------- 7.93s download : Download_container | Download image if required -------------------------------------------- 7.87s download : Download_container | Download image if required -------------------------------------------- 7.30s download : Download_file | Download item -------------------------------------------------------------- 6.96s container-engine/containerd : Download_file | Download item ------------------------------------------- 6.55s download : Download_container | Download image if required -------------------------------------------- 6.24s ``` ## 環境檢查 * 到 master 節點上設定 kubeconfig ``` $ mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config ``` * 確認安裝相關應用 ``` $ kubectl get no -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME master1 Ready control-plane 8m4s v1.32.2 172.20.7.91 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.0.5 master2 Ready control-plane 7m51s v1.32.2 172.20.7.92 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.0.5 master3 Ready control-plane 7m46s v1.32.2 172.20.7.93 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.0.5 worker1 Ready worker 7m15s v1.32.2 172.20.7.94 <none> Ubuntu 24.04.2 LTS 6.8.0-60-generic containerd://2.0.5 $ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx ingress-nginx-controller-9jzv9 1/1 Running 0 6m40s ingress-nginx ingress-nginx-controller-l4lt7 1/1 Running 0 6m40s ingress-nginx ingress-nginx-controller-qw52v 1/1 Running 0 6m40s ingress-nginx ingress-nginx-controller-qzdd5 1/1 Running 0 6m40s kube-system calico-kube-controllers-588d6df6c9-qzqr6 1/1 Running 0 6m48s kube-system calico-node-4xfms 1/1 Running 0 6m59s kube-system calico-node-9wbmh 1/1 Running 0 6m59s kube-system calico-node-fn9hn 1/1 Running 0 6m59s kube-system calico-node-sgp99 1/1 Running 0 6m59s kube-system coredns-5c54f84c97-mtxsb 1/1 Running 0 6m33s kube-system coredns-5c54f84c97-zkkwh 1/1 Running 0 6m26s kube-system dns-autoscaler-56cb45595c-jsxmq 1/1 Running 0 6m32s kube-system kube-apiserver-master1 1/1 Running 0 8m19s kube-system kube-apiserver-master2 1/1 Running 0 8m5s kube-system kube-apiserver-master3 1/1 Running 0 8m2s kube-system kube-controller-manager-master1 1/1 Running 1 8m19s kube-system kube-controller-manager-master2 1/1 Running 1 8m7s kube-system kube-controller-manager-master3 1/1 Running 1 8m2s kube-system kube-proxy-7z79s 1/1 Running 0 8m2s kube-system kube-proxy-b4kts 1/1 Running 0 7m31s kube-system kube-proxy-bc25c 1/1 Running 0 8m7s kube-system kube-proxy-s8r5g 1/1 Running 0 8m13s kube-system kube-scheduler-master1 1/1 Running 1 8m19s kube-system kube-scheduler-master2 1/1 Running 1 8m6s kube-system kube-scheduler-master3 1/1 Running 1 8m2s kube-system kube-vip-master1 1/1 Running 1 (6m7s ago) 8m19s kube-system kube-vip-master2 1/1 Running 0 8m5s kube-system kube-vip-master3 1/1 Running 0 8m2s kube-system metrics-server-5dff58bc89-2pb6t 1/1 Running 0 6m23s kube-system nodelocaldns-4zhpl 1/1 Running 0 6m30s kube-system nodelocaldns-5mmk8 1/1 Running 0 6m30s kube-system nodelocaldns-fd24d 1/1 Running 0 6m30s kube-system nodelocaldns-nfmm4 1/1 Running 0 6m30s local-path-storage local-path-provisioner-7d4b6f8ccf-fthlv 1/1 Running 0 6m35s metallb-system controller-7db65b4d6c-gbfnr 1/1 Running 0 6m15s metallb-system speaker-5tcwx 1/1 Running 0 6m15s metallb-system speaker-d5msf 1/1 Running 0 6m15s metallb-system speaker-fwslt 1/1 Running 0 6m15s metallb-system speaker-mr7g6 1/1 Running 0 6m15s ``` * 檢查每個節點都有添加 label ``` $ kubectl get no -l my=test NAME STATUS ROLES AGE VERSION master1 Ready control-plane 6m48s v1.32.2 master2 Ready control-plane 6m16s v1.32.2 master3 Ready control-plane 6m9s v1.32.2 worker1 Ready worker 5m41s v1.32.2 ``` * 檢查 kube-proxy 已開啟 ipvs ``` $ kubectl -n kube-system logs $(kubectl get pod -n kube-system -l k8s-app=kube-proxy -o custom-columns=NAME:.metadata.name | grep -v NAME | head -n 1) | grep "Using ipvs Proxier" I0703 17:35:10.379474 1 server_linux.go:231] "Using ipvs Proxier" $ curl -w "\n" http://localhost:10249/proxyMode ipvs ``` * 確認 metrics-server 正常 ``` $ kubectl top no NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%) master1 81m 2% 1036Mi 9% master2 73m 2% 1000Mi 9% master3 70m 2% 1003Mi 9% worker1 41m 1% 640Mi 5% ``` * 已安裝 local-path ``` $ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 8m15s ``` ## 參考 https://github.com/kubernetes-sigs/kubespray