# Setting kube-proxy ipvs mode ## Rancher custom rke2 setting * 需在所有節點上做以下設定 * 在 sles-15-sp5 安裝 ipvs 所需套件 ``` $ sudo zypper install -y conntrack-tools kmod ``` * 設定 ipvs 所需 modules ``` $ sudo nano /etc/modules ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 ``` * 需重新開機 ``` $ sudo reboot ``` ### rancher 介面設定 * 針對要修改的 rke2 cluster 點選 Edit YAML ![image](https://hackmd.io/_uploads/B1nzKTnra.png) * 在 `machineGlobalConfig:` 欄位新增以下參數 ``` kube-proxy-arg: - proxy-mode=ipvs - ipvs-strict-arp=true kube-proxy-extra-mount: - "/lib/modules:/lib/modules:ro" ``` ![image](https://hackmd.io/_uploads/Hk6Sup2Ha.png) * 檢查 kube-proxy 的 log 是否已開啟 ipvs 功能 * 預設會是 round-robin ``` $ kubectl -n kube-system logs $(kubectl get pod -n kube-system -l component=kube-proxy -o custom-columns=NAME:.metadata.name | grep -v NAME | head -n 1) | grep "Using ipvs Proxier" I1205 15:54:24.555343 1 server_others.go:250] "Using ipvs Proxier" ``` * 檢視是否為 ipvs mode ``` $ curl -w "\n" http://localhost:10249/proxyMode ipvs ``` --- ## rke2 enable IPVS Mode * 在 sles-15-sp5 安裝 ipvs 所需套件 ``` $ sudo zypper install -y which conntrack-tools kmod ``` * 設定 ipvs 所需 modules ``` $ sudo nano /etc/modules ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 ``` * 需重新開機 ``` $ sudo reboot ``` * 再以安裝好的 rke2 上設定 ``` $ sudo vim /etc/rancher/rke2/config.yaml node-name: - "rms1" token: my-shared-secret kube-proxy-arg: - proxy-mode=ipvs - ipvs-strict-arp=true kube-proxy-extra-mount: - "/lib/modules:/lib/modules:ro" ``` ![image](https://hackmd.io/_uploads/rJKnQpko6.png) * 重啟 rke2 ``` $ sudo systemctl restart rke2-server ``` * 檢查是否開啟 ipvs ``` $ curl -w "\n" http://localhost:10249/proxyMode ipvs $ kubectl -n kube-system logs $(kubectl get pod -n kube-system -l component=kube-proxy -o custom-columns=NAME:.metadata.name | grep -v NAME | head -n 1) | grep "Using ipvs Proxier" I0206 22:33:16.985551 1 server_others.go:236] "Using ipvs Proxier" ``` --- ## Enable IPVS Mode in Kube Proxy on Talos K8S ```bash! # 1. 修改 controlplane.yaml (在第 443 行左右，cluster.proxy 底下) $ nano controlplane.yaml cluster: proxy: image: xxx mode: ipvs <<- 新增這行字串 # 2. 套用 machine config $ TALOS_ENDPOINT=192.168.61.21 $ talosctl apply-config -e $TALOS_ENDPOINT -n $TALOS_ENDPOINT --file controlplane.yaml --talosconfig=./talosconfig Applied configuration without a reboot # 3. 更新 K8S 套用設定 (可指定原版本不升級) $ talosctl upgrade-k8s -n $TALOS_ENDPOINT -e $TALOS_ENDPOINT --to 1.28.1 --talosconfig=./talosconfig ...以上執行結果省略 > processing manifest apps/v1.DaemonSet/kube-system/kube-proxy --- a/apps/v1.DaemonSet/kube-system/kube-proxy +++ b/apps/v1.DaemonSet/kube-system/kube-proxy @@ -2,7 +2,7 @@ kind: DaemonSet metadata: annotations: - deprecated.daemonset.template.generation: "1" + deprecated.daemonset.template.generation: "2" labels: k8s-app: kube-proxy tier: node @@ -28,7 +28,7 @@ - --conntrack-max-per-core=0 - --hostname-override=$(NODE_NAME) - --kubeconfig=/etc/kubernetes/kubeconfig - - --proxy-mode=iptables + - --proxy-mode=ipvs env: - name: NODE_NAME valueFrom: # 4. 檢查是否成功設定 IPVS Mode $ kubectl -n kube-system logs $(kubectl get pod -n kube-system -l k8s-app=kube-proxy -o custom-columns=NAME:.metadata.name | tail -n 1) | grep "Using ipvs Proxier" I1205 15:41:06.480164 1 server_others.go:218] "Using ipvs Proxier" ``` > There's a caveat here - kube-proxy is a bootstrap manifest, it lives fully in Kubernetes. Talos by default doesn't update them, so you need to run `talosctl upgrade-k8s` to the same Kubernetes version to get things updated. ## 安裝 ipvs rule 檢查套件 ``` $ sudo apt install -y ipvsadm $ sudo ipvsadm -ln IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 172.22.72.1:31679 rr -> 10.244.72.15:80 Masq 1 0 0 TCP 172.22.72.1:31816 rr -> 10.244.72.15:443 Masq 1 0 0 TCP 172.22.72.1:32142 rr -> 10.244.72.182:80 Masq 1 0 0 TCP 172.22.72.100:80 rr -> 10.244.72.15:80 Masq 1 0 0 ...... ``` #### 參考文件 https://github.com/rancher/rke2/issues/4120 https://github.com/rancher/rke2/issues/3710 https://github.com/siderolabs/talos/discussions/7835