--- # System prepended metadata title: K3S 寶典 tags: [work] --- # K3S 寶典 ![](https://i.imgur.com/trCCPWr.png) * 安裝 ``` $ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.27.4+k3s1 K3S_KUBECONFIG_MODE="644" sh -s - --disable=local-storage --disable=metrics-server ``` * K3s 的二進制文件大小約為 70MB，相較於標準的 Kubernetes 發行版來說非常小。它使用的系統資源也更少，適合在邊緣計算、嵌入式系統或資源受限的虛擬機器上運行。K3s 的部署速度也非常迅速，只需幾分鐘就可以啟動一個 Kubernetes 叢集。 * Kine 是 k3s 專門為了達到輕量化和效能優化而開發的自有儲存解決方案，它不使用傳統的儲存後端（例如 etcd 或 Consul）。相反，Kine 使用 SQLite 數據庫作為儲存引擎，通過在每個節點上存儲資源狀態的方式，實現 Kubernetes API 的持久化存儲。 * 因為 k8s 只會把叢集 metadata 存到 etcd，而透過 kine 可以把叢集 metadata 資料存到異質性的資料庫，讓企業可以選擇自己的資料庫，像是 mysql、sqlite、etcd 等。 * 因為 k3s 通常都是建置在 edge 環境，所以 k3s 透過 Tunnel proxy 來找到 node，而不是讓 kubelet 直接開對外的 port，建立的 tunnel 就是加密資料，可以保護傳輸的資料不被攻擊。不過加密會吃許多 CPU、記憶體資源，因為加密就是在做壓縮和解壓縮。 ``` $ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.25.7+k3s1 K3S_KUBECONFIG_MODE="644" sh - $ kubectl get no NAME STATUS ROLES AGE VERSION test Ready control-plane,master 36s v1.25.7+k3s1 $ kubectl api-resources NAME SHORTNAMES APIVERSION NAMESPACED KIND bindings v1 true Binding componentstatuses cs v1 false ComponentStatus configmaps cm v1 true ConfigMap ........ $ kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% test 28m 1% 1483Mi 42% ``` * kubectl 是透過 `/etc/rancher/k3s/k3s.yaml` 這個檔案跟 k3s 溝通，並不是放在 `.kube/config` * 但是如果要使用 helm ，就會需要把 `k3s.yaml` 複製到 `.kube/config` ``` $ ls -l /etc/rancher/k3s/k3s.yaml -rw-r--r-- 1 root root 2957 Nov 30 03:48 /etc/rancher/k3s/k3s.yaml ``` ## 加入 worker * worker 上安裝 k3s 執行檔 * 注意執行檔版本要跟 k8s 一致 ``` $ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s $ chmod +x k3s ``` * 在 master 上查看 token ``` $ sudo cat /var/lib/rancher/k3s/server/node-token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26 ``` * 在 worker 上執行，指定要加入 master 的 ipw ``` $ sudo ./k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26 & [sudo] password for root: INFO[0000] Acquiring lock file /var/lib/rancher/k3s/data/.lock INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/dc43f496a0a9ac19d3b2444d390db38e0cfb38e672721f838b075422b8734994 INFO[0000] Starting k3s agent v1.27.4+k3s1 (36645e73) ...... ``` * 幫 worker 些上標籤 ``` $ kubectl label node dw1 node-role.kubernetes.io/worker= node/dw1 labeled ``` ``` $ kubectl get no NAME STATUS ROLES AGE VERSION dm1 Ready control-plane,master 23m v1.27.4+k3s1 dw1 Ready worker 5m44s v1.27.4+k3s1 ``` * suse linux reboot 時自動啟動 k3s ``` # 要給執行權限 $ sudo cat /etc/init.d/boot.local #!/bin/bash sudo /home/rancher/k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965b ``` ## alpine up board 加入 worker ``` $ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s $ chmod +x k3s $ echo "cgroup_memory=1 cgroup_enable=memory" >> /boot/cmdline.txt # alpine 需要開啟 cgroup $ sudo rc-update add cgroups $ sudo rc-service cgroups start ``` ## 加入 worker ``` $ sudo ./k3s agent --server https://10.40.41.101:6443 --token K108bf39e8fc757dfe3a713e5a38dbbe987c1f242bba6334f82487ac5a626422f3f::server:53b04590aa6263f92c9d2218ec4bff45 & ``` ## 樹梅派設定 cgroup ``` # 新增 cgroup_memory=1 cgroup_enable=memory $ cat /boot/cmdline.txt console=serial0,115200 console=tty1 root=PARTUUID=d7d3c80e-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=TW cgroup_memory=1 cgroup_enable=memory $ reboot ``` ## 移除 k3s * 刪除 master ``` # systemctl stop k3s.service # /usr/local/bin/k3s-killall.sh # /usr/local/bin/k3s-uninstall.sh * 移除目錄 $ sudo rm -rf /etc/ceph \ /etc/cni \ /etc/kubernetes \ /etc/rancher \ /opt/cni \ /opt/rke \ /run/secrets/kubernetes.io \ /run/calico \ /run/flannel \ /var/lib/calico \ /var/lib/etcd \ /var/lib/cni \ /var/lib/kubelet \ /var/lib/rancher\ /var/log/containers \ /var/log/kube-audit \ /var/log/pods \ /var/run/calico # reboot ``` * 刪除 worker ``` # /usr/local/bin/k3s-agent-uninstall.sh ``` ## 參數設定 ``` curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} \ INSTALL_K3S_EXEC="\ --flannel-backend=none \ --disable=servicelb \ --disable=traefik \ --disable-network-policy \ --cluster-cidr=$CLUSTER_CIDR \ --service-cidr=$SERVICE_CIDR \ --cluster-dns=$CLUSTER_DNS \ --datastore-endpoint=etcd \ --node-ip=$NODE_IP \ --data-dir=/opt/rancher/k3s \ --write-kubeconfig-mode=$K3S_KUBECONFIG_MODE \ --kube-apiserver-arg=default-not-ready-toleration-seconds=5 \ --kube-apiserver-arg=default-unreachable-toleration-seconds=5 \ --kube-controller-manager-arg=node-monitor-period=2s \ --kube-controller-manager-arg=node-monitor-grace-period=20s \ --kube-controller-manager-arg=pod-eviction-timeout=5s \ --kubelet-arg=node-status-update-frequency=4s \ --kube-proxy-arg=proxy-mode=ipvs \ " \ sh -s - server --cluster-init ``` ## 查看資源 ``` $ ps -eo user,pid,cmd,%mem,%cpu | head -n 1;ps -eo user,pid,cmd,%mem,%cpu | grep -v grep | grep "/usr/local/bin/k3s server" USER PID CMD %MEM %CPU root 12176 /usr/local/bin/k3s server 10.1 25.2 ``` ## k3s 自我檢查 * 使用 k3s 命令檢查 ``` $ k3s check-config cat: /sys/kernel/security/apparmor/profiles: No such file or directory Verifying binaries in /var/lib/rancher/k3s/data/862d15a37357a297d9e0dbbd282cb581265660f14ec13667a58a7ccb63ce3da5/bin: - sha256sum: good - links: good ...... STATUS: pass ``` ## 檢查 k3s 的 etcd * 撰寫安裝腳本 ``` $ cat < install-etcd.sh #!/bin/bash ETCD_VER=v3.4.13 # choose either URL GOOGLE_URL=https://storage.googleapis.com/etcd GITHUB_URL=https://github.com/etcd-io/etcd/releases/download DOWNLOAD_URL=${GOOGLE_URL} rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz etcd --version etcdctl version EOF ``` * 開始安裝 ``` $ chmod +x install-etcd.sh $ ./install-etcd.sh ``` * 檢查 etcd 效能 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf ``` * 檢查 etcd endpoint 狀態 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table ``` * 檢查 etcd endpoint 是否健康 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table ``` * 列出 etcd 告警 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl alarm list ``` * 執行壓縮操作 etcd ``` $ rev=$(ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out fields | grep Revision | cut -d: -f2) ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl compact $rev ``` * etcd 碎片整理，將分散在磁盤上的數據塊重新排列，使其更加緊湊，從而提高 etcd 的讀寫性能和磁盤空間利用率。 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl defrag --cluster ``` * etcd 集群中獲取以 "/" 為前綴的所有鍵名，並以列表形式輸出 ``` $ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get / --prefix --keys-only ``` * 列出 etcd 版本 ``` $ curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version ``` ## k3s troubleshooting(阿胖寶解決) ![image](https://hackmd.io/_uploads/Sy-xAvfe0.png) * 解決辦法，加入 `INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true` ``` $ curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true K3S_KUBECONFIG_MODE="644" sh - ``` ## Backup and Restore(SQLite) * 建立測試用 pod ``` $ kubectl run test --image=nginx $ kubectl get po NAME READY STATUS RESTARTS AGE test 1/1 Running 0 41s ``` * SQLite 只需要備份 `/var/lib/rancher/k3s/server` 這個目錄即可 ``` $ ls -l /var/lib/rancher/k3s/server total 12 drwx------ 1 root root 240 Aug 16 13:52 cred drwx------ 1 root root 64 Aug 16 13:52 db srw------- 1 root root 0 Aug 16 13:52 kine.sock drwx------ 1 root root 162 Aug 16 13:52 manifests lrwxrwxrwx 1 root root 33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token drwx------ 1 root root 12 Aug 16 13:52 static drwx------ 1 root root 1180 Aug 16 13:52 tls -rw------- 1 root root 109 Aug 16 13:52 token $ tar czvf k3sbk.tar.gz /var/lib/rancher/k3s/server $ ls -lah k3sbk.tar.gz -rw-r--r-- 1 root root 1.4M Aug 16 13:56 k3sbk.tar.gz ``` * 測試還原，將 pod 刪除 ``` $ kubectl delete po test pod "test" deleted ``` ``` $ tar zxvf k3sbk.tar.gz -C / $ ls -l /var/lib/rancher/k3s/server/ total 12 drwx------ 1 root root 240 Aug 16 13:52 cred drwx------ 1 root root 64 Aug 16 13:52 db srw------- 1 root root 0 Aug 16 14:06 kine.sock drwx------ 1 root root 162 Aug 16 13:52 manifests lrwxrwxrwx 1 root root 33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token drwx------ 1 root root 12 Aug 16 13:52 static drwx------ 1 root root 1180 Aug 16 13:52 tls -rw------- 1 root root 109 Aug 16 14:06 token ``` * 重啟 k3s 並確認 pod 是否恢復 ``` $ systemctl restart k3s $ kubectl get po NAME READY STATUS RESTARTS AGE test 1/1 Running 0 13m ``` ## k3s 高可用安裝 * 建立第一個 K3S Master ``` $ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_KUBECONFIG_MODE="644" sh -s server --cluster-init ``` * 建立其他 K3S Master ``` $ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s server ``` * 建立 K3S worker ``` $ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s - ``` ## 檢視憑證 ``` $ ls -l /var/lib/rancher/k3s/server/tls/ total 112 -rw-r--r-- 1 root root 1177 Oct 16 2025 client-admin.crt -rw------- 1 root root 227 Oct 16 2025 client-admin.key -rw-r--r-- 1 root root 1178 Oct 16 2025 client-auth-proxy.crt -rw------- 1 root root 227 Oct 16 2025 client-auth-proxy.key -rw-r--r-- 1 root root 570 Aug 22 13:33 client-ca.crt -rw------- 1 root root 227 Aug 22 13:33 client-ca.key -rw-r--r-- 1 root root 1165 Oct 16 2025 client-controller.crt -rw------- 1 root root 227 Oct 16 2025 client-controller.key -rw-r--r-- 1 root root 1161 Oct 16 2025 client-k3s-cloud-controller.crt -rw------- 1 root root 227 Oct 16 2025 client-k3s-cloud-controller.key -rw-r--r-- 1 root root 1153 Oct 16 2025 client-k3s-controller.crt -rw------- 1 root root 227 Oct 16 2025 client-k3s-controller.key -rw-r--r-- 1 root root 1177 Oct 16 2025 client-kube-apiserver.crt -rw------- 1 root root 227 Oct 16 2025 client-kube-apiserver.key -rw------- 1 root root 227 Aug 22 13:33 client-kubelet.key -rw-r--r-- 1 root root 1149 Oct 16 2025 client-kube-proxy.crt -rw------- 1 root root 227 Oct 16 2025 client-kube-proxy.key -rw-r--r-- 1 root root 1153 Oct 16 2025 client-scheduler.crt -rw------- 1 root root 227 Oct 16 2025 client-scheduler.key -rw-r--r-- 1 root root 3357 Aug 22 13:36 dynamic-cert.json drwxr-xr-x 1 root root 292 Aug 22 13:33 etcd -rw-r--r-- 1 root root 591 Aug 22 13:33 request-header-ca.crt -rw------- 1 root root 227 Aug 22 13:33 request-header-ca.key -rw-r--r-- 1 root root 566 Aug 22 13:33 server-ca.crt -rw------- 1 root root 227 Aug 22 13:33 server-ca.key -rw------- 1 root root 1679 Aug 22 13:33 service.key -rw-r--r-- 1 root root 1356 Oct 16 2025 serving-kube-apiserver.crt -rw------- 1 root root 227 Oct 16 2025 serving-kube-apiserver.key -rw------- 1 root root 227 Aug 22 13:33 serving-kubelet.key drwx------ 1 root root 124 Aug 22 13:33 temporary-certs $ openssl x509 -in /var/lib/rancher/k3s/server/tls/server-ca.crt -noout -text ``` ## 開啟 debug ``` $ vim /etc/rancher/k3s/config.yaml debug: true $ systemctl restart k3s.service ``` #### 連結 https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408 ###### tags: `work`