# K3S 寶典
* 安裝
```
$ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.27.4+k3s1 K3S_KUBECONFIG_MODE="644" sh -s - --disable=local-storage --disable=metrics-server
```
* K3s 的二進制文件大小約為 70MB，相較於標準的 Kubernetes 發行版來說非常小。它使用的系統資源也更少，適合在邊緣計算、嵌入式系統或資源受限的虛擬機器上運行。K3s 的部署速度也非常迅速，只需幾分鐘就可以啟動一個 Kubernetes 叢集。
* Kine 是 k3s 專門為了達到輕量化和效能優化而開發的自有儲存解決方案，它不使用傳統的儲存後端（例如 etcd 或 Consul）。相反，Kine 使用 SQLite 數據庫作為儲存引擎，通過在每個節點上存儲資源狀態的方式，實現 Kubernetes API 的持久化存儲。
* 因為 k8s 只會把叢集 metadata 存到 etcd，而透過 kine 可以把叢集 metadata 資料存到異質性的資料庫，讓企業可以選擇自己的資料庫，像是 mysql、sqlite、etcd 等。
* 因為 k3s 通常都是建置在 edge 環境，所以 k3s 透過 Tunnel proxy 來找到 node，而不是讓 kubelet 直接開對外的 port，建立的 tunnel 就是加密資料，可以保護傳輸的資料不被攻擊。不過加密會吃許多 CPU、記憶體資源，因為加密就是在做壓縮和解壓縮。
```
$ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.25.7+k3s1 K3S_KUBECONFIG_MODE="644" sh -
$ kubectl get no
NAME STATUS ROLES AGE VERSION
test Ready control-plane,master 36s v1.25.7+k3s1
$ kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
........
$ kubectl top node
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
test 28m 1% 1483Mi 42%
```
* kubectl 是透過 `/etc/rancher/k3s/k3s.yaml` 這個檔案跟 k3s 溝通，並不是放在 `.kube/config`
* 但是如果要使用 helm ，就會需要把 `k3s.yaml` 複製到 `.kube/config`
```
$ ls -l /etc/rancher/k3s/k3s.yaml
-rw-r--r-- 1 root root 2957 Nov 30 03:48 /etc/rancher/k3s/k3s.yaml
```
## 加入 worker
* worker 上安裝 k3s 執行檔
* 注意執行檔版本要跟 k8s 一致
```
$ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s
$ chmod +x k3s
```
* 在 master 上查看 token
```
$ sudo cat /var/lib/rancher/k3s/server/node-token
K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26
```
* 在 worker 上執行，指定要加入 master 的 ipw
```
$ sudo ./k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26 &
[sudo] password for root:
INFO[0000] Acquiring lock file /var/lib/rancher/k3s/data/.lock
INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/dc43f496a0a9ac19d3b2444d390db38e0cfb38e672721f838b075422b8734994
INFO[0000] Starting k3s agent v1.27.4+k3s1 (36645e73)
......
```
* 幫 worker 些上標籤
```
$ kubectl label node dw1 node-role.kubernetes.io/worker=
node/dw1 labeled
```
```
$ kubectl get no
NAME STATUS ROLES AGE VERSION
dm1 Ready control-plane,master 23m v1.27.4+k3s1
dw1 Ready worker 5m44s v1.27.4+k3s1
```
* suse linux reboot 時自動啟動 k3s
```
# 要給執行權限
$ sudo cat /etc/init.d/boot.local
#!/bin/bash
sudo /home/rancher/k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965b
```
## alpine up board 加入 worker
```
$ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s
$ chmod +x k3s
$ echo "cgroup_memory=1 cgroup_enable=memory" >> /boot/cmdline.txt
# alpine 需要開啟 cgroup
$ sudo rc-update add cgroups
$ sudo rc-service cgroups start
```
## 加入 worker
```
$ sudo ./k3s agent --server https://10.40.41.101:6443 --token K108bf39e8fc757dfe3a713e5a38dbbe987c1f242bba6334f82487ac5a626422f3f::server:53b04590aa6263f92c9d2218ec4bff45 &
```
## 樹梅派設定 cgroup
```
# 新增 cgroup_memory=1 cgroup_enable=memory
$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=d7d3c80e-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=TW cgroup_memory=1 cgroup_enable=memory
$ reboot
```
## 移除 k3s
* 刪除 master
```
# systemctl stop k3s.service
# /usr/local/bin/k3s-killall.sh
# /usr/local/bin/k3s-uninstall.sh
* 移除目錄
$ sudo rm -rf /etc/ceph \
/etc/cni \
/etc/kubernetes \
/etc/rancher \
/opt/cni \
/opt/rke \
/run/secrets/kubernetes.io \
/run/calico \
/run/flannel \
/var/lib/calico \
/var/lib/etcd \
/var/lib/cni \
/var/lib/kubelet \
/var/lib/rancher\
/var/log/containers \
/var/log/kube-audit \
/var/log/pods \
/var/run/calico
# reboot
```
* 刪除 worker
```
# /usr/local/bin/k3s-agent-uninstall.sh
```
## 參數設定
```
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} \
INSTALL_K3S_EXEC="\
--flannel-backend=none \
--disable=servicelb \
--disable=traefik \
--disable-network-policy \
--cluster-cidr=$CLUSTER_CIDR \
--service-cidr=$SERVICE_CIDR \
--cluster-dns=$CLUSTER_DNS \
--datastore-endpoint=etcd \
--node-ip=$NODE_IP \
--data-dir=/opt/rancher/k3s \
--write-kubeconfig-mode=$K3S_KUBECONFIG_MODE \
--kube-apiserver-arg=default-not-ready-toleration-seconds=5 \
--kube-apiserver-arg=default-unreachable-toleration-seconds=5 \
--kube-controller-manager-arg=node-monitor-period=2s \
--kube-controller-manager-arg=node-monitor-grace-period=20s \
--kube-controller-manager-arg=pod-eviction-timeout=5s \
--kubelet-arg=node-status-update-frequency=4s \
--kube-proxy-arg=proxy-mode=ipvs \
" \
sh -s - server --cluster-init
```
## 查看資源
```
$ ps -eo user,pid,cmd,%mem,%cpu | head -n 1;ps -eo user,pid,cmd,%mem,%cpu | grep -v grep | grep "/usr/local/bin/k3s server"
USER PID CMD %MEM %CPU
root 12176 /usr/local/bin/k3s server 10.1 25.2
```
## k3s 自我檢查
* 使用 k3s 命令檢查
```
$ k3s check-config
cat: /sys/kernel/security/apparmor/profiles: No such file or directory
Verifying binaries in /var/lib/rancher/k3s/data/862d15a37357a297d9e0dbbd282cb581265660f14ec13667a58a7ccb63ce3da5/bin:
- sha256sum: good
- links: good
......
STATUS: pass
```
## 檢查 k3s 的 etcd
* 撰寫安裝腳本
```
$ cat <<EOF > install-etcd.sh
#!/bin/bash
ETCD_VER=v3.4.13
# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test
curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
etcd --version
etcdctl version
EOF
```
* 開始安裝
```
$ chmod +x install-etcd.sh
$ ./install-etcd.sh
```
* 檢查 etcd 效能
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf
```
* 檢查 etcd endpoint 狀態
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table
```
* 檢查 etcd endpoint 是否健康
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table
```
* 列出 etcd 告警
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl alarm list
```
* 執行壓縮操作 etcd
```
$ rev=$(ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out fields | grep Revision | cut -d: -f2)
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl compact $rev
```
* etcd 碎片整理，將分散在磁盤上的數據塊重新排列，使其更加緊湊，從而提高 etcd 的讀寫性能和磁盤空間利用率。
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl defrag --cluster
```
* etcd 集群中獲取以 "/" 為前綴的所有鍵名，並以列表形式輸出
```
$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get / --prefix --keys-only
```
* 列出 etcd 版本
```
$ curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version
```
## k3s troubleshooting(阿胖寶解決)
* 解決辦法，加入 `INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true`
```
$ curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true K3S_KUBECONFIG_MODE="644" sh -
```
## Backup and Restore(SQLite)
* 建立測試用 pod
```
$ kubectl run test --image=nginx
$ kubectl get po
NAME READY STATUS RESTARTS AGE
test 1/1 Running 0 41s
```
* SQLite 只需要備份 `/var/lib/rancher/k3s/server` 這個目錄即可
```
$ ls -l /var/lib/rancher/k3s/server
total 12
drwx------ 1 root root 240 Aug 16 13:52 cred
drwx------ 1 root root 64 Aug 16 13:52 db
srw------- 1 root root 0 Aug 16 13:52 kine.sock
drwx------ 1 root root 162 Aug 16 13:52 manifests
lrwxrwxrwx 1 root root 33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token
drwx------ 1 root root 12 Aug 16 13:52 static
drwx------ 1 root root 1180 Aug 16 13:52 tls
-rw------- 1 root root 109 Aug 16 13:52 token
$ tar czvf k3sbk.tar.gz /var/lib/rancher/k3s/server
$ ls -lah k3sbk.tar.gz
-rw-r--r-- 1 root root 1.4M Aug 16 13:56 k3sbk.tar.gz
```
* 測試還原，將 pod 刪除
```
$ kubectl delete po test
pod "test" deleted
```
```
$ tar zxvf k3sbk.tar.gz -C /
$ ls -l /var/lib/rancher/k3s/server/
total 12
drwx------ 1 root root 240 Aug 16 13:52 cred
drwx------ 1 root root 64 Aug 16 13:52 db
srw------- 1 root root 0 Aug 16 14:06 kine.sock
drwx------ 1 root root 162 Aug 16 13:52 manifests
lrwxrwxrwx 1 root root 33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token
drwx------ 1 root root 12 Aug 16 13:52 static
drwx------ 1 root root 1180 Aug 16 13:52 tls
-rw------- 1 root root 109 Aug 16 14:06 token
```
* 重啟 k3s 並確認 pod 是否恢復
```
$ systemctl restart k3s
$ kubectl get po
NAME READY STATUS RESTARTS AGE
test 1/1 Running 0 13m
```
## k3s 高可用安裝
* 建立第一個 K3S Master
```
$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_KUBECONFIG_MODE="644" sh -s server --cluster-init
```
* 建立其他 K3S Master
```
$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s server
```
* 建立 K3S worker
```
$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s -
```
## 檢視憑證
```
$ ls -l /var/lib/rancher/k3s/server/tls/
total 112
-rw-r--r-- 1 root root 1177 Oct 16 2025 client-admin.crt
-rw------- 1 root root 227 Oct 16 2025 client-admin.key
-rw-r--r-- 1 root root 1178 Oct 16 2025 client-auth-proxy.crt
-rw------- 1 root root 227 Oct 16 2025 client-auth-proxy.key
-rw-r--r-- 1 root root 570 Aug 22 13:33 client-ca.crt
-rw------- 1 root root 227 Aug 22 13:33 client-ca.key
-rw-r--r-- 1 root root 1165 Oct 16 2025 client-controller.crt
-rw------- 1 root root 227 Oct 16 2025 client-controller.key
-rw-r--r-- 1 root root 1161 Oct 16 2025 client-k3s-cloud-controller.crt
-rw------- 1 root root 227 Oct 16 2025 client-k3s-cloud-controller.key
-rw-r--r-- 1 root root 1153 Oct 16 2025 client-k3s-controller.crt
-rw------- 1 root root 227 Oct 16 2025 client-k3s-controller.key
-rw-r--r-- 1 root root 1177 Oct 16 2025 client-kube-apiserver.crt
-rw------- 1 root root 227 Oct 16 2025 client-kube-apiserver.key
-rw------- 1 root root 227 Aug 22 13:33 client-kubelet.key
-rw-r--r-- 1 root root 1149 Oct 16 2025 client-kube-proxy.crt
-rw------- 1 root root 227 Oct 16 2025 client-kube-proxy.key
-rw-r--r-- 1 root root 1153 Oct 16 2025 client-scheduler.crt
-rw------- 1 root root 227 Oct 16 2025 client-scheduler.key
-rw-r--r-- 1 root root 3357 Aug 22 13:36 dynamic-cert.json
drwxr-xr-x 1 root root 292 Aug 22 13:33 etcd
-rw-r--r-- 1 root root 591 Aug 22 13:33 request-header-ca.crt
-rw------- 1 root root 227 Aug 22 13:33 request-header-ca.key
-rw-r--r-- 1 root root 566 Aug 22 13:33 server-ca.crt
-rw------- 1 root root 227 Aug 22 13:33 server-ca.key
-rw------- 1 root root 1679 Aug 22 13:33 service.key
-rw-r--r-- 1 root root 1356 Oct 16 2025 serving-kube-apiserver.crt
-rw------- 1 root root 227 Oct 16 2025 serving-kube-apiserver.key
-rw------- 1 root root 227 Aug 22 13:33 serving-kubelet.key
drwx------ 1 root root 124 Aug 22 13:33 temporary-certs
$ openssl x509 -in /var/lib/rancher/k3s/server/tls/server-ca.crt -noout -text
```
## 開啟 debug
```
$ vim /etc/rancher/k3s/config.yaml
debug: true
$ systemctl restart k3s.service
```
#### 連結
https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408
###### tags: `work`
Sign in with Wallet
Connect another wallet