# K3S 寶典

* 安裝

$ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.27.4+k3s1 K3S_KUBECONFIG_MODE="644" sh -s - --disable=local-storage --disable=metrics-server

* K3s 的二進制文件大小約為 70MB，相較於標準的 Kubernetes 發行版來說非常小。它使用的系統資源也更少，適合在邊緣計算、嵌入式系統或資源受限的虛擬機器上運行。K3s 的部署速度也非常迅速，只需幾分鐘就可以啟動一個 Kubernetes 叢集。

* Kine 是 k3s 專門為了達到輕量化和效能優化而開發的自有儲存解決方案，它不使用傳統的儲存後端（例如 etcd 或 Consul）。相反，Kine 使用 SQLite 數據庫作為儲存引擎，通過在每個節點上存儲資源狀態的方式，實現 Kubernetes API 的持久化存儲。

* 因為 k8s 只會把叢集 metadata 存到 etcd，而透過 kine 可以把叢集 metadata 資料存到異質性的資料庫，讓企業可以選擇自己的資料庫，像是 mysql、sqlite、etcd 等。

* 因為 k3s 通常都是建置在 edge 環境，所以 k3s 透過 Tunnel proxy 來找到 node，而不是讓 kubelet 直接開對外的 port，建立的 tunnel 就是加密資料，可以保護傳輸的資料不被攻擊。不過加密會吃許多 CPU、記憶體資源，因為加密就是在做壓縮和解壓縮。

$ curl -sfL https://get.k3s.io/ | INSTALL_K3S_VERSION=v1.25.7+k3s1 K3S_KUBECONFIG_MODE="644" sh -

$ kubectl get no
NAME   STATUS   ROLES                  AGE   VERSION
test   Ready    control-plane,master   36s   v1.25.7+k3s1

$ kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
configmaps                        cm           v1                                     true         ConfigMap
........


$ kubectl top node
NAME   CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
test   28m          1%     1483Mi          42%

* kubectl 是透過 `/etc/rancher/k3s/k3s.yaml` 這個檔案跟 k3s 溝通，並不是放在 `.kube/config`
* 但是如果要使用 helm ，就會需要把 `k3s.yaml` 複製到 `.kube/config`

$ ls -l /etc/rancher/k3s/k3s.yaml
-rw-r--r-- 1 root root 2957 Nov 30 03:48 /etc/rancher/k3s/k3s.yaml

## 加入 worker
* worker 上安裝 k3s 執行檔
* 注意執行檔版本要跟 k8s 一致

$ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s

$ chmod +x k3s

* 在 master 上查看 token

$ sudo cat /var/lib/rancher/k3s/server/node-token
K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26

* 在 worker 上執行，指定要加入 master 的 ipw

$ sudo ./k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965bd3866b5e6330f26 &
[sudo] password for root:
INFO[0000] Acquiring lock file /var/lib/rancher/k3s/data/.lock
INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/dc43f496a0a9ac19d3b2444d390db38e0cfb38e672721f838b075422b8734994
INFO[0000] Starting k3s agent v1.27.4+k3s1 (36645e73)
......

* 幫 worker 些上標籤

$ kubectl label node dw1 node-role.kubernetes.io/worker=
node/dw1 labeled

$ kubectl get no
NAME   STATUS   ROLES                  AGE     VERSION
dm1    Ready    control-plane,master   23m     v1.27.4+k3s1
dw1    Ready    worker                 5m44s   v1.27.4+k3s1

* suse linux reboot 時自動啟動 k3s

# 要給執行權限
$ sudo cat /etc/init.d/boot.local
#!/bin/bash

sudo /home/rancher/k3s agent --server https://192.168.11.105:6443 --token K10f057c1ce3e24e2e851ff966d7494e0e038528f310979eaf634eff1ac51daf885::server:0b57d1704f199965b

## alpine up board 加入 worker

$ wget https://github.com/k3s-io/k3s/releases/download/v1.27.4%2Bk3s1/k3s

$ chmod +x k3s

$ echo "cgroup_memory=1 cgroup_enable=memory" >> /boot/cmdline.txt

# alpine 需要開啟 cgroup
$ sudo rc-update add cgroups

$ sudo rc-service cgroups start

## 加入 worker

$ sudo ./k3s agent --server https://10.40.41.101:6443 --token K108bf39e8fc757dfe3a713e5a38dbbe987c1f242bba6334f82487ac5a626422f3f::server:53b04590aa6263f92c9d2218ec4bff45 &

## 樹梅派設定 cgroup

# 新增 cgroup_memory=1 cgroup_enable=memory
$ cat /boot/cmdline.txt
console=serial0,115200 console=tty1 root=PARTUUID=d7d3c80e-02 rootfstype=ext4 fsck.repair=yes rootwait quiet splash plymouth.ignore-serial-consoles cfg80211.ieee80211_regdom=TW cgroup_memory=1 cgroup_enable=memory

$ reboot

## 移除 k3s
* 刪除 master

# systemctl stop k3s.service

# /usr/local/bin/k3s-killall.sh

# /usr/local/bin/k3s-uninstall.sh

* 移除目錄
$ sudo rm -rf /etc/ceph \
       /etc/cni \
       /etc/kubernetes \
       /etc/rancher \
       /opt/cni \
       /opt/rke \
       /run/secrets/kubernetes.io \
       /run/calico \
       /run/flannel \
       /var/lib/calico \
       /var/lib/etcd \
       /var/lib/cni \
       /var/lib/kubelet \
       /var/lib/rancher\
       /var/log/containers \
       /var/log/kube-audit \
       /var/log/pods \
       /var/run/calico


# reboot

* 刪除 worker

# /usr/local/bin/k3s-agent-uninstall.sh

## 參數設定

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=${K3S_VERSION} \
INSTALL_K3S_EXEC="\
--flannel-backend=none \
--disable=servicelb \
--disable=traefik \
--disable-network-policy \
--cluster-cidr=$CLUSTER_CIDR \
--service-cidr=$SERVICE_CIDR \
--cluster-dns=$CLUSTER_DNS \
--datastore-endpoint=etcd \
--node-ip=$NODE_IP \
--data-dir=/opt/rancher/k3s \
--write-kubeconfig-mode=$K3S_KUBECONFIG_MODE \
--kube-apiserver-arg=default-not-ready-toleration-seconds=5 \
--kube-apiserver-arg=default-unreachable-toleration-seconds=5 \
--kube-controller-manager-arg=node-monitor-period=2s \
--kube-controller-manager-arg=node-monitor-grace-period=20s \
--kube-controller-manager-arg=pod-eviction-timeout=5s \
--kubelet-arg=node-status-update-frequency=4s \
--kube-proxy-arg=proxy-mode=ipvs \
" \
sh -s - server --cluster-init

## 查看資源

$ ps -eo user,pid,cmd,%mem,%cpu | head -n 1;ps -eo user,pid,cmd,%mem,%cpu | grep -v grep | grep "/usr/local/bin/k3s server"
USER       PID CMD                         %MEM %CPU
root     12176 /usr/local/bin/k3s server   10.1 25.2

## k3s 自我檢查
* 使用 k3s 命令檢查

$ k3s check-config
cat: /sys/kernel/security/apparmor/profiles: No such file or directory

Verifying binaries in /var/lib/rancher/k3s/data/862d15a37357a297d9e0dbbd282cb581265660f14ec13667a58a7ccb63ce3da5/bin:
- sha256sum: good
- links: good
......
STATUS: pass

## 檢查 k3s 的 etcd
* 撰寫安裝腳本

$ cat <<EOF > install-etcd.sh
#!/bin/bash
ETCD_VER=v3.4.13

# choose either URL
GOOGLE_URL=https://storage.googleapis.com/etcd
GITHUB_URL=https://github.com/etcd-io/etcd/releases/download
DOWNLOAD_URL=${GOOGLE_URL}

rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
rm -rf /tmp/etcd-download-test && mkdir -p /tmp/etcd-download-test

curl -L ${DOWNLOAD_URL}/${ETCD_VER}/etcd-${ETCD_VER}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz
tar xzvf /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz -C /usr/local/bin --strip-components=1
rm -f /tmp/etcd-${ETCD_VER}-linux-amd64.tar.gz

etcd --version
etcdctl version
EOF

* 開始安裝

$ chmod +x install-etcd.sh

$ ./install-etcd.sh

* 檢查 etcd 效能

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl check perf

* 檢查 etcd endpoint 狀態

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --cluster --write-out=table

* 檢查 etcd endpoint 是否健康

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint health --cluster --write-out=table

* 列出 etcd 告警

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl alarm list

* 執行壓縮操作 etcd

$ rev=$(ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl endpoint status --write-out fields | grep Revision | cut -d: -f2)
ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl compact $rev

* etcd 碎片整理，將分散在磁盤上的數據塊重新排列，使其更加緊湊，從而提高 etcd 的讀寫性能和磁盤空間利用率。

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl defrag --cluster

* etcd 集群中獲取以 "/" 為前綴的所有鍵名，並以列表形式輸出

$ ETCDCTL_ENDPOINTS='https://127.0.0.1:2379' ETCDCTL_CACERT='/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt' ETCDCTL_CERT='/var/lib/rancher/k3s/server/tls/etcd/server-client.crt' ETCDCTL_KEY='/var/lib/rancher/k3s/server/tls/etcd/server-client.key' ETCDCTL_API=3 etcdctl get / --prefix --keys-only

* 列出 etcd 版本

$ curl -L --cacert /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/etcd/server-client.crt --key /var/lib/rancher/k3s/server/tls/etcd/server-client.key https://127.0.0.1:2379/version

## k3s troubleshooting(阿胖寶解決)
* 解決辦法，加入 `INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true`

$ curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true K3S_KUBECONFIG_MODE="644" sh -

## Backup and Restore(SQLite)

* 建立測試用 pod

$ kubectl run test --image=nginx

$ kubectl get po
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          41s

* SQLite 只需要備份 `/var/lib/rancher/k3s/server` 這個目錄即可

$ ls -l /var/lib/rancher/k3s/server
total 12
drwx------ 1 root root  240 Aug 16 13:52 cred
drwx------ 1 root root   64 Aug 16 13:52 db
srw------- 1 root root    0 Aug 16 13:52 kine.sock
drwx------ 1 root root  162 Aug 16 13:52 manifests
lrwxrwxrwx 1 root root   33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token
drwx------ 1 root root   12 Aug 16 13:52 static
drwx------ 1 root root 1180 Aug 16 13:52 tls
-rw------- 1 root root  109 Aug 16 13:52 token

$ tar czvf k3sbk.tar.gz /var/lib/rancher/k3s/server

$ ls -lah k3sbk.tar.gz
-rw-r--r-- 1 root root 1.4M Aug 16 13:56 k3sbk.tar.gz

* 測試還原，將 pod 刪除

$ kubectl delete po test
pod "test" deleted

$ tar zxvf k3sbk.tar.gz -C /

$ ls -l /var/lib/rancher/k3s/server/
total 12
drwx------ 1 root root  240 Aug 16 13:52 cred
drwx------ 1 root root   64 Aug 16 13:52 db
srw------- 1 root root    0 Aug 16 14:06 kine.sock
drwx------ 1 root root  162 Aug 16 13:52 manifests
lrwxrwxrwx 1 root root   33 Aug 16 13:52 node-token -> /var/lib/rancher/k3s/server/token
drwx------ 1 root root   12 Aug 16 13:52 static
drwx------ 1 root root 1180 Aug 16 13:52 tls
-rw------- 1 root root  109 Aug 16 14:06 token

* 重啟 k3s 並確認 pod 是否恢復

$ systemctl restart k3s

$ kubectl get po
NAME   READY   STATUS    RESTARTS   AGE
test   1/1     Running   0          13m

## k3s 高可用安裝
* 建立第一個 K3S Master

$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_KUBECONFIG_MODE="644" sh -s server --cluster-init

* 建立其他 K3S Master

$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s server

* 建立 K3S worker

$ curl -sfL https://get.k3s.io | K3S_TOKEN="mysecret" K3S_URL="https://192.168.11.52:6443" K3S_KUBECONFIG_MODE="644" sh -s -

## 檢視憑證

$ ls -l /var/lib/rancher/k3s/server/tls/
total 112
-rw-r--r-- 1 root root 1177 Oct 16  2025 client-admin.crt
-rw------- 1 root root  227 Oct 16  2025 client-admin.key
-rw-r--r-- 1 root root 1178 Oct 16  2025 client-auth-proxy.crt
-rw------- 1 root root  227 Oct 16  2025 client-auth-proxy.key
-rw-r--r-- 1 root root  570 Aug 22 13:33 client-ca.crt
-rw------- 1 root root  227 Aug 22 13:33 client-ca.key
-rw-r--r-- 1 root root 1165 Oct 16  2025 client-controller.crt
-rw------- 1 root root  227 Oct 16  2025 client-controller.key
-rw-r--r-- 1 root root 1161 Oct 16  2025 client-k3s-cloud-controller.crt
-rw------- 1 root root  227 Oct 16  2025 client-k3s-cloud-controller.key
-rw-r--r-- 1 root root 1153 Oct 16  2025 client-k3s-controller.crt
-rw------- 1 root root  227 Oct 16  2025 client-k3s-controller.key
-rw-r--r-- 1 root root 1177 Oct 16  2025 client-kube-apiserver.crt
-rw------- 1 root root  227 Oct 16  2025 client-kube-apiserver.key
-rw------- 1 root root  227 Aug 22 13:33 client-kubelet.key
-rw-r--r-- 1 root root 1149 Oct 16  2025 client-kube-proxy.crt
-rw------- 1 root root  227 Oct 16  2025 client-kube-proxy.key
-rw-r--r-- 1 root root 1153 Oct 16  2025 client-scheduler.crt
-rw------- 1 root root  227 Oct 16  2025 client-scheduler.key
-rw-r--r-- 1 root root 3357 Aug 22 13:36 dynamic-cert.json
drwxr-xr-x 1 root root  292 Aug 22 13:33 etcd
-rw-r--r-- 1 root root  591 Aug 22 13:33 request-header-ca.crt
-rw------- 1 root root  227 Aug 22 13:33 request-header-ca.key
-rw-r--r-- 1 root root  566 Aug 22 13:33 server-ca.crt
-rw------- 1 root root  227 Aug 22 13:33 server-ca.key
-rw------- 1 root root 1679 Aug 22 13:33 service.key
-rw-r--r-- 1 root root 1356 Oct 16  2025 serving-kube-apiserver.crt
-rw------- 1 root root  227 Oct 16  2025 serving-kube-apiserver.key
-rw------- 1 root root  227 Aug 22 13:33 serving-kubelet.key
drwx------ 1 root root  124 Aug 22 13:33 temporary-certs

$ openssl x509 -in /var/lib/rancher/k3s/server/tls/server-ca.crt -noout -text

## 開啟 debug

$ vim /etc/rancher/k3s/config.yaml
debug: true

$ systemctl restart k3s.service

#### 連結

https://gist.github.com/superseb/0c06164eef5a097c66e810fe91a9d408

###### tags: `work`