# RKE2 kubelet log rotate config

* rke2 預設 kubelet log 每分大小為 10Mi，總共會存 5 份

## kubelet log 存放位置

* 根據 pod 在不同 node 找到 log 存放位置

$ ls -alh /var/log/pods/

## 設定 rancher custom rke2

* Cluster Management -> `指定 downstream cluster` Edit Config -> Advanced

調整為以下
> kubelet log 每分大小為 10Ki，總共會存 3 份
> container-log-max-files=3
> container-log-max-size=10Ki


## 原生 rke2 叢集
* 如果是 HA 架構需要在每一台 node 上設定

調整為以下
> kubelet log 每分大小為 10Ki，總共會存 3 份
> container-log-max-files=3
> container-log-max-size=10Ki

$ kubectl get no
NAME   STATUS   ROLES                       AGE     VERSION
r1     Ready    control-plane,etcd,master   5d22h   v1.28.10+rke2r1
r2     Ready    control-plane,etcd,master   5d22h   v1.28.10+rke2r1
r3     Ready    control-plane,etcd,master   5d22h   v1.28.10+rke2r1

* 設定 r1

$ nano /etc/rancher/rke2/config.yaml
node-name:
  - "r1"
token: my-shared-secret
etcd-extra-env: TZ=Asia/Taipei
kube-apiserver-extra-env: TZ=Asia/Taipei
kube-controller-manager-extra-env: TZ=Asia/Taipei
kube-proxy-extra-env: TZ=Asia/Taipei
kube-scheduler-extra-env: TZ=Asia/Taipei
cloud-controller-manager-extra-env: TZ=Asia/Taipei
kubelet-arg:                           # add
  - container-log-max-files=3          # add
  - container-log-max-size=10Ki        # add

$ systemctl restart rke2-server.service

* 設定 r2

$ nano /etc/rancher/rke2/config.yaml
server: https://192.168.11.52:9345
node-name:
  - "r2"
token: my-shared-secret
etcd-extra-env: TZ=Asia/Taipei
kube-apiserver-extra-env: TZ=Asia/Taipei
kube-controller-manager-extra-env: TZ=Asia/Taipei
kube-proxy-extra-env: TZ=Asia/Taipei
kube-scheduler-extra-env: TZ=Asia/Taipei
cloud-controller-manager-extra-env: TZ=Asia/Taipei
kubelet-arg:                           # add
  - container-log-max-files=3          # add
  - container-log-max-size=10Ki        # add

$ systemctl restart rke2-server.service

* 設定 r3

$ nano /etc/rancher/rke2/config.yaml
server: https://192.168.11.52:9345
node-name:
  - "r3"
token: my-shared-secret
etcd-extra-env: TZ=Asia/Taipei
kube-apiserver-extra-env: TZ=Asia/Taipei
kube-controller-manager-extra-env: TZ=Asia/Taipei
kube-proxy-extra-env: TZ=Asia/Taipei
kube-scheduler-extra-env: TZ=Asia/Taipei
cloud-controller-manager-extra-env: TZ=Asia/Taipei
kubelet-arg:                           # add
  - container-log-max-files=3          # add
  - container-log-max-size=10Ki        # add

$ systemctl restart rke2-server.service

## 檢查是否符合預期

* rancher custom rke2 檢查

$ cat /etc/rancher/rke2/config.yaml.d/50-rancher.yaml
......
  ],
  "kubelet-arg": [
    "container-log-max-files=3",
    "container-log-max-size=10Ki"
  ],
......

* 原生 rke2 叢集檢查

$ cat /etc/rancher/rke2/config.yaml
......
kubelet-arg:
  - container-log-max-files=3
  - container-log-max-size=10Ki

## 測試建立 pod

$ echo 'apiVersion: v1
kind: Pod
metadata:
  name: pod-log
spec:
  containers:
  - name: pod-log
    image: alpine
    command: ["/bin/sh"]
    args:
    - -c
    - |
      mkdir -p /var/log;i=0;
      while true;
      do
        echo "$(date) INFO $i";  
        i=$((i+1));sleep 1;
      done
' | kubectl apply -f -

$ kubectl get po -o wide
NAME      READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
pod-log   1/1     Running   0          13s   10.42.0.196   cilium-m1   <none>           <none>

* log 檔案大小為 10ki 左右，並且只會保留 3 份，且最後一份會使用 gz 壓縮

$ ls -alh /var/log/pods/default_pod-log_8aab8b04-2f0c-4544-9148-a9bac11ead34/pod-log/
total 24K
drwxr-xr-x 1 root root  100 Jul  9 15:52 .
drwxr-xr-x 1 root root   14 Jul  9 15:45 ..
-rw-r----- 1 root root 5.1K Jul  9 15:53 0.log
-rw-r--r-- 1 root root 1.8K Jul  9 15:52 0.log.20240709-155010.gz
-rw-r----- 1 root root  11K Jul  9 15:52 0.log.20240709-155221

## 參考

https://github.com/rancher/rke2/discussions/3002#discussioncomment-2862822