---
# System prepended metadata

title: Kubeadm 開啟 audit log 收集 k8s 系統 log
tags: [工具]

---

# Kubeadm 開啟 audit log 收集 k8s 系統 log

## 環境準備

* 已準備好一座 k8s
```
$ kubectl get no
NAME   STATUS   ROLES           AGE    VERSION
m1     Ready    control-plane   137m   v1.34.1
m2     Ready    control-plane   134m   v1.34.1
m3     Ready    control-plane   133m   v1.34.1
w1     Ready    <none>          133m   v1.34.1
```


## apiserver 開啟 audit log 設定
* 在每一台 contorl plane 設定 `audit-policy`
  - `None` : 當 policy 規則把某些事件設成 `None`，那類事件就不會寫入審計日誌。
  - `Metadata` : 只記錄元資料（metadata）：像是時間、使用者、來源 IP、verb（get/list/create/...）、requestURI、resource、namespace、objectRef、結果狀態（responseStatus）等。
  - `Request` : 除了 metadata 外，會把 request body（requestObject） 抓下來，但通常不包含完整 response body。
  - `RequestResponse` : 表示會把所有 API 物件的內容都記錄下來。包含 metadata、requestObject、以及 responseObject（API 回傳內容）。適合追蹤複雜問題、debug 或法遵需要完全記錄請求/回應內容的情境。
```
$ sudo nano /etc/kubernetes/audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: RequestResponse
```


* apiserver 新增以下參數
  - `audit-policy-file` : 指定 audit log policy 設定檔位置。
  - `audit-log-path` : 指定 audit log 要存放在哪個位置。
  - `audit-log-maxage` : 保留 log 檔案的最長天數。
  - `audit-log-maxbackup` : 要保留多少 log 檔案。當超過這個數量時，最舊的備份會被刪掉。
  - `audit-log-maxsize` : 單一 log 檔案達到此大小後會被 rotate，單位是 MB。
```
$ kubectl edit cm -n kube-system kubeadm-config
......
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
      - name: audit-policy-file
        value: "/etc/kubernetes/audit-policy.yaml"
      - name: audit-log-path
        value: "/var/log/kubernetes/audit-logs.txt"
      - name: audit-log-maxage
        value: "10"
      - name: audit-log-maxbackup
        value: "2"
      - name: audit-log-maxsize
        value: "100"
      extraVolumes:
      - name: audit-policy
        hostPath: /etc/kubernetes/
        mountPath: /etc/kubernetes/
        readOnly: true
      - name: audit-log
        hostPath: /var/log/kubernetes/
        mountPath: /var/log/kubernetes/
```
![image](https://hackmd.io/_uploads/Sk0nsa_0xe.png)

* 將 configmap 匯出
```
$ kubectl get cm -n kube-system kubeadm-config -o jsonpath={.data.ClusterConfiguration} > config.yaml
```
* 把 `config.yaml` 匯入到每一台 control plane 後套用設定到 `kube-apiserver` static pod
```
$ sudo kubeadm init phase control-plane apiserver --config ./config.yaml
```
* 重啟 kubelet
```
$ sudo systemctl daemon-reload

$ sudo systemctl restart kubelet

$ sudo crictl ps --name kube-apiserver
CONTAINER           IMAGE                                                              CREATED             STATE               NAME                ATTEMPT             POD ID              POD                 NAMESPACE
eff3881e1f2fc       c3994bc6961024917ec0aeee02e62828108c21a52d87648e30f3080d9cbadc97   3 seconds ago       Running             kube-apiserver      0                   acf65d9e37fe4       kube-apiserver-m1   kube-system
```

## 驗證

* 在每一台 control plane 都可以在以下目錄看到 log
```
$ ls -l /var/log/kubernetes/audit-logs.txt
-rw------- 1 root root 5983397 Oct 24 17:53 /var/log/kubernetes/audit-logs.txt
```
* 測試產生一個 deployment
```
$ kubectl create deploy test --image=nginx
```

* 分析 audit log
```
$ sudo jq -c '
  select(
    .verb=="create"
    and (.requestURI | test("deployments"))
    and (
      (.objectRef.name=="test") 
      or (.requestObject.metadata?.name=="test")
      or (.requestURI | test("/deployments/test"))
    )
    and (.objectRef.namespace // "default")=="default"
  )
' /var/log/kubernetes/audit-logs.txt | jq .
{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "RequestResponse",
  "auditID": "f4323778-30ec-4d7c-8e56-d95ac4e6166e",
  "stage": "ResponseComplete",
  "requestURI": "/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-create&fieldValidation=Strict",
  "verb": "create",
  "user": {
    "username": "kubernetes-admin",
    "groups": [
      "kubeadm:cluster-admins",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/credential-id": [
        "X509SHA256=14769e835adae953af4c55e4fe8678be9f0b4703f1675b3b3dead4298dd7a988"
      ]
    }
  },
  "sourceIPs": [
    "10.10.7.32"
  ],
  "userAgent": "kubectl/v1.34.1 (linux/amd64) kubernetes/93248f9",
  "objectRef": {
    "resource": "deployments",
    "namespace": "default",
    "name": "test",
    "apiGroup": "apps",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "metadata": {},
    "code": 201
  },
  "requestObject": {
    "kind": "Deployment",
    "apiVersion": "apps/v1",
    "metadata": {
      "name": "test",
      "labels": {
        "app": "test"
      }
    },
    "spec": {
      "replicas": 1,
      "selector": {
        "matchLabels": {
          "app": "test"
        }
      },
      "template": {
        "metadata": {
          "labels": {
            "app": "test"
          }
        },
        "spec": {
          "containers": [
            {
              "name": "nginx",
              "image": "nginx",
              "resources": {},
              "terminationMessagePath": "/dev/termination-log",
              "terminationMessagePolicy": "File",
              "imagePullPolicy": "Always"
            }
          ],
          "restartPolicy": "Always",
          "terminationGracePeriodSeconds": 30,
          "dnsPolicy": "ClusterFirst",
          "securityContext": {},
          "schedulerName": "default-scheduler"
        }
      },
      "strategy": {
        "type": "RollingUpdate",
        "rollingUpdate": {
          "maxUnavailable": "25%",
          "maxSurge": "25%"
        }
      },
      "revisionHistoryLimit": 10,
      "progressDeadlineSeconds": 600
    },
    "status": {}
  },
  "responseObject": {
    "kind": "Deployment",
    "apiVersion": "apps/v1",
    "metadata": {
      "name": "test",
      "namespace": "default",
      "uid": "27d498e2-0a90-4851-87ed-c310ccbb84e5",
      "resourceVersion": "44602",
      "generation": 1,
      "creationTimestamp": "2025-10-24T10:01:47Z",
      "labels": {
        "app": "test"
      },
      "managedFields": [
        {
          "manager": "kubectl-create",
          "operation": "Update",
          "apiVersion": "apps/v1",
          "time": "2025-10-24T10:01:47Z",
          "fieldsType": "FieldsV1",
          "fieldsV1": {
            "f:metadata": {
              "f:labels": {
                ".": {},
                "f:app": {}
              }
            },
            "f:spec": {
              "f:progressDeadlineSeconds": {},
              "f:replicas": {},
              "f:revisionHistoryLimit": {},
              "f:selector": {},
              "f:strategy": {
                "f:rollingUpdate": {
                  ".": {},
                  "f:maxSurge": {},
                  "f:maxUnavailable": {}
                },
                "f:type": {}
              },
              "f:template": {
                "f:metadata": {
                  "f:labels": {
                    ".": {},
                    "f:app": {}
                  }
                },
                "f:spec": {
                  "f:containers": {
                    "k:{\"name\":\"nginx\"}": {
                      ".": {},
                      "f:image": {},
                      "f:imagePullPolicy": {},
                      "f:name": {},
                      "f:resources": {},
                      "f:terminationMessagePath": {},
                      "f:terminationMessagePolicy": {}
                    }
                  },
                  "f:dnsPolicy": {},
                  "f:restartPolicy": {},
                  "f:schedulerName": {},
                  "f:securityContext": {},
                  "f:terminationGracePeriodSeconds": {}
                }
              }
            }
          }
        }
      ]
    },
    "spec": {
      "replicas": 1,
      "selector": {
        "matchLabels": {
          "app": "test"
        }
      },
      "template": {
        "metadata": {
          "labels": {
            "app": "test"
          }
        },
        "spec": {
          "containers": [
            {
              "name": "nginx",
              "image": "nginx",
              "resources": {},
              "terminationMessagePath": "/dev/termination-log",
              "terminationMessagePolicy": "File",
              "imagePullPolicy": "Always"
            }
          ],
          "restartPolicy": "Always",
          "terminationGracePeriodSeconds": 30,
          "dnsPolicy": "ClusterFirst",
          "securityContext": {},
          "schedulerName": "default-scheduler"
        }
      },
      "strategy": {
        "type": "RollingUpdate",
        "rollingUpdate": {
          "maxUnavailable": "25%",
          "maxSurge": "25%"
        }
      },
      "revisionHistoryLimit": 10,
      "progressDeadlineSeconds": 600
    },
    "status": {}
  },
  "requestReceivedTimestamp": "2025-10-24T10:01:47.337569Z",
  "stageTimestamp": "2025-10-24T10:01:47.367700Z",
  "annotations": {
    "authorization.k8s.io/decision": "allow",
    "authorization.k8s.io/reason": "RBAC: allowed by ClusterRoleBinding \"kubeadm:cluster-admins\" of ClusterRole \"cluster-admin\" to Group \"kubeadm:cluster-admins\""
  }
}
```

## 參考
https://kubernetes.io/zh-cn/docs/tasks/debug/debug-cluster/audit/

https://www.cnblogs.com/zhangmingcheng/p/16539514.html