# Esun_C.N._logging & Monitoring

## CKA課程
### 01-Logging-and-Monitoring-Section-Introduction

### 02-Monitor-Cluster-Components
透過Metrics Server監控 cluster components，透過Node內kubelet，執行API回傳需監控的資訊給Metrics Server，再透過metrics api回傳給kubectl top呈現監控結果。

* 圖1
https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

* 操作步驟
* 從git下載 `$ git clone https://github.com/kubernetes-incubator/metrics-server.git`
* 部屬Server `$ kubectl create -f metric-server/deploy/1.8+/`
* 下指令檢視資源使用狀況(CPU、Memory、Disk)
`$ kubectl top node`
`$ kubectl top pod`

* 其他工具監控，如:Prometheus、Elastic_Stack、DataDog、dynatrace
* Ref
* https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/04-Logging-and-Monitoring/02-Monitor-Cluster-Components.md

### 03-Practice-Test-Monitor-Cluster-Components
N/A

### 04-Managing-Application-Logs
* Docker log
* 透過`$ docker logs -f ecf` 查詢log資訊

* K8S log
* 可用`$ kubectl logs -f <pod-name> <container-name>`查詢指定pod內的container log
例如`$ kubectl logs -f even-simulator-pod event-simulator`

https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts

##### Ref:
* https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/04-Logging-and-Monitoring/04-Managing-Application-Logs.md
* https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts

### 05-Download-Presentation-Deck
N/A
### 06-Practice-Test-Managing-Application-Logs
N/A

---
## 目的
* 除錯
* 監控cluster行為

---
## 內容
* Kubernetes defaults to the "json-file" logging driver, in which docker writes the stdout/stderr streams to a file in the json format as shown below
> {“log”: “The actual log line”, “stream”: “stderr”, “time”: “2016-10-05T00:00:30.082640485Z”}

* Ref
https://github.com/kubernetes/design-proposals-archive/blob/main/node/kubelet-cri-logging.md

## logging方式
* Logging
* 做法與傳統VM不一樣的部分為，Container內的log會因Container被回收而log也會清掉，會額外傳至特定位置儲存
![方式1]()
* Container Logging
* Node Logging
* Cluster Logging
* Event logging
* `kubectl get events -n <namespace>`
* `kubectl describe pod <pod-name>`
* Audit logging
* Kubernetes audit logs are detailed descriptions of each call made to the kube-apiserver
* Ingress Logging

* Ref
* https://sematext.com/guides/kubernetes-logging/

## log類型
1. **Application log**,例如:Nginx、Tomcat log

2. **Node-level log**，例如:Kubernetes自身各大组件的log（包括 kubelet、kube-proxy 等），容器運行的log（比如 Docker），存在/var/log內

* 位置
![類型1]()
https://yashbindlish.medium.com/under-the-hood-an-introduction-to-kubernetes-architecture-bb9d8599f837
* **Control Plane nodes**(Master Node)
/var/log/kube-apiserver.log - **API Server**, responsible for serving the API

/var/log/kube-scheduler.log - **Scheduler**, responsible for making scheduling decisions

/var/log/kube-controller-manager.log - a component that runs most Kubernetes built-in controllers, with the notable exception of scheduling (the kube-scheduler handles scheduling).

* **Worker Nodes**
/var/log/kubelet.log - logs from the **kubelet**, responsible for running containers on the node
/var/log/kube-proxy.log - logs from **kube-proxy**, which is responsible for directing traffic to Service endpoints

* Ref
* https://kubernetes.io/docs/concepts/cluster-administration/system-logs/
* https://kubernetes.io/docs/tasks/debug/debug-cluster/
* https://kubernetes.io/docs/concepts/cluster-administration/logging/

3. **Cluster-level log**
K8S内各種Event(事件)，比如通過kubebctl create創建Pod後，可以透過kubectl describe pod pod-xxx命令查看Pod的Event訊息
* 內容
* kubectl describe pods ${POD_NAME}
* 位置
* 透過收集各node-level-log方式，彙整到指定位置

* Ref
* https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
* https://www.crowdstrike.com/guides/kubernetes-logging/#:~:text=A%20node%2Dlevel%20logging%20agent%20is%20an%20always%2Don%20service,resource%20constraint%20in%20the%20nodes.

### log收集方式

1. **Container_log直接推送到指定的log收集位置**
https://github.com/kubernetes/website/blob/master/static/images/docs/user-guide/logging/logging-from-application.png
* 直接透過應用程式的SDK設定，但如此會相依於應用程式
* 透過Container執行時，設定Logging Driver，如:fluentd，Container_log（stdout、stderr）直接寫進fluentd中

* Logrotate用途
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-node-level.png
* 設定kubelet進行自動輪巡log進行收集，其一次上限為10Mib(The kubelet sends this information to the container runtime (using CRI), and the runtime writes the container logs to the given location)

---

2. **於Node上透過DaemonSet部署Agent，來收集Node內的log**
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-node-agent.png
* 其Agent的權限要可以進到container內的指定路徑，/var/lib/docker/containers/~
* 對於K8S cluster的log收集應用是較為推薦的，不僅可節省资源，也不影響Container內部運行
* the logs are available through the kubectl command
* 缺點：要求log直接輸出Container的stdout和stderr

---

3. **在Pod使用streaming container(Sidecar)來收集log**
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-streaming-sidecar.png
* 優點，部署簡單
* 缺點，應用程式和Sidecar會寫入相同log，造成空間重覆佔用的狀況，造成浪費；無法用kubectl logs指令查看log

* 對於一開始直接用container回寫log給外部儲存空間，可用Sidecar方式獨立處理log。
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-sidecar-agent.png

##### Ref
* https://blog.51cto.com/u_14035463/5588058
* https://blog.51cto.com/u_7961702/5527104?articleABtest=0
* https://ithelp.ithome.com.tw/articles/10214707

---
##### Reference
* https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
* log指令 https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs
* log說明 https://ithelp.ithome.com.tw/articles/10277646
* log類型 https://blog.51cto.com/u_14035463/5588058
* Monitor https://blog.51cto.com/u_14992974/2547606?articleABtest=0
* log_lifecycle https://github.com/kubernetes/design-proposals-archive/blob/main/node/kubelet-cri-logging.md