---
# System prepended metadata

title: Esun_C.N._logging & Monitoring

---

# Esun_C.N._logging & Monitoring

## CKA課程
### 01-Logging-and-Monitoring-Section-Introduction

### 02-Monitor-Cluster-Components
透過Metrics Server監控 cluster components，透過Node內kubelet，執行API回傳需監控的資訊給Metrics Server，再透過metrics api回傳給kubectl top呈現監控結果。

* 圖1
https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-metrics-pipeline/

* 操作步驟
    * 從git下載 `$ git clone https://github.com/kubernetes-incubator/metrics-server.git`
    * 部屬Server `$ kubectl create -f metric-server/deploy/1.8+/`
    * 下指令檢視資源使用狀況(CPU、Memory、Disk)
        `$ kubectl top node`
        `$ kubectl top pod`

* 其他工具監控，如:Prometheus、Elastic_Stack、DataDog、dynatrace
* Ref
    * https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/04-Logging-and-Monitoring/02-Monitor-Cluster-Components.md

### 03-Practice-Test-Monitor-Cluster-Components
N/A

### 04-Managing-Application-Logs
* Docker log
    * 透過`$ docker logs -f ecf` 查詢log資訊
    
* K8S log
    * 可用`$ kubectl logs -f <pod-name> <container-name>`查詢指定pod內的container log
        例如`$ kubectl logs -f even-simulator-pod event-simulator` 

![圖2](https://i.imgur.com/VP2MO3X.png)
https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts

##### Ref:
* https://github.com/kodekloudhub/certified-kubernetes-administrator-course/blob/master/docs/04-Logging-and-Monitoring/04-Managing-Application-Logs.md
* https://blog.aquasec.com/kubernetes-security-pod-escape-log-mounts

### 05-Download-Presentation-Deck
N/A
### 06-Practice-Test-Managing-Application-Logs
N/A

---
## 目的
* 除錯
* 監控cluster行為

---
## 內容
* Kubernetes defaults to the "json-file" logging driver, in which docker writes the stdout/stderr streams to a file in the json format as shown below
> {“log”: “The actual log line”, “stream”: “stderr”, “time”: “2016-10-05T00:00:30.082640485Z”}

* Ref
https://github.com/kubernetes/design-proposals-archive/blob/main/node/kubelet-cri-logging.md

## logging方式
* Logging
    * 做法與傳統VM不一樣的部分為，Container內的log會因Container被回收而log也會清掉，會額外傳至特定位置儲存
![方式1]()
* Container Logging
* Node Logging
* Cluster Logging
* Event logging
    * `kubectl get events -n <namespace>`
    * `kubectl describe pod <pod-name>`
* Audit logging
    * Kubernetes audit logs are detailed descriptions of each call made to the kube-apiserver
* Ingress Logging

* Ref
    * https://sematext.com/guides/kubernetes-logging/

## log類型
1. **Application log**,例如:Nginx、Tomcat log


2. **Node-level log**，例如:Kubernetes自身各大组件的log（包括 kubelet、kube-proxy 等），容器運行的log（比如 Docker），存在/var/log內

    * 位置 
    ![類型1]()
    https://yashbindlish.medium.com/under-the-hood-an-introduction-to-kubernetes-architecture-bb9d8599f837
        * **Control Plane nodes**(Master Node)
            /var/log/kube-apiserver.log - **API Server**, responsible for serving the API
            
            /var/log/kube-scheduler.log - **Scheduler**, responsible for making scheduling decisions
            
            /var/log/kube-controller-manager.log - a component that runs most Kubernetes built-in controllers, with the notable exception of scheduling (the kube-scheduler handles scheduling).

        * **Worker Nodes**
            /var/log/kubelet.log - logs from the **kubelet**, responsible for running containers on the node
            /var/log/kube-proxy.log - logs from **kube-proxy**, which is responsible for directing traffic to Service endpoints

    * Ref
        * https://kubernetes.io/docs/concepts/cluster-administration/system-logs/
        * https://kubernetes.io/docs/tasks/debug/debug-cluster/
        * https://kubernetes.io/docs/concepts/cluster-administration/logging/


3. **Cluster-level log** 
K8S内各種Event(事件)，比如通過kubebctl create創建Pod後，可以透過kubectl describe pod pod-xxx命令查看Pod的Event訊息
    * 內容
        * kubectl describe pods ${POD_NAME}
    * 位置
        * 透過收集各node-level-log方式，彙整到指定位置

    * Ref
        * https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
        * https://www.crowdstrike.com/guides/kubernetes-logging/#:~:text=A%20node%2Dlevel%20logging%20agent%20is%20an%20always%2Don%20service,resource%20constraint%20in%20the%20nodes.
        


### log收集方式

1. **Container_log直接推送到指定的log收集位置**
![收集1](https://i.imgur.com/ydG0W7m.png)
https://github.com/kubernetes/website/blob/master/static/images/docs/user-guide/logging/logging-from-application.png
    * 直接透過應用程式的SDK設定，但如此會相依於應用程式
    * 透過Container執行時，設定Logging Driver，如:fluentd，Container_log（stdout、stderr）直接寫進fluentd中

* Logrotate用途
![收集2](https://i.imgur.com/6wvt95V.png)
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-node-level.png
    * 設定kubelet進行自動輪巡log進行收集，其一次上限為10Mib(The kubelet sends this information to the container runtime (using CRI), and the runtime writes the container logs to the given location)

---

2. **於Node上透過DaemonSet部署Agent，來收集Node內的log**
![收集3](https://i.imgur.com/daMQwLS.png)
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-node-agent.png
    * 其Agent的權限要可以進到container內的指定路徑，/var/lib/docker/containers/~
    * 對於K8S cluster的log收集應用是較為推薦的，不僅可節省资源，也不影響Container內部運行
    * the logs are available through the kubectl command
    * 缺點：要求log直接輸出Container的stdout和stderr

---

3. **在Pod使用streaming container(Sidecar)來收集log**
![收集4](https://i.imgur.com/eJEoDCT.png)
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-streaming-sidecar.png
    * 優點，部署簡單
    * 缺點，應用程式和Sidecar會寫入相同log，造成空間重覆佔用的狀況，造成浪費；無法用kubectl logs指令查看log
    
* 對於一開始直接用container回寫log給外部儲存空間，可用Sidecar方式獨立處理log。
![收集5](https://i.imgur.com/SBsSXQ1.png)
https://github.com/kubernetes/website/blob/main/static/images/docs/user-guide/logging/logging-with-sidecar-agent.png

##### Ref
* https://blog.51cto.com/u_14035463/5588058
* https://blog.51cto.com/u_7961702/5527104?articleABtest=0
* https://ithelp.ithome.com.tw/articles/10214707


---
##### Reference
* https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
* log指令  https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#logs
* log說明 https://ithelp.ithome.com.tw/articles/10277646
* log類型 https://blog.51cto.com/u_14035463/5588058
* Monitor https://blog.51cto.com/u_14992974/2547606?articleABtest=0
* log_lifecycle https://github.com/kubernetes/design-proposals-archive/blob/main/node/kubelet-cri-logging.md
