# K8s Ephemeral Storage
* 在每個 k8s node 上的 kubelet 在運行時，總是會需要儲存一些資料在本地端，因此以下的其中兩個空間就會被拿來利用：
1. kubelet root directory: `/var/lib/kubelet`
2. log directory: `/var/log/pods`
* 由於被 kubelet 管理到的部份，例如：pod log, image layer, container writable latyer，甚至是在 pod spec 中定義 emptyDir，都會利用到上面提到的這兩個空間，而 ephemeral-storage 就是針對上述這兩個空間進行管理。
* 如果 pod 有使用 emptyDir，那麼在沒有重啟這個 pod 的情況下會一直使用 `/var/lib/kubelet`，而 ephemeral-storage 可以有效限制這個空間。
## 實作
* 設定的單位跟 memory 其實一樣，可以是單一字母的 E, P, T, G, M, K，也可以是雙字母的 Ei, Pi, Ti, Gi, Mi, Ki，以下是一個簡單的設定範例：
* 由於帶有 ephemeral-storage limit，因此 k8s 就會持續監控 pod 中的 container 的磁碟使用狀況是否有超過設定限制，pod 就會被趕走。
```
apiVersion: v1
kind: Pod
metadata:
name: ephemeral-test
spec:
# 整個 pod 一共要求 4G 的 ephemeral storage 空間
# 以及使用 ephemeral storage 的空間限制為 8G
containers:
- name: db
image: docker.io/taiwanese/mydb
resources:
# db 要求 2G 的 ephemeral storage 空間
requests:
ephemeral-storage: "2Gi"
# db 使用 ephemeral storage 的空間限制為 4G
limits:
ephemeral-storage: "4Gi"
- name: wp
image: wordpress
resources:
# wp 要求 2G 的 ephemeral storage 空間
requests:
ephemeral-storage: "2Gi"
# wp 使用 ephemeral storage 的空間限制為 4G
limits:
ephemeral-storage: "4Gi"
```
* 進到 db container 並產生 5g 檔案
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ephemeral-test 2/2 Running 0 8s 10.42.5.236 cilium-m3 <none> <none>
$ kubectl exec -it ephemeral-test -c db -- bash
$ dd if=/dev/zero of=./5GB_file bs=1G count=5
```
* 因為 db container 超過了 4Gi 的空間 limit 因此被 kubelet stop 驅離了
```
$ kubectl describe po ephemeral-test
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 96s default-scheduler Successfully assigned default/ephemeral-test to cilium-m3
Normal Pulling 92s kubelet Pulling image "docker.io/taiwanese/mydb"
Normal Pulled 91s kubelet Successfully pulled image "docker.io/taiwanese/mydb" in 1.482s (1.482s including waiting)
Normal Created 91s kubelet Created container db
Normal Started 90s kubelet Started container db
Normal Pulling 90s kubelet Pulling image "wordpress"
Normal Pulled 89s kubelet Successfully pulled image "wordpress" in 1.68s (1.68s including waiting)
Normal Created 88s kubelet Created container wp
Normal Started 88s kubelet Started container wp
Warning Evicted 5s kubelet Container db exceeded its local ephemeral storage limit "4Gi".
Normal Killing 5s kubelet Stopping container db
Normal Killing 5s kubelet Stopping container wp
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ephemeral-test 0/2 Completed 0 2m59s 10.42.5.236 cilium-m3 <none> <none>
```
* 設定 `emptyDir.sizeLimit` 限制 emptyDir 大小
```
# 整個 pod 一共要求 4G 的 ephemeral storage 空間
# 以及使用 ephemeral storage 的空間限制為 8G
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 1
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- name: db
image: docker.io/taiwanese/mydb
resources:
requests:
ephemeral-storage: "2Gi" # db 要求 2G 的 ephemeral storage 空間
limits:
ephemeral-storage: "4Gi" # db 使用 ephemeral storage 的空間限制為 4G
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
- name: wp
image: wordpress
resources:
requests:
ephemeral-storage: "2Gi" # wp 要求 2G 的 ephemeral storage 空間
limits:
ephemeral-storage: "4Gi" # wp 使用 ephemeral storage 的空間限制為 4G
volumeMounts:
- name: ephemeral
mountPath: "/tmp"
volumes:
- name: ephemeral
emptyDir:
sizeLimit: "8Gi" # 設定整個 Pod 的 emptyDir storage 總限制為 8Gi
```
* 進到 db container 並在 `/tmp` 產生 8g 檔案
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
frontend-7bc6468c99-4n4p6 2/2 Running 0 16s 10.42.5.237 cilium-m3 <none> <none>
$ kubectl exec -it frontend-7bc6468c99-4n4p6 -c db -- bash
$ dd if=/dev/zero of=/tmp/8GB_file bs=1G count=8
```
* 由於 `emptyDir` 使用量超過了 8g limit 因此 pod 被驅離了
```
$ kubectl describe po frontend-7bc6468c99-4n4p6
......
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m57s default-scheduler Successfully assigned default/frontend-7bc6468c99-4n4p6 to cilium-m3
Normal Pulling 5m55s kubelet Pulling image "docker.io/taiwanese/mydb"
Normal Pulled 5m53s kubelet Successfully pulled image "docker.io/taiwanese/mydb" in 1.999s (1.999s including waiting)
Normal Created 5m53s kubelet Created container db
Normal Started 5m52s kubelet Started container db
Normal Pulling 5m52s kubelet Pulling image "wordpress"
Normal Pulled 5m50s kubelet Successfully pulled image "wordpress" in 2.044s (2.044s including waiting)
Normal Created 5m50s kubelet Created container wp
Normal Started 5m49s kubelet Started container wp
Warning Evicted 16s kubelet Pod ephemeral local storage usage exceeds the total limit of containers 8Gi.
Normal Killing 16s kubelet Stopping container db
Normal Killing 16s kubelet Stopping container wp
```
* 原本的 pod 會被停止
```
$ kubectl get po -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
frontend-7bc6468c99-4n4p6 0/2 Completed 0 7m9s 10.42.5.237 cilium-m3 <none> <none>
frontend-7bc6468c99-ms5bk 2/2 Running 0 81s 10.42.5.18 cilium-m3 <none> <none>
```
## 參考
https://godleon.github.io/blog/Kubernetes/k8s-Scheduling-Manage-Compute-Resource-for-Container/
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
Sign in with Wallet
Connect another wallet