---
# System prepended metadata

title: K8s Ephemeral Storage
tags: [K8S]

---

# K8s Ephemeral Storage

* 在每個 k8s node 上的 kubelet 在運行時，總是會需要儲存一些資料在本地端，因此以下的其中兩個空間就會被拿來利用：

1. kubelet root directory: `/var/lib/kubelet`
2. log directory: `/var/log/pods`

* 由於被 kubelet 管理到的部份，例如：pod log, image layer, container writable latyer，甚至是在 pod spec 中定義 emptyDir，都會利用到上面提到的這兩個空間，而 ephemeral-storage 就是針對上述這兩個空間進行管理。
* 如果 pod 有使用 emptyDir，那麼在沒有重啟這個 pod 的情況下會一直使用 `/var/lib/kubelet`，而 ephemeral-storage 可以有效限制這個空間。


## 實作
* 設定的單位跟 memory 其實一樣，可以是單一字母的 E, P, T, G, M, K，也可以是雙字母的 Ei, Pi, Ti, Gi, Mi, Ki，以下是一個簡單的設定範例：
* 由於帶有 ephemeral-storage limit，因此 k8s 就會持續監控 pod 中的 container 的磁碟使用狀況是否有超過設定限制，pod 就會被趕走。

```
apiVersion: v1
kind: Pod
metadata:
  name: ephemeral-test
spec:
  # 整個 pod 一共要求 4G 的 ephemeral storage 空間
  # 以及使用 ephemeral storage 的空間限制為 8G
  containers:
  - name: db
    image: docker.io/taiwanese/mydb
    resources:
      # db 要求 2G 的 ephemeral storage 空間
      requests:
        ephemeral-storage: "2Gi"
      # db 使用 ephemeral storage 的空間限制為 4G
      limits:
        ephemeral-storage: "4Gi"
  - name: wp
    image: wordpress
    resources:
      # wp 要求 2G 的 ephemeral storage 空間
      requests:
        ephemeral-storage: "2Gi"
      # wp 使用 ephemeral storage 的空間限制為 4G
      limits:
        ephemeral-storage: "4Gi"
```

* 進到 db container 並產生 5g 檔案
```
$ kubectl get po -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
ephemeral-test              2/2     Running   0          8s    10.42.5.236   cilium-m3   <none>           <none>

$ kubectl exec -it ephemeral-test -c db -- bash

$ dd if=/dev/zero of=./5GB_file bs=1G count=5
```
* 因為 db container 超過了 4Gi 的空間 limit 因此被 kubelet stop 驅離了
```
$ kubectl describe po ephemeral-test
......
Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  96s   default-scheduler  Successfully assigned default/ephemeral-test to cilium-m3
  Normal   Pulling    92s   kubelet            Pulling image "docker.io/taiwanese/mydb"
  Normal   Pulled     91s   kubelet            Successfully pulled image "docker.io/taiwanese/mydb" in 1.482s (1.482s including waiting)
  Normal   Created    91s   kubelet            Created container db
  Normal   Started    90s   kubelet            Started container db
  Normal   Pulling    90s   kubelet            Pulling image "wordpress"
  Normal   Pulled     89s   kubelet            Successfully pulled image "wordpress" in 1.68s (1.68s including waiting)
  Normal   Created    88s   kubelet            Created container wp
  Normal   Started    88s   kubelet            Started container wp
  Warning  Evicted    5s    kubelet            Container db exceeded its local ephemeral storage limit "4Gi".
  Normal   Killing    5s    kubelet            Stopping container db
  Normal   Killing    5s    kubelet            Stopping container wp

$ kubectl get po -owide
NAME                        READY   STATUS      RESTARTS   AGE     IP            NODE        NOMINATED NODE   READINESS GATES
ephemeral-test              0/2     Completed   0          2m59s   10.42.5.236   cilium-m3   <none>           <none>
```



* 設定 `emptyDir.sizeLimit` 限制 emptyDir 大小
```
# 整個 pod 一共要求 4G 的 ephemeral storage 空間
# 以及使用 ephemeral storage 的空間限制為 8G
apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 1
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: db
        image: docker.io/taiwanese/mydb
        resources:
          requests:
            ephemeral-storage: "2Gi"   # db 要求 2G 的 ephemeral storage 空間
          limits:
            ephemeral-storage: "4Gi"  # db 使用 ephemeral storage 的空間限制為 4G
        volumeMounts:
        - name: ephemeral
          mountPath: "/tmp"
      - name: wp
        image: wordpress
        resources:
          requests:
            ephemeral-storage: "2Gi"  # wp 要求 2G 的 ephemeral storage 空間
          limits:
            ephemeral-storage: "4Gi"  # wp 使用 ephemeral storage 的空間限制為 4G
        volumeMounts:
        - name: ephemeral
          mountPath: "/tmp"
      volumes:
        - name: ephemeral
          emptyDir:
            sizeLimit: "8Gi" # 設定整個 Pod 的 emptyDir storage 總限制為 8Gi
```
* 進到 db container 並在 `/tmp` 產生 8g 檔案
```
$ kubectl get po -owide
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE        NOMINATED NODE   READINESS GATES
frontend-7bc6468c99-4n4p6   2/2     Running   0          16s   10.42.5.237   cilium-m3   <none>           <none>

$ kubectl exec -it frontend-7bc6468c99-4n4p6 -c db  -- bash

$ dd if=/dev/zero of=/tmp/8GB_file bs=1G count=8
```

* 由於 `emptyDir` 使用量超過了 8g limit 因此 pod 被驅離了 
```
$ kubectl describe po frontend-7bc6468c99-4n4p6
......
Events:
  Type     Reason     Age    From               Message
  ----     ------     ----   ----               -------
  Normal   Scheduled  5m57s  default-scheduler  Successfully assigned default/frontend-7bc6468c99-4n4p6 to cilium-m3
  Normal   Pulling    5m55s  kubelet            Pulling image "docker.io/taiwanese/mydb"
  Normal   Pulled     5m53s  kubelet            Successfully pulled image "docker.io/taiwanese/mydb" in 1.999s (1.999s including waiting)
  Normal   Created    5m53s  kubelet            Created container db
  Normal   Started    5m52s  kubelet            Started container db
  Normal   Pulling    5m52s  kubelet            Pulling image "wordpress"
  Normal   Pulled     5m50s  kubelet            Successfully pulled image "wordpress" in 2.044s (2.044s including waiting)
  Normal   Created    5m50s  kubelet            Created container wp
  Normal   Started    5m49s  kubelet            Started container wp
  Warning  Evicted    16s    kubelet            Pod ephemeral local storage usage exceeds the total limit of containers 8Gi.
  Normal   Killing    16s    kubelet            Stopping container db
  Normal   Killing    16s    kubelet            Stopping container wp
```
* 原本的 pod 會被停止
```
$ kubectl get po -owide
NAME                        READY   STATUS      RESTARTS   AGE    IP            NODE        NOMINATED NODE   READINESS GATES
frontend-7bc6468c99-4n4p6   0/2     Completed   0          7m9s   10.42.5.237   cilium-m3   <none>           <none>
frontend-7bc6468c99-ms5bk   2/2     Running     0          81s    10.42.5.18    cilium-m3   <none>           <none>
```


## 參考
https://godleon.github.io/blog/Kubernetes/k8s-Scheduling-Manage-Compute-Resource-for-Container/

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/