---
title: K8S la8(PV & PVC)
tags: k8s
---
K8S la8(PV & PVC)
===
[TOC]
## What is PV、PVC
### PersistentVolume (PV)
PV 是用來管理 cluster 中 storage 的資源，支援像是 NFS、Glusterfs、HostPath等多種存儲插件，擁有獨立的生命週期。
### PersistentVolumeClaim (PVC)
PVC 是針對 PV 請求特定大小、[StorageClass](https://kubernetes.io/docs/concepts/storage/storage-classes/)等等，PVC 是屬於 Namespace 層級的資源。
## How PV、PVC
### 建立 PV
0. 前置作業: 使用 GCP FileStore 作為 NFS Server
1. 點選「建立執行個體」
:::info
:bulb: 提醒: 第一次使用需要先同意使用 FileStore API
:::
2. 完成參數設定
只需要填執行個體ID及檔案共用區名稱，之後點選建立
3. 等待建立完成
4. 完成後查看詳細資料
1. 製作 pv.yml
```yaml=
apiVersion: v1
kind: PersistentVolume
metadata:
name: lab-deposit-log-pv
labels:
claim: lab-deposit-log
spec:
storageClassName: manual
capacity:
storage: 10Gi
accessModes:
- ReadWriteMany
nfs:
path: /nfs_storage
server: 10.144.135.26
```
- 參數說明
- `metadata.labels`: 設定與 PV 連結的參數
- `spec.storageClassName`: 定義 [Storage Class](https://kubernetes.io/docs/concepts/storage/storage-classes/)名稱，讓相同 Storage Class 的 PV 與 PVC 連結
- `spec.capacity`: 此 PV 有多少容量
- `spec.accessModes`: PV 的掛載方式，有三種模式: ReadWriteOnce(RWO), ReadOnlyMany(ROX), ReadWriteMany(RWX) 三種
- ReadWriteOnce: 只能被單個 node 以讀寫方式掛載
- ReadOnlyMany: 複數個 node 以讀方式掛載
- ReadWriteMany: 複數個 node 以讀寫方式掛載
- `spec.nfs`: nfs 放入 nfs server 的連線資訊。可以換成其他插件，像是 hostPath。
2. 將 PV 佈署到 K8s
```shell=
kubectl apply -f <path-to-pv-config>
```
3. 查看 PV
```shell=
kubectl get pv
```
```
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
lab-deposit-log-pv 10Gi RWX Retain Available manual 7s
```
### 建立 PVC
1. 製作 pvc.yml
```yaml=
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-deposit-log-pvc
spec:
storageClassName: manual
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
selector:
matchLabels:
claim: lab-deposit-log
```
- 參數說明
- `spec.accessModes`: 與 PV 設定相同
- `spec.resource.request.storage`: 設定 PVC 所需要的資源量
- `spec.volumeName`: 將 PVC 與 PV 連結所需
- `spec.selector.matchLabels`: 讓 PVC 與特定的 PV 進行連結。
2. 將 PVC 佈署到 K8s
```shell=
kubectl apply -f <path-to-pvc-config> -n <namespace-name>
```
3. 查看 PVC
```shell=
kubectl get pvc -n <namespace-name>
```
```
$ kubectl get pvc -n my-namespace
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
lab-deposit-log-pvc Bound lab-deposit-log-pv 10Gi RWX manual 7s
```
4. 確認與 PV 綁定
```shell=
kubectl get pv
```
```
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
lab-deposit-log-pv 10Gi RWX Retain Bound my-namespace/lab-deposit-log-pvc manual 86s
```
5. 反向測試 - 當 PVC 指定的資源超過 PV 上限
當 cluster 中，沒有 PV 能滿足此 PVC 的請求時，PVC 會不斷處於未綁定的狀態直到 cluster中有符合的 PV 為止。
調整 pv.yml 將 storageClassName 改為 GKE 預設的 standard
```yaml=
apiVersion: v1
kind: PersistentVolume
metadata:
name: lab-deposit-log-pv
labels:
claim: lab-deposit-log
spec:
storageClassName: standard
capacity:
storage: 10Gi
accessModes:
- ReadWriteOnce
nfs:
path: /nfs_storage
server: 10.31.201.122
```
調整 pvc.yml
```yaml=
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-deposit-log-pvc
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 11Gi
selector:
matchLabels:
claim: lab-deposit-log
```
部屬到 K8s 後，查看 pvc 資訊可以發現 pvc 一直處在 Pending 的狀態，而且 GCE 上的 standard 的 StorageClass 不支援自動生成 PV，所以 PVC 就會一直 Pending 住。
```
$ kubectl describe pvc -n my-namespace lab-deposit-log-pvc
Name: lab-deposit-log-pvc
Namespace: my-namespace
StorageClass: standard
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 99s (x190 over 6h52m) persistentvolume-controller Failed to provision volume with StorageClass "standard": claim.Spec.Selector is not supported for dynamic provisioning on GCE
```
6. 測試當 PVC 沒有指定 StorageClass 情境
調整 pvc.yml
```yaml=
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: lab-deposit-log-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 3Gi
selector:
matchLabels:
claim: lab-deposit-log
```
部屬到 K8s 後，查看 pvc 資訊
```
$ kubectl describe pvc -n my-namespace
Name: lab-deposit-log-pvc
Namespace: my-namespace
StorageClass: standard
Status: Pending
Volume:
Labels: <none>
Annotations: volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/gce-pd
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ProvisioningFailed 11s (x8 over 2m39s) persistentvolume-controller Failed to provision volume with StorageClass "standard": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce ReadOnlyMany] are supported
```
會發現當 pvc 沒有指定 StorageClass 下，K8s 會自動給一個 StorageClass 進去，下`kubectl get storageClass`指令查看 K8s 上的 StorageClass
```
$ kubectl get storageClass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
premium-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer true 8h
standard (default) kubernetes.io/gce-pd Delete Immediate true 8h
standard-rwo pd.csi.storage.gke.io Delete WaitForFirstConsumer true 8h
```
可以看到 standard 這個 StorageClass 是 GKE 預設的。
### Pod 與 PVC連結
0. 調整 Dockerfile，並重新包成 image 後上傳至 DockerHub
新增一個`docker-entrypoint.sh`
```shell=
#!/bin/sh
exec java -Dlogging.config=/app/config/logback.xml -jar ${APP_NAME}.jar --spring.config.name=application
```
調整 Dockerfile
```dockerfile=
FROM openjdk:8-jre-alpine
COPY deposit.jar /app/deposit.jar
COPY ./docker-entrypoint.sh /app/docker-entrypoint.sh
ENV APP_NAME deposit
WORKDIR /app
VOLUME /app/config
VOLUME /app/logs
EXPOSE 8080
ENTRYPOINT [ "./docker-entrypoint.sh" ]
```
包成 image 及 推到 DockerHub
```
...請自行練習
```
1. 調整 ConfigMap，將 logback.xml 加進去，並更新到 K8s
```yaml=
apiVersion: v1
kind: ConfigMap
metadata:
name: deposit-config
data:
application.yml: |-
server:
port: 8080
shutdown: graceful
spring:
application:
name: deposit
management:
endpoint:
shutdown:
enabled: true
health:
probes:
enabled: true
endpoints:
web:
exposure:
include: '*'
health:
livenessState:
enabled: true
readinessState:
enabled: true
endpoints:
shutdown:
enabled: true
version: 'v1.0.2'
backend-endpoint: 'http://my-external-service:8080/api/customer'
logback.xml: |-
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<springProperty scope="context" name="serverName" source="HOSTNAME" />
<appender name="console" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>[%d][%p][%t][%C-%L] %m%n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>./logs/app.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>./logs/app.%d{yyyy-MM-dd-HH}.log</fileNamePattern>
</rollingPolicy>
<encoder>
<pattern>[%d][%p][%t][%C-%L] %m%n</pattern>
<charset>utf8</charset>
</encoder>
</appender>
<logger name="org.hibernate.SQL" additivity="false" >
<level value="info" />
<appender-ref ref="console" />
</logger>
<root level="info">
<appender-ref ref="file" />
</root>
</configuration>
```
2. Deployment 中，pvc 相關設定範例
```yaml=
spec:
volumes:
- name: app-log-storage
persistentVolumeClaim:
claimName: lab-deposit-log-pvc
containers:
volumeMounts:
- name: app-log-storage
mountPath: /app/logs
```
3. 調整 Deployment，將 pvc 相關參數放進去後，重新佈署
```yaml=
apiVersion: apps/v1
kind: Deployment
metadata:
name: lab-my-deposit
spec:
replicas: 1
selector:
matchLabels:
app: lab-my-deposit
template:
metadata:
labels:
app: lab-my-deposit
spec:
containers:
- name: my-deposit
image: <dockerhub-id>/<image-name>:<version>
ports:
- containerPort: 8080
volumeMounts:
- mountPath: /app/config/application.yml
subPath: application.yml
name: app-config
readOnly: true
- mountPath: /app/config/logback.xml
subPath: logback.xml
name: app-config
readOnly: true
- mountPath: /app/logs
name: app-log-storage
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTP
initialDelaySeconds: 20
periodSeconds: 5
timeoutSeconds: 60
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTP
initialDelaySeconds: 20
timeoutSeconds: 30
periodSeconds: 10
successThreshold: 1
failureThreshold: 5
resources:
requests:
memory: "600Mi"
cpu: "300m"
limits:
memory: "600Mi"
cpu: "1"
volumes:
- name: app-config
configMap:
name: deposit-config
- name: app-log-storage
persistentVolumeClaim:
claimName: lab-deposit-log-pvc
```
4. 進入 deposit pod 查看
```shell=
kubectl exec -n <namespace-name> -it <pod-name> sh
```
```
$ kubectl exec -n my-namespace -it lab-my-deposit-7b6f9cbc44-6b8dd sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
/app # cd logs
/app/logs # ls -lSt
total 72
-rw-r--r-- 1 root root 31954 Apr 6 09:18 app.log
-rw-r--r-- 1 root root 16713 Apr 6 09:00 app.2022-04-06-08.log
drwx------ 2 root root 16384 Apr 6 02:11 lost+found
```
5. 查看 log
```
/app # cat app.2022-04-06-08.log
[2022-04-06 08:51:18,169][INFO][main][org.springframework.boot.StartupInfoLogger-55] Starting DepositApplication v2.6.3 using Java 1.8.0_212 on lab-my-deposit-7b6f9cbc44-6b8dd with PID 1 (/app/deposit.jar started by root in /app)
[2022-04-06 08:51:18,227][INFO][main][org.springframework.boot.SpringApplication-637] No active profile set, falling back to default profiles: default
[2022-04-06 08:51:23,824][INFO][main][org.springframework.boot.web.embedded.tomcat.TomcatWebServer-108] Tomcat initialized with port(s): 8080 (http)
[2022-04-06 08:51:23,856][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Initializing ProtocolHandler ["http-nio-8080"]
[2022-04-06 08:51:23,858][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting service [Tomcat]
[2022-04-06 08:51:23,859][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting Servlet engine: [Apache Tomcat/9.0.56]
[2022-04-06 08:51:24,071][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Initializing Spring embedded WebApplicationContext
[2022-04-06 08:51:24,072][INFO][main][org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext-290] Root WebApplicationContext: initialization completed in 5626 ms
[2022-04-06 08:51:38,141][INFO][main][org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver-58] Exposing 14 endpoint(s) beneath base path '/actuator'
[2022-04-06 08:51:38,267][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting ProtocolHandler ["http-nio-8080"]
[2022-04-06 08:51:38,351][INFO][main][org.springframework.boot.web.embedded.tomcat.TomcatWebServer-220] Tomcat started on port(s): 8080 (http) with context path ''
[2022-04-06 08:51:38,431][INFO][main][org.springframework.boot.StartupInfoLogger-61] Started DepositApplication in 22.973 seconds (JVM running for 24.578)
[2022-04-06 08:51:39,743][INFO][http-nio-8080-exec-2][org.apache.juli.logging.DirectJDKLog-173] Initializing Spring DispatcherServlet 'dispatcherServlet'
[2022-04-06 08:51:39,743][INFO][http-nio-8080-exec-2][org.springframework.web.servlet.FrameworkServlet-525] Initializing Servlet 'dispatcherServlet'
[2022-04-06 08:51:39,745][INFO][http-nio-8080-exec-2][org.springframework.web.servlet.FrameworkServlet-547] Completed initialization in 2 ms
[2022-04-06 08:51:39,944][INFO][http-nio-8080-exec-2][com.webcomm.helper.HealthProvider-16] UP
...中間略
[com.webcomm.helper.HealthProvider-16] UP
```
### 驗證是否存進 nfs server
1. 將 net-tool 加上 pvc 後重新佈署
2. 進到 net-tool 查看 /app/logs
```
/app # cd logs
/app # ls -lSt
total 60
-rw-r--r-- 1 root root 22824 Apr 6 09:13 app.log
-rw-r--r-- 1 root root 16713 Apr 6 09:00 app.2022-04-06-08.log
drwx------ 2 root root 16384 Apr 6 02:11 lost+found
```
3. 選一個 log 查看是否為 deposit 的 log
```
/app # cat app.2022-04-06-08.log
[2022-04-06 08:51:18,169][INFO][main][org.springframework.boot.StartupInfoLogger-55] Starting DepositApplication v2.6.3 using Java 1.8.0_212 on lab-my-deposit-7b6f9cbc44-6b8dd with PID 1 (/app/deposit.jar started by root in /app)
[2022-04-06 08:51:18,227][INFO][main][org.springframework.boot.SpringApplication-637] No active profile set, falling back to default profiles: default
[2022-04-06 08:51:23,824][INFO][main][org.springframework.boot.web.embedded.tomcat.TomcatWebServer-108] Tomcat initialized with port(s): 8080 (http)
[2022-04-06 08:51:23,856][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Initializing ProtocolHandler ["http-nio-8080"]
[2022-04-06 08:51:23,858][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting service [Tomcat]
[2022-04-06 08:51:23,859][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting Servlet engine: [Apache Tomcat/9.0.56]
[2022-04-06 08:51:24,071][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Initializing Spring embedded WebApplicationContext
[2022-04-06 08:51:24,072][INFO][main][org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext-290] Root WebApplicationContext: initialization completed in 5626 ms
[2022-04-06 08:51:38,141][INFO][main][org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver-58] Exposing 14 endpoint(s) beneath base path '/actuator'
[2022-04-06 08:51:38,267][INFO][main][org.apache.juli.logging.DirectJDKLog-173] Starting ProtocolHandler ["http-nio-8080"]
[2022-04-06 08:51:38,351][INFO][main][org.springframework.boot.web.embedded.tomcat.TomcatWebServer-220] Tomcat started on port(s): 8080 (http) with context path ''
[2022-04-06 08:51:38,431][INFO][main][org.springframework.boot.StartupInfoLogger-61] Started DepositApplication in 22.973 seconds (JVM running for 24.578)
[2022-04-06 08:51:39,743][INFO][http-nio-8080-exec-2][org.apache.juli.logging.DirectJDKLog-173] Initializing Spring DispatcherServlet 'dispatcherServlet'
[2022-04-06 08:51:39,743][INFO][http-nio-8080-exec-2][org.springframework.web.servlet.FrameworkServlet-525] Initializing Servlet 'dispatcherServlet'
[2022-04-06 08:51:39,745][INFO][http-nio-8080-exec-2][org.springframework.web.servlet.FrameworkServlet-547] Completed initialization in 2 ms
[2022-04-06 08:51:39,944][INFO][http-nio-8080-exec-2][com.webcomm.helper.HealthProvider-16] UP
...中間略
[com.webcomm.helper.HealthProvider-16] UP
```
確認為 deposit 的 log，完成驗證
### 刪除 PV & PVC
- 刪除 PV
```shell=
kubectl delete pv <pv-name>
```
- 刪除 PVC
```shell=
kubectl delete pvc -n <namespace-name> <pvc-name>
```
:::danger
:bulb: 提醒: 測完後記得將 nfs-server 刪除，等要用得時候再建!!!
:::
## 參考資料
- [Persistent Volumes](https://kubernetes.io/docs/concepts/storage/persistent-volumes/)
- [GKE-Persistent volumes and dynamic provisioning](https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes)
- [Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/)
- [FileStore-Creating instances](https://cloud.google.com/filestore/docs/creating-instances#cloud-console)
Sign in with Wallet
Connect another wallet