# k8s 創建 StatefulSet 不使用 storageclass 創建 pv/pvc

## 問題背景

客戶環境不允許使用任何的 `storageclass` ，但需要使用 StatefulSet 創建應用，並且資料要做永存，該如何解決？

## 解決辦法

1. 透過 k8s 原生的 `pv/pvc` 功能，並且 pv 使用 `local` 的方式創建，並讓 StatefulSet 的 pod 使用。
2. 由於 `pv/pvc` 是透過宣告的 `storage` 大小來綁定，而那麼如果一次創建多個 `pv/pvc` 都是相同的 `storage` 大小可能會導致 `pv/pvc` 互相綁定錯誤。所以 pvc 需要透過 `label selector` 找到對應的 pv。
3. 了解 StatefulSet 預設建立 pvc 的名稱規則是 `<volumeClaimTemplate.metadata.name>-<statefulset.metadata.name>-<ordinal>` ，因此透過宣告 StatefulSet 的 `<volumeClaimTemplate.metadata.name>` 就可以找到對應的 pvc。
4. `storageclass` 需設為空值，如果不是空直 StatefulSet 會使用 default 的 `storageclass` 創建 pvc。

### 實作
* 在每個節點上創建目錄

$ sudo mkdir /opt/local

* 創建三個對應的 pv/pvc
- pvc 透過 `label selector` 找到對應的 pv
- 創建的 pv 是使用 `local` 因此要更改自己的節點名稱

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: html-storage-nginx-0
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      my: demo-0
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-local-0
  labels:
    my: demo-0
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/opt/local"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master1
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: html-storage-nginx-1
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      my: demo-1
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-local-1
  labels:
    my: demo-1
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/opt/local"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: html-storage-nginx-2
spec:
  storageClassName: ""
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  selector:
    matchLabels:
      my: demo-2
---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-local-2
  labels:
    my: demo-2
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  local:
    path: "/opt/local"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - master3

* 確認各自對應的 pv,pvc 互相 bound

$ kubectl get pvc,pv
NAME                                         STATUS   VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/html-storage-nginx-0   Bound    pv-local-0   1Gi        RWO                           <unset>                 32m
persistentvolumeclaim/html-storage-nginx-1   Bound    pv-local-1   1Gi        RWO                           <unset>                 32m
persistentvolumeclaim/html-storage-nginx-2   Bound    pv-local-2   1Gi        RWO                           <unset>                 32m

NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                          STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pv-local-0   1Gi        RWO            Retain           Bound    default/html-storage-nginx-0                  <unset>                          32m
persistentvolume/pv-local-1   1Gi        RWO            Retain           Bound    default/html-storage-nginx-1                  <unset>                          32m
persistentvolume/pv-local-2   1Gi        RWO            Retain           Bound    default/html-storage-nginx-2                  <unset>                          32m

* 建立 nginx StatefulSet 並且不宣告 `storageClassName`。
* StatefulSet 預設建立 pvc 的名稱規則是 `<volumeClaimTemplate.metadata.name>-<statefulset.metadata.name>-<ordinal>` 透過這個命名方式來找到我們創建的 pvc 名稱。
* 因此透過宣告 `<volumeClaimTemplate.metadata.name>` 就可以找到對應的 pvc。

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: quay.io/cooloo9871/nginx
          volumeMounts:
            - name: html-storage
              mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
    - metadata:
        name: html-storage
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: ""

## 驗證
* 確認 nginx-0 的 pod 是在 `master1` node 上，代表他是使用 `html-storage-nginx-0` 這個 pvc

$ kubectl get po -owide
NAME      READY   STATUS    RESTARTS   AGE   IP              NODE      NOMINATED NODE   READINESS GATES
nginx-0   1/1     Running   0          29s   10.244.137.68   master1   <none>           <none>
nginx-1   1/1     Running   0          27s   10.244.180.3    master2   <none>           <none>
nginx-2   1/1     Running   0          16s   10.244.136.2    master3   <none>           <none>

$ kubectl describe pod nginx-0 | grep -A4 Volumes
Volumes:
  html-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  html-storage-nginx-0
    ReadOnly:   false