# MinIO & s3fs & Directpv
* 透過 s3fs 搭配 DaemonSet 會將 minio 的 Buckets 掛載到每一台 node 上的目錄區，因此後續 pod 就可以透過 hostpath 使用 volume，並且允許漂移到不同節點上。
* minio 實際會透過 Directpv 存储，直接將資料寫到硬碟。

## 事前準備
* 需準備 1m2w k8s cluster，本次範例 k8s 架構 1m3w
* 每台 worker node 需要額外再加兩顆 Disk
## 部屬 Directpv
* Directpv 目的就是要來取代 hostpath，讓資料可以直接儲存到 Disk(SSD、Nvme)，可以提高 IO 效能。
* 主要提供給物件儲存、資料庫和需要高效能環境而設計的。
* 安裝 krew

$ curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/download/v0.4.4/krew-linux_amd64.tar.gz"
      
$ tar zxvf krew-linux_amd64.tar.gz 

$ ./krew-linux_amd64 install krew 

$ echo 'export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH"' | sudo tee -a /etc/profile

$ rm krew-linux_amd64.tar.gz krew-linux_amd64 LICENSE

* 重新 login 後測試

$ kubectl krew
krew is the kubectl plugin manager.
You can invoke krew through kubectl: "kubectl krew [command]..."

Usage:
  kubectl krew [command]

Available Commands:
  help        Help about any command
  index       Manage custom plugin indexes
  info        Show information about an available plugin
  install     Install kubectl plugins
  list        List installed kubectl plugins
  search      Discover kubectl plugins
  uninstall   Uninstall plugins
  update      Update the local copy of the plugin index
  upgrade     Upgrade installed plugins to newer versions
  version     Show krew version and diagnostics

Flags:
  -h, --help      help for krew
  -v, --v Level   number for the log level verbosity

Use "kubectl krew [command] --help" for more information about a command.

* 透過 krew 安裝 directpv plugin

$ kubectl krew install directpv

* 安裝 directpv 到 k8s

$ kubectl directpv install

* 顯示安裝完成畫面

$ kubectl directpv info

* directpv 運作流程

# 可自行調整 controller 數量
$ kubectl -n directpv get all
NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-58df549f48-d288q   3/3     Running   0          178m
pod/controller-58df549f48-mqr9g   3/3     Running   0          178m
pod/controller-58df549f48-s8xnd   3/3     Running   0          178m
pod/node-server-7zhz9             4/4     Running   0          178m
pod/node-server-f9l2k             4/4     Running   0          178m
pod/node-server-ljpgh             4/4     Running   0          178m
pod/node-server-zpww7             4/4     Running   0          178m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/node-server   4         4         4       4            4           <none>          178m

NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   3/3     3            3           178m

NAME                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/controller-58df549f48   3         3         3       178m


$ kubectl get sc
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io      directpv-min-io         Delete          WaitForFirstConsumer   true                   3h

* 檢查 DirectPV discover driver

$ kubectl directpv discover --nodes=cilium-w{1...3} --drives=sd{b...f}

 Discovered node 'cilium-w3' ✔
 Discovered node 'cilium-w1' ✔
 Discovered node 'cilium-w2' ✔

┌─────────────────────┬───────────┬───────┬────────┬────────────┬────────────────────┬───────────┬─────────────┐
│ ID                  │ NODE      │ DRIVE │ SIZE   │ FILESYSTEM │ MAKE               │ AVAILABLE │ DESCRIPTION │
├─────────────────────┼───────────┼───────┼────────┼────────────┼────────────────────┼───────────┼─────────────┤
│ 8:48$jHViBmYbA0Q... │ cilium-w1 │ sdd   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
│ 8:64$fW3WQftFoXo... │ cilium-w1 │ sde   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
│ 8:32$QQwAiXmZknC... │ cilium-w2 │ sdc   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
│ 8:48$NKcJ3N3TaE8... │ cilium-w2 │ sdd   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
│ 8:16$EHcAYec/Wa7... │ cilium-w3 │ sdb   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
│ 8:32$xrd5+hDeqje... │ cilium-w3 │ sdc   │ 30 GiB │ -          │ QEMU QEMU_HARDDISK │ YES       │ -           │
└─────────────────────┴───────────┴───────┴────────┴────────────┴────────────────────┴───────────┴─────────────┘

Generated 'drives.yaml' successfully.

* DirectPV add driver

$ cat drives.yaml
version: v1
nodes:
    - name: cilium-w1
      drives:
        - id: 8:64$fW3WQftFoXorsSbSQFO4UvvRi6OqxYkAe+Bxa/F41PY=
          name: sde
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"
        - id: 8:48$jHViBmYbA0Q3zrvycywUHNO6SuGJCHclmeBVwHeHVXk=
          name: sdd
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"
    - name: cilium-w2
      drives:
        - id: 8:48$NKcJ3N3TaE8/14aOuJeeqrjLmf3y4sgWFO3L0dm7EbQ=
          name: sdd
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"
        - id: 8:32$QQwAiXmZknCllNQJfYoTt4AxcUIR4EsNl6MCJ5nJIQk=
          name: sdc
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"
    - name: cilium-w3
      drives:
        - id: 8:32$xrd5+hDeqje4AlsYeMKeAu840b3FOoPnfZQ7ZsVrRYQ=
          name: sdc
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"
        - id: 8:16$EHcAYec/Wa76OFbhSSWRMDf0jFdw+t+uKiWGv86tD5Q=
          name: sdb
          size: 32212254720
          make: QEMU QEMU_HARDDISK
          select: "yes"

* 加入 driver

$ kubectl directpv init --dangerous drives.yaml

 ███████████████████████████████████████████████████████████████████████████ 100%

 Processed initialization request '72cc955c-2b29-4618-9974-d70385305aa9' for node 'cilium-w1' ✔
 Processed initialization request 'fdc82163-3109-4e1e-8bdf-5a753f4db4b7' for node 'cilium-w2' ✔
 Processed initialization request '72e8fd44-408b-4cbc-a473-e745d5525828' for node 'cilium-w3' ✔

┌──────────────────────────────────────┬───────────┬───────┬─────────┐
│ REQUEST_ID                           │ NODE      │ DRIVE │ MESSAGE │
├──────────────────────────────────────┼───────────┼───────┼─────────┤
│ 72cc955c-2b29-4618-9974-d70385305aa9 │ cilium-w1 │ sdd   │ Success │
│ 72cc955c-2b29-4618-9974-d70385305aa9 │ cilium-w1 │ sde   │ Success │
│ fdc82163-3109-4e1e-8bdf-5a753f4db4b7 │ cilium-w2 │ sdc   │ Success │
│ fdc82163-3109-4e1e-8bdf-5a753f4db4b7 │ cilium-w2 │ sdd   │ Success │
│ 72e8fd44-408b-4cbc-a473-e745d5525828 │ cilium-w3 │ sdb   │ Success │
│ 72e8fd44-408b-4cbc-a473-e745d5525828 │ cilium-w3 │ sdc   │ Success │
└──────────────────────────────────────┴───────────┴───────┴─────────┘

* 列出 directpv 使用的 driver

$ kubectl directpv list drives
┌───────────┬──────┬────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE      │ NAME │ MAKE               │ SIZE   │ FREE   │ VOLUMES │ STATUS │
├───────────┼──────┼────────────────────┼────────┼────────┼─────────┼────────┤
│ cilium-w1 │ sdd  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ cilium-w1 │ sde  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ cilium-w2 │ sdc  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ cilium-w2 │ sdd  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ cilium-w3 │ sdb  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ cilium-w3 │ sdc  │ QEMU QEMU_HARDDISK │ 30 GiB │ 30 GiB │ -       │ Ready  │
└───────────┴──────┴────────────────────┴────────┴────────┴─────────┴────────┘

* 移除 Driver

$ kubectl directpv remove --drives=sdd --nodes=cilium-w1
Removing cilium-w1/sdd

$ kubectl directpv remove --drives=sde --nodes=cilium-w1
Removing cilium-w1/sde

$ kubectl directpv remove --drives=sdc --nodes=cilium-w2
Removing cilium-w2/sdc

$ kubectl directpv remove --drives=sdd --nodes=cilium-w2
Removing cilium-w2/sdd

$ kubectl directpv remove --drives=sdb --nodes=cilium-w3
Removing cilium-w3/sdb

$ kubectl directpv remove --drives=sdc --nodes=cilium-w3
Removing cilium-w3/sdc

## 部屬 minio
* 使用 Statefulset 部屬 minio，會部署在 `kube-system` namespace
* 因為有 cpu 版本的問題所以選 cpuv1 這個版本的 image
* 需先安裝好 metallb，`loadBalancerIP` 需換成自己環境的網段
* minio-job 負責在 minio 建立 fbs、mysql 的 Buckets

apiVersion: v1
kind: Service
metadata:
  name: svc-stsminio
  namespace: kube-system
spec:
  type: LoadBalancer
  allocateLoadBalancerNodePorts: false
  loadBalancerIP: 192.168.11.145                 # change
  ports:
    - port: 9000
      targetPort: 9000
      protocol: TCP
      name: mc
    - port: 9001
      targetPort: 9001
      protocol: TCP
      name: mcweb
  selector:
    app: stsminio
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: stsminio
  namespace: kube-system
  labels:
    app: stsminio
spec:
  serviceName: "svc-stsminio"
  replicas: 2
  podManagementPolicy: "Parallel"
  selector:
    matchLabels:
      app: stsminio
  template:
    metadata:
      labels:
        app: stsminio
    spec:
      containers:
      - name: stsminio
        image: quay.io/minio/minio:RELEASE.2023-11-01T01-57-10Z-cpuv1
        volumeMounts:
        - name: minio-data0
          mountPath: /data0
        - name: minio-data1
          mountPath: /data1
        args:
        # Change this according to No. of replicas, namespace and volumes
        - server
        - --console-address
        - :9001
        - http://stsminio-{0...1}.svc-stsminio.kube-system.svc.cluster.local/data{0...1}
        env:
        # MinIO access key and secret key
        - name: MINIO_ACCESS_KEY
          value: "minio"
        - name: MINIO_SECRET_KEY
          value: "minio123"
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /minio/health/live
            port: http
            scheme: HTTP
          initialDelaySeconds: 120
          periodSeconds: 15
          successThreshold: 1
          timeoutSeconds: 10
        ports:
        - containerPort: 9000
          name: mc
          protocol: TCP
        - containerPort: 9001
          name: mcweb
          protocol: TCP
  volumeClaimTemplates: # This is the specification in which you reference the StorageClass
  - metadata:
      name: minio-data0
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: directpv-min-io
  - metadata:
      name: minio-data1
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 20Gi
      storageClassName: directpv-min-io
---
apiVersion: batch/v1
kind: Job
metadata:
  name: minio-job
  namespace: kube-system
spec:
  # If the field is set to 0, the Job will be eligible to be automatically deleted immediately
  # after it finishes.
  ttlSecondsAfterFinished: 0
  template:
    metadata:
      name: minio-job
    spec:
      containers:
      - name: minio-job
        image: docker.io/taiwanese/alp.mc.rclone.s3fs
        tty: true
        command: ["sh", "-c"]
        args:
        - |
          sleep 60
          mc config host add tk8s http://svc-stsminio.kube-system:9000 minio minio123
          mc admin config set tk8s storage_class standard=EC:1
          mc mb tk8s/fbs
          sleep 30
      restartPolicy: Never

* 登入 minio web
* 已經建立好 fbs、mysql Buckets

* 安裝 mc CLI

$ sudo curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /usr/local/bin/mc

$ sudo chmod +x /usr/local/bin/mc

* 檢視 minio url

$ mc config host ls
gcs
  URL       : https://storage.googleapis.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v2
  Path      : dns

local
  URL       : http://localhost:9000
  AccessKey :
  SecretKey :
  API       :
  Path      : auto

play
  URL       : https://play.min.io
  AccessKey : Q3AM3UQ867SPQQA43P2F
  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
  API       : S3v4
  Path      : auto

s3
  URL       : https://s3.amazonaws.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v4
  Path      : dns

tk8s
  URL       : http://192.168.11.145:9000
  AccessKey : minio
  SecretKey : minio123
  API       : s3v4
  Path      : auto

* 登入 minio 並測試上傳資料到 minio

$ mc alias set tk8s http://192.168.11.145:9000 minio minio123
Added `tk8s` successfully.

$ mc ls tk8s
[2024-03-27 15:17:06 CST]     0B fbs/

$ echo 123 > test.txt

$ mc cp test.txt tk8s/fbs/
/home/rancher/test.txt:               4 B / 4 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 37 B/s 0s

$ mc ls tk8s/fbs/
[2024-03-25 10:12:43 CST]     4B STANDARD test.txt

# 刪除在 fbs 下的 test.txt 檔案 
$ mc rm tk8s/fbs/test.txt
Removed `tk8s/fbs/test.txt`.

## 部屬 s3fs
* `mountPropagation: Bidirectional` 對 container 內的磁碟區所做的變更會反映在主機中。

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: minio-s3fs
  name: minio-s3fs
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: minio-s3fs
  template:
    metadata:
      labels:
        app: minio-s3fs
    spec:
      containers:
      - name: s3fs
        image: docker.io/taiwanese/alp.s3fs.mc
        tty: true
        lifecycle:
          postStart:
            exec:
              command:
                - /bin/bash
                - -c
                - |
                  sleep 120
                  echo 'minio:minio123' > /tmp/.passwd-s3fs
                  chmod 600 /tmp/.passwd-s3fs
                  echo 'user_allow_other' >> /etc/fuse.conf
                  s3fs fbs /opt/fbs -o allow_other -o passwd_file=/tmp/.passwd-s3fs -o url=http://svc-stsminio.kube-system:9000 -o use_path_request_style -o dbglevel=info -o curldbg &>/dev/null &
          preStop:
            exec:
              command: 
                - /bin/bash"
                - -c
                - |
                  umount -f /opt/fbs
        securityContext:
          privileged: true
        volumeMounts:
        - name: fbs    
          mountPath: /opt/fbs
          mountPropagation: Bidirectional
      volumes:
      - name: fbs
        hostPath:
          path: /opt/fbs

## 測試 File Browser
* 建立 File Browser
* 使用 hostPath volume

kind: Deployment
apiVersion: apps/v1
metadata:
  name: fbs
  labels: 
    app: fbs
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fbs
  template:
    metadata:
      labels:
        app: fbs
    spec:
      containers:
      - name: fbs
        image: taiwanese/alp.fbs
        ports:
        - containerPort: 4000
        volumeMounts:
        - mountPath: /srv
          name: fbs-volume
      volumes:
      - name: fbs-volume
        hostPath:
          # directory location on host
          path: /opt/fbs
---
kind: Service
apiVersion: v1
metadata:
  name: svc-fbs
spec:
  type: NodePort
  selector:
    app: fbs
  ports:
  - port: 8080
    targetPort: 4000

* 使用 nodeport 放式登入
> 帳號:admin
> 密碼:admin

* 在 File Browser 丟入一個檔案

* 測試 pod 移轉 node，資料是否永存

$ kubectl get po -owide
NAME                               READY   STATUS    RESTARTS      AGE     IP            NODE        NOMINATED NODE   READINESS GATES
fbs-6fb5b97d98-q72rg               1/1     Running   0             11m     10.42.1.15    cilium-w1   <none>           <none>

$ kubectl scale deploy fbs --replicas=0

$ kubectl cordon cilium-w1

$ kubectl scale deploy fbs --replicas=1

$ kubectl uncordon cilium-w1

$ kubectl get po -owide
NAME                               READY   STATUS    RESTARTS      AGE     IP            NODE        NOMINATED NODE   READINESS GATES
fbs-6fb5b97d98-n252x               1/1     Running   0             31s     10.42.2.250   cilium-w2   <none>           <none>

* 再次進入 File Browser 檢查，透過 minio 可以讓掛載 hostpath 做到共享不同主機的目錄

## 參考連結
https://github.com/minio/directpv