# Test Ceph RBD with Talos Kubernetes <style> .indent-title-1{ margin-left: 1em; } .indent-title-2{ margin-left: 2em; } .indent-title-3{ margin-left: 3em; } </style> ## Preface <div class="indent-title-1"> 本篇文章會主要會先在 Talos Kubernetes 上建立 Ceph-csi 和 Ceph StorageClass，再實作本次測試目標。 可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節 :::warning :::spoiler {state="open"} 目錄 [TOC] ::: </div> ## 目標 <div class="indent-title-1"> 測試 Deployment Object 可不可以共享同一個 Ceph RBD </div> ## 測試環境 <div class="indent-title-1"> ```bash $ kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME m1 Ready control-plane 2d6h v1.28.3 192.168.61.31 <none> Talos (v1.5.5) 6.1.61-talos containerd://1.6.23 w1 Ready <none> 2d6h v1.28.3 192.168.61.34 <none> Talos (v1.5.5) 6.1.61-talos containerd://1.6.23 w2 Ready <none> 2d6h v1.28.3 192.168.61.35 <none> Talos (v1.5.5) 6.1.61-talos containerd://1.6.23 $ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints NAME TAINTS m1 [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]] w1 <none> w2 <none> ``` > Talos OS version: v1.5.5 > K8S version: v1.28.3 > Ceph Version: 18.2.0 Reef (stable) > 使用 `nodeName` 可以無視節點上的 Taints，或是使用 toleration (容忍度)，可以讓 Pod 容忍節點的 Taints </div> </div> </div> </div> </div> </div> </div> </div> ## Ceph 設定 ### 1. 建立 Ceph Pool <div class="indent-title-1"> 在 Ceph Node 執行以下命令 : ```bash $ ceph osd pool create kubernetes # Use the rbd tool to initialize the pool $ rbd pool init kubernetes # 設定 pool 最大容量為 10GB $ ceph osd pool set-quota kubernetes max_bytes $((10 * 1024 * 1024 * 1024)) ``` > Ceph pool 是否有其他 init 方式? </div> ### 2. 建立 Ceph Client 身分驗證 <div class="indent-title-1"> ```! $ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes' ``` 螢幕輸出 : ``` [client.kubernetes] key = AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw== ``` ![image](https://hackmd.io/_uploads/SkUIfrEwp.png) </div> ## 設定與部屬 Ceph-CSI RBD plugins ### 1. Download ceph-csi <div class="indent-title-1"> 在 Talos k8s 外部管理主機執行以下命令 : ```bash $ cd ~ && git clone https://github.com/ceph/ceph-csi.git ``` </div> ### 2. 建立並切換至 csi-ceph namespace <div class="indent-title-1"> ```bash $ kubectl create ns csi-ceph namespace/csi-ceph created $ kubectl config set-context --current --namespace=csi-ceph Context "admin@topgun" modified. ``` </div> ### 3. 配置 ceph-csi 設定檔 <div class="indent-title-1"> #### 3.1. 獲取 Ceph monitor 和 fsid 資訊 <div class="indent-title-2"> 在 Proxmox Ceph Node ( monitor ) 執行以下命令 : ```bash $ ceph mon dump ``` 螢幕輸出 : ``` epoch 4 fsid cd4925d8-31e9-4245-a392-469d661fedce last_changed 2023-12-16T22:35:48.168822+0800 created 2023-11-22T09:44:04.138233+0800 min_mon_release 18 (reef) election_strategy: 1 0: [v2:192.168.61.11:3300/0,v1:192.168.61.11:6789/0] mon.pve 1: [v2:192.168.61.12:3300/0,v1:192.168.61.12:6789/0] mon.pve2 2: [v2:192.168.61.13:3300/0,v1:192.168.61.13:6789/0] mon.pve3 dumped monmap epoch 4 ``` </div> #### 3.2. 設定 Ceph-csi configmap <div class="indent-title-2"> 在 Talos 外部管理主機執行以下命令 : ```! $ cd ~/ceph-csi/deploy/rbd/kubernetes $ cat <<EOF > csi-config-map.yaml --- apiVersion: v1 kind: ConfigMap data: config.json: |- [ { "clusterID": "cd4925d8-31e9-4245-a392-469d661fedce", "monitors": [ "192.168.61.11:6789", "192.168.61.12:6789", "192.168.61.13:6789" ] } ] metadata: name: ceph-csi-config EOF ``` > 須設定 `clusterID` 和 `monitors 的 IP Address` </div> #### 3.3. 設定 csidriver 的 pod 不要 Mount host 主機的 `/etc/selinux` 到 pods 裡面 <div class="indent-title-2"> ```! $ sed -i 's|seLinuxMount: true|seLinuxMount: false|g' csidriver.yaml $ sed -i '/- mountPath: \/etc\/selinux/,+2d' csi-rbdplugin.yaml $ sed -i '/- name: etc-selinux/,+2d' csi-rbdplugin.yaml ``` </div> #### 3.4. 將所有 Yaml 中定義的物件都更換為 csi-ceph Namespace <div class="indent-title-2"> ```! $ sed -i 's|namespace: default|namespace: csi-ceph|g' *.yaml ``` </div> #### 3.5. 設定 csi-rbdplugin-provisioner 和 csi-rbdplugin 的 Pod 能夠在 Contorlplane Node 上執行 <div class="indent-title-2"> ```! $ sed -i '36i\ tolerations:\n - operator: Exists' csi-rbdplugin-provisioner.yaml $ sed -i '24i\ tolerations:\n - operator: Exists' csi-rbdplugin.yaml ``` </div> #### 3.6. 產生 CEPH-CSI cephx secret <div class="indent-title-2"> ``` $ cat <<EOF > ~/ceph-csi/examples/rbd/secret.yaml --- apiVersion: v1 kind: Secret metadata: name: csi-rbd-secret namespace: csi-ceph stringData: userID: kubernetes userKey: AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw== # Encryption passphrase encryptionPassphrase: test_passphrase EOF ``` > 須設定 namespace、userID 和 userKey 的值 </div> #### 3.7. 建立 CEPH-CSI cephx secret <div class="indent-title-2"> ```! $ kubectl apply -f ~/ceph-csi/examples/rbd/secret.yaml ``` </div> </div> ### 4. 部屬 ceph-csi <div class="indent-title-1"> #### 4.1. 設定 csi-ceph Namespace 中的 Pod 能夠擁有 privileged 權限 <div class="indent-title-2"> ```! $ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce=privileged ``` > `csi-rbdplugin` 和 `csi-rbdplugin-provisioner` 的 Pod 會需要 privileged 的權限。 > Talos K8S PSA 預設所有的 Pod 不能有 privileged 的權限，此時可以透過幫 Namespace 貼上面這個 label ，就能夠覆蓋掉 K8S PSA 的設定。 </div> #### 4.2. 最新版本的 ceph-csi 還需要另一個 ConfigMap 物件來定義 Ceph 的設定資訊，以便新增至 CSI Container 內的 `ceph.conf` 檔案中 <div class="indent-title-2"> ```! $ kubectl apply -f ~/ceph-csi/deploy/ceph-conf.yaml ``` > This is a sample configmap that helps define a Ceph configuration as required by the CSI plugins. </div> #### 4.3. 開始部屬 ceph-csi <div class="indent-title-2"> ```! $ cd ~/ceph-csi/examples/rbd $ ./plugin-deploy.sh ~/ceph-csi/deploy/rbd/kubernetes ## 移除 vault $ kubectl delete -f ../kms/vault/vault.yaml ``` </div> #### 4.4. 檢視 ceph-csi 部屬狀態 <div class="indent-title-2"> ``` $ kubectl get all ``` 螢幕輸出 : ``` NAME READY STATUS RESTARTS AGE pod/csi-rbdplugin-2dls2 3/3 Running 0 111m pod/csi-rbdplugin-j7fx6 3/3 Running 0 111m pod/csi-rbdplugin-l84w5 3/3 Running 0 111m pod/csi-rbdplugin-provisioner-ddb58c7f4-brz59 7/7 Running 0 111m pod/csi-rbdplugin-provisioner-ddb58c7f4-jdwcr 7/7 Running 0 111m pod/csi-rbdplugin-provisioner-ddb58c7f4-td422 7/7 Running 0 111m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/csi-metrics-rbdplugin ClusterIP 10.98.76.185 <none> 8080/TCP 111m service/csi-rbdplugin-provisioner ClusterIP 10.104.214.86 <none> 8080/TCP 111m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/csi-rbdplugin 3 3 3 3 3 <none> 111m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/csi-rbdplugin-provisioner 3/3 3 3 111m NAME DESIRED CURRENT READY AGE replicaset.apps/csi-rbdplugin-provisioner-ddb58c7f4 3 3 3 111m ``` </div> #### 4.5. 設定 StorageClass Yaml 檔 <div class="indent-title-2"> ``` $ cat <<EOF > csi-rbd-sc.yaml --- apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: csi-rbd-sc provisioner: rbd.csi.ceph.com parameters: clusterID: cd4925d8-31e9-4245-a392-469d661fedce pool: kubernetes imageFeatures: layering csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret csi.storage.k8s.io/provisioner-secret-namespace: csi-ceph csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret csi.storage.k8s.io/controller-expand-secret-namespace: csi-ceph csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret csi.storage.k8s.io/node-stage-secret-namespace: csi-ceph reclaimPolicy: Delete allowVolumeExpansion: true mountOptions: - discard EOF ``` > 須設定 `clusterID` 和 `pool` 的值 </div> #### 4.6. 建立 StorageClass <div class="indent-title-2"> ``` $ kubectl apply -f csi-rbd-sc.yaml ``` </div> #### 4.7. 檢視 StorageClass <div class="indent-title-2"> ``` $ kubectl get sc csi-rbd-sc ``` 螢幕輸出 : ``` NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE csi-rbd-sc rbd.csi.ceph.com Delete Immediate true 9s ``` > 以上環境部屬完畢，以下開始測試本次實作目標。 </div> </div> </div> </div> </div> --- ## 開始實作測試目標 ### 1. 設定 PVC <div class="indent-title-1"> ```! $ cat <<EOF > raw-block-pvc-rwx.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: raw-block-pvc spec: accessModes: - ReadWriteMany volumeMode: Block resources: requests: storage: 1Gi storageClassName: csi-rbd-sc EOF ``` > Note: Using ceph-csi, specifying Filesystem for volumeMode can support both ReadWriteOnce and ReadOnlyMany accessMode claims, and specifying Block for volumeMode can support ReadWriteOnce, ReadWriteMany, and ReadOnlyMany accessMode claims. </div> ### 2. 建立 PVC Yaml 檔 <div class="indent-title-1"> ```! $ kubectl apply -f raw-block-pvc-rwx.yaml ``` </div> ### 3. 檢查 PVC 建立狀態 <div class="indent-title-1"> ```! $ kubectl get pvc ``` 螢幕輸出 : ```! NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE raw-block-pvc Bound pvc-629444ef-f0c0-4afd-8263-443d6f4beb25 1Gi RWX csi-rbd-sc 3s ``` </div> ### 4. 設定 Deployment Object Yaml 檔 <div class="indent-title-1"> ``` $ cat <<EOF > raw-block-deployment.yaml --- apiVersion: apps/v1 kind: Deployment metadata: name: pod-with-raw-block-volume labels: os: alpine spec: replicas: 3 selector: matchLabels: os: alpine template: metadata: labels: os: alpine spec: containers: - name: alpine image: quay.io/cloudwalker/alp.base:latest command: ["/bin/sleep", "infinity"] volumeDevices: - name: data devicePath: /dev/xvda securityContext: capabilities: add: ["SYS_ADMIN"] volumes: - name: data persistentVolumeClaim: claimName: raw-block-pvc affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: os operator: In values: - alpine topologyKey: kubernetes.io/hostname tolerations: - operator: Exists EOF ``` </div> ### 5. 建立 Deployment Object Yaml 檔 <div class="indent-title-1"> ```! $ kubectl apply -f raw-block-deployment.yaml ``` </div> ### 6. 檢查 Pods 運作狀態 <div class="indent-title-1"> ```! $ kubectl get pods -l os=alpine -o wide ``` 螢幕輸出 : ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod-with-raw-block-volume-8677b97b-m64wt 1/1 Running 0 26s 10.244.1.19 m1 <none> <none> pod-with-raw-block-volume-8677b97b-nnpt2 1/1 Running 0 26s 10.244.0.53 w1 <none> <none> pod-with-raw-block-volume-8677b97b-x2qh9 1/1 Running 0 26s 10.244.2.46 w2 <none> <none> ``` </div> </div> </div> </div> </div> </div> </div> ### 7. 進入不同的 Pods 檢視已掛載的 Ceph Block Device <div class="indent-title-1"> ``` $ kubectl exec pod-with-raw-block-volume-8677b97b-2dfj9 -- ls -l /dev/xvda $ kubectl exec pod-with-raw-block-volume-8677b97b-ft4kw -- ls -l /dev/xvda $ kubectl exec pod-with-raw-block-volume-8677b97b-sbp6m -- ls -l /dev/xvda ``` 3 次執行的結果應該跟以下螢幕輸出類似 : ``` brw------- 1 root root 252, 0 Dec 19 09:25 /dev/xvda ``` </div> ### 8. 測試使用 XFS 檔案系統 Fotmat Ceph Block Device，並將 /ceph 目錄掛載上去，最後建立 test 空檔案 <div class="indent-title-1"> ``` $ kubectl exec -it pod-with-raw-block-volume-8677b97b-m64wt -- bash pod-with-raw-block-volume-8677b97b-m64wt:/# apk add xfsprogs pod-with-raw-block-volume-8677b97b-m64wt:/# mkfs.xfs /dev/xvda pod-with-raw-block-volume-8677b97b-m64wt:/# mkdir /ceph pod-with-raw-block-volume-8677b97b-m64wt:/# mount /dev/xvda /ceph pod-with-raw-block-volume-8677b97b-m64wt:/# lsblk /dev/xvda NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS rbd0 252:0 0 1G 0 disk /ceph pod-with-raw-block-volume-8677b97b-m64wt:/# touch /ceph/test pod-with-raw-block-volume-8677b97b-m64wt:/# exit exit ``` </div> ### 9. 進到另外兩個 Pods 檢查是否有看到 test 這個空檔案 <div class="indent-title-1"> ``` $ kubectl exec -it pod-with-raw-block-volume-8677b97b-nnpt2 -- bash pod-with-raw-block-volume-8677b97b-nnpt2:/# mkdir /ceph pod-with-raw-block-volume-8677b97b-nnpt2:/# mount /dev/xvda /ceph pod-with-raw-block-volume-8677b97b-nnpt2:/# ls -l /ceph/ total 0 -rw-r--r-- 1 root root 0 Dec 19 13:54 test pod-with-raw-block-volume-8677b97b-nnpt2:/# exit exit $ kubectl exec -it pod-with-raw-block-volume-8677b97b-x2qh9 -- bash pod-with-raw-block-volume-8677b97b-x2qh9:/# mkdir /ceph pod-with-raw-block-volume-8677b97b-x2qh9:/# mount /dev/xvda /ceph pod-with-raw-block-volume-8677b97b-x2qh9:/# ls -l /ceph/ total 0 -rw-r--r-- 1 root root 0 Dec 19 13:54 test pod-with-raw-block-volume-8677b97b-x2qh9:/# exit exit ``` </div> ### 10. 列出在 kubernetes pool 底下的 RBD images <div class="indent-title-1"> 在 Ceph Node 執行以下命令 ``` $ rbd ls kubernetes ``` 螢幕輸出 : ``` csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e ``` </div> ### 11. 查看 RBD Image 的詳細資訊 <div class="indent-title-1"> ``` $ rbd info kubernetes/csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e ``` 螢幕輸出 : ``` rbd image 'csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 13c94844fea3c6 block_name_prefix: rbd_data.13c94844fea3c6 format: 2 features: layering op_features: flags: create_timestamp: Tue Dec 19 21:50:23 2023 access_timestamp: Tue Dec 19 21:50:23 2023 modify_timestamp: Tue Dec 19 21:50:23 2023 ``` </div> </div> </div> </div> </div> </div> </div> ## 清除測試環境 <div class="indent-title-1"> ```! ## 在 Talos 外部管理主機執行以下命令 $ kubectl delete -f raw-block-deployment.yaml,raw-block-pvc-rwx.yaml $ kubectl delete -f csi-rbd-sc.yaml,secret.yaml $ kubectl delete -f ~/ceph-csi/deploy/ceph-conf.yaml $ ./plugin-teardown.sh ~/ceph-csi/deploy/rbd/kubernetes/ $ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce- $ kubectl get all,configmap,secret NAME DATA AGE configmap/kube-root-ca.crt 1 20h $ kubectl config set-context --current --namespace=default $ kubectl delete ns csi-ceph $ cd ~ && rm -r ceph-csi/ ## 在 Ceph Node 執行以下命令 $ rbd -p kubernetes ls $ ceph auth rm client.kubernetes $ ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it ``` </div> ## 參考文件 - [BLOCK DEVICES AND KUBERNETES - Ceph Docs](https://docs.ceph.com/en/quincy/rbd/rbd-kubernetes/) - [How to test RBD and CephFS plugins with Kubernetes 1.14+ - Ceph/ceph-csi Github](https://github.com/ceph/ceph-csi/tree/devel/examples) - [設定 StorageClass (以 Ceph RBD 為例) - 小信豬的原始部落](https://godleon.github.io/blog/Kubernetes/k8s-Config-StorageClass-with-Ceph-RBD/) - [Pod Security - Talos Docs](https://www.talos.dev/v1.6/kubernetes-guides/configuration/pod-security/)