---
# System prepended metadata

title: Test Ceph RBD with Talos Kubernetes

---

# Test Ceph RBD with Talos Kubernetes

<style>
.indent-title-1{
  margin-left: 1em;
}
.indent-title-2{
  margin-left: 2em;
}
.indent-title-3{
  margin-left: 3em;
}
</style>

## Preface

<div class="indent-title-1">

本篇文章會主要會先在 Talos Kubernetes 上建立 Ceph-csi 和 Ceph StorageClass，再實作本次測試目標。

可以透過點擊以下目錄，選擇想看的內容，跳轉至特定章節

:::warning

:::spoiler {state="open"} 目錄

[TOC]

:::

</div>

## 目標

<div class="indent-title-1">

測試 Deployment Object 可不可以共享同一個 Ceph RBD

</div>

## 測試環境

<div class="indent-title-1">

```bash
$ kubectl get nodes -o wide
NAME   STATUS   ROLES           AGE    VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
m1     Ready    control-plane   2d6h   v1.28.3   192.168.61.31   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23
w1     Ready    <none>          2d6h   v1.28.3   192.168.61.34   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23
w2     Ready    <none>          2d6h   v1.28.3   192.168.61.35   <none>        Talos (v1.5.5)   6.1.61-talos     containerd://1.6.23

$ kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
NAME   TAINTS
m1     [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
w1     <none>
w2     <none>
```

> Talos OS version: v1.5.5
> K8S version: v1.28.3
> Ceph Version: 18.2.0 Reef (stable)
> 使用 `nodeName` 可以無視節點上的 Taints，或是使用 toleration (容忍度)，可以讓 Pod 容忍節點的 Taints

</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>

## Ceph 設定

### 1. 建立 Ceph Pool

<div class="indent-title-1">

在 Ceph Node 執行以下命令 : 

```bash
$ ceph osd pool create kubernetes

# Use the rbd tool to initialize the pool
$ rbd pool init kubernetes

# 設定 pool 最大容量為 10GB
$ ceph osd pool set-quota kubernetes max_bytes $((10 * 1024 * 1024 * 1024))
```

> Ceph pool 是否有其他 init 方式?

</div>

### 2. 建立 Ceph Client 身分驗證

<div class="indent-title-1">

```!
$ ceph auth get-or-create client.kubernetes mon 'profile rbd' osd 'profile rbd pool=kubernetes' mgr 'profile rbd pool=kubernetes'
```

螢幕輸出 : 

```
[client.kubernetes]
        key = AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw==
```

![image](https://hackmd.io/_uploads/SkUIfrEwp.png)


</div>



## 設定與部屬 Ceph-CSI RBD plugins

### 1. Download ceph-csi

<div class="indent-title-1">

在 Talos k8s 外部管理主機執行以下命令 : 

```bash
$ cd ~ && git clone https://github.com/ceph/ceph-csi.git
```

</div>

### 2. 建立並切換至 csi-ceph namespace

<div class="indent-title-1">

```bash
$ kubectl create ns csi-ceph
namespace/csi-ceph created

$ kubectl config set-context --current --namespace=csi-ceph
Context "admin@topgun" modified.
```

</div>

### 3. 配置 ceph-csi 設定檔

<div class="indent-title-1">

#### 3.1. 獲取 Ceph monitor 和 fsid 資訊

<div class="indent-title-2">

在 Proxmox Ceph Node ( monitor ) 執行以下命令 : 

```bash
$ ceph mon dump
```

螢幕輸出 : 

```
epoch 4
fsid cd4925d8-31e9-4245-a392-469d661fedce
last_changed 2023-12-16T22:35:48.168822+0800
created 2023-11-22T09:44:04.138233+0800
min_mon_release 18 (reef)
election_strategy: 1
0: [v2:192.168.61.11:3300/0,v1:192.168.61.11:6789/0] mon.pve
1: [v2:192.168.61.12:3300/0,v1:192.168.61.12:6789/0] mon.pve2
2: [v2:192.168.61.13:3300/0,v1:192.168.61.13:6789/0] mon.pve3
dumped monmap epoch 4
```

</div>

#### 3.2. 設定 Ceph-csi configmap

<div class="indent-title-2">

在 Talos 外部管理主機執行以下命令 : 

```!
$ cd ~/ceph-csi/deploy/rbd/kubernetes

$ cat <<EOF > csi-config-map.yaml
---
apiVersion: v1
kind: ConfigMap
data:
  config.json: |-
    [
      {
        "clusterID": "cd4925d8-31e9-4245-a392-469d661fedce",
        "monitors": [
          "192.168.61.11:6789",
          "192.168.61.12:6789",
          "192.168.61.13:6789"
        ]
      }
    ]
metadata:
  name: ceph-csi-config
EOF
```

> 須設定 `clusterID` 和 `monitors 的 IP Address`

</div>

#### 3.3. 設定 csidriver 的 pod 不要 Mount host 主機的 `/etc/selinux` 到 pods 裡面

<div class="indent-title-2">

```!
$ sed -i 's|seLinuxMount: true|seLinuxMount: false|g' csidriver.yaml
$ sed -i '/- mountPath: \/etc\/selinux/,+2d' csi-rbdplugin.yaml
$ sed -i '/- name: etc-selinux/,+2d' csi-rbdplugin.yaml
```

</div>

#### 3.4. 將所有 Yaml 中定義的物件都更換為 csi-ceph Namespace

<div class="indent-title-2">

```!
$ sed -i 's|namespace: default|namespace: csi-ceph|g' *.yaml
```

</div>

#### 3.5. 設定 csi-rbdplugin-provisioner 和 csi-rbdplugin 的 Pod 能夠在 Contorlplane Node 上執行

<div class="indent-title-2">

```!
$ sed -i '36i\      tolerations:\n        - operator: Exists' csi-rbdplugin-provisioner.yaml
$ sed -i '24i\      tolerations:\n        - operator: Exists' csi-rbdplugin.yaml
```

</div>

#### 3.6. 產生 CEPH-CSI cephx secret

<div class="indent-title-2">

```
$ cat <<EOF > ~/ceph-csi/examples/rbd/secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
  name: csi-rbd-secret
  namespace: csi-ceph
stringData:
  userID: kubernetes
  userKey: AQBqkoFlJoe0MxAAb3UMOPeQSsDWI3FTRj8UTw==

  # Encryption passphrase
  encryptionPassphrase: test_passphrase
EOF
```

> 須設定 namespace、userID 和 userKey 的值

</div>

#### 3.7. 建立 CEPH-CSI cephx secret

<div class="indent-title-2">

```!
$ kubectl apply -f ~/ceph-csi/examples/rbd/secret.yaml
```

</div>
</div>

### 4. 部屬 ceph-csi 

<div class="indent-title-1">

#### 4.1. 設定 csi-ceph Namespace 中的 Pod 能夠擁有 privileged 權限

<div class="indent-title-2">

```!
$ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce=privileged
```
    
> `csi-rbdplugin` 和 `csi-rbdplugin-provisioner` 的 Pod 會需要 privileged 的權限。

> Talos K8S PSA 預設所有的 Pod 不能有 privileged 的權限，此時可以透過幫 Namespace 貼上面這個 label ，就能夠覆蓋掉 K8S PSA 的設定。 
    
</div>

#### 4.2. 最新版本的 ceph-csi 還需要另一個 ConfigMap 物件來定義 Ceph 的設定資訊，以便新增至 CSI Container 內的 `ceph.conf` 檔案中

<div class="indent-title-2">

```!
$ kubectl apply -f ~/ceph-csi/deploy/ceph-conf.yaml
```

> This is a sample configmap that helps define a Ceph configuration as required by the CSI plugins.

</div>

#### 4.3. 開始部屬 ceph-csi 

<div class="indent-title-2">

```!
$ cd ~/ceph-csi/examples/rbd

$ ./plugin-deploy.sh ~/ceph-csi/deploy/rbd/kubernetes

## 移除 vault
$ kubectl delete -f ../kms/vault/vault.yaml
```

</div>

#### 4.4. 檢視 ceph-csi 部屬狀態

<div class="indent-title-2">

```
$ kubectl get all
```

螢幕輸出 : 

```
NAME                                            READY   STATUS    RESTARTS   AGE
pod/csi-rbdplugin-2dls2                         3/3     Running   0          111m
pod/csi-rbdplugin-j7fx6                         3/3     Running   0          111m
pod/csi-rbdplugin-l84w5                         3/3     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-brz59   7/7     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-jdwcr   7/7     Running   0          111m
pod/csi-rbdplugin-provisioner-ddb58c7f4-td422   7/7     Running   0          111m

NAME                                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/csi-metrics-rbdplugin       ClusterIP   10.98.76.185    <none>        8080/TCP   111m
service/csi-rbdplugin-provisioner   ClusterIP   10.104.214.86   <none>        8080/TCP   111m

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/csi-rbdplugin   3         3         3       3            3           <none>          111m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/csi-rbdplugin-provisioner   3/3     3            3           111m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/csi-rbdplugin-provisioner-ddb58c7f4   3         3         3       111m
```

</div>



#### 4.5. 設定 StorageClass Yaml 檔

<div class="indent-title-2">

```
$ cat <<EOF > csi-rbd-sc.yaml
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: csi-rbd-sc
provisioner: rbd.csi.ceph.com
parameters:
   clusterID: cd4925d8-31e9-4245-a392-469d661fedce
   pool: kubernetes
   imageFeatures: layering
   csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
   csi.storage.k8s.io/provisioner-secret-namespace: csi-ceph
   csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
   csi.storage.k8s.io/controller-expand-secret-namespace: csi-ceph
   csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
   csi.storage.k8s.io/node-stage-secret-namespace: csi-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true
mountOptions:
   - discard
EOF
```

> 須設定 `clusterID` 和 `pool` 的值 

</div>

#### 4.6. 建立 StorageClass

<div class="indent-title-2">

```
$ kubectl apply -f csi-rbd-sc.yaml
```

</div>

#### 4.7. 檢視 StorageClass

<div class="indent-title-2">

```
$ kubectl get sc csi-rbd-sc
```

螢幕輸出 : 

```
NAME         PROVISIONER        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
csi-rbd-sc   rbd.csi.ceph.com   Delete          Immediate           true                   9s
```


> 以上環境部屬完畢，以下開始測試本次實作目標。

</div>
</div>
</div>
</div>
</div>

---

## 開始實作測試目標

###  1. 設定 PVC

<div class="indent-title-1">

```!
$ cat <<EOF > raw-block-pvc-rwx.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw-block-pvc
spec:
  accessModes:
    - ReadWriteMany
  volumeMode: Block
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-rbd-sc
EOF
```

> Note: Using ceph-csi, specifying Filesystem for volumeMode can support both ReadWriteOnce and ReadOnlyMany accessMode claims, and specifying Block for volumeMode can support ReadWriteOnce, ReadWriteMany, and ReadOnlyMany accessMode claims.

</div>

###  2. 建立 PVC Yaml 檔

<div class="indent-title-1">

```!
$ kubectl apply -f raw-block-pvc-rwx.yaml
```

</div>

###  3. 檢查 PVC 建立狀態

<div class="indent-title-1">

```!
$ kubectl get pvc
```

螢幕輸出 : 

```!
NAME            STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
raw-block-pvc   Bound    pvc-629444ef-f0c0-4afd-8263-443d6f4beb25   1Gi        RWX            csi-rbd-sc     3s
```

</div>

###  4. 設定 Deployment Object Yaml 檔

<div class="indent-title-1">

```
$ cat <<EOF > raw-block-deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pod-with-raw-block-volume
  labels:
    os: alpine
spec:
  replicas: 3
  selector:
    matchLabels:
      os: alpine
  template:
    metadata:
      labels:
        os: alpine
    spec:
      containers:
      - name: alpine
        image: quay.io/cloudwalker/alp.base:latest
        command: ["/bin/sleep", "infinity"]
        volumeDevices:
          - name: data
            devicePath: /dev/xvda
        securityContext:
          capabilities:
            add: ["SYS_ADMIN"]
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: raw-block-pvc
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: os
                operator: In
                values:
                - alpine
            topologyKey: kubernetes.io/hostname
      tolerations:
      - operator: Exists
EOF
```

</div>

###  5. 建立 Deployment Object Yaml 檔

<div class="indent-title-1">

```!
$ kubectl apply -f raw-block-deployment.yaml
```

</div>

###  6. 檢查 Pods 運作狀態

<div class="indent-title-1">

```!
$ kubectl get pods -l os=alpine -o wide
```

螢幕輸出 : 

```
NAME                                       READY   STATUS    RESTARTS   AGE   IP            NODE   NOMINATED NODE   READINESS GATES
pod-with-raw-block-volume-8677b97b-m64wt   1/1     Running   0          26s   10.244.1.19   m1     <none>           <none>
pod-with-raw-block-volume-8677b97b-nnpt2   1/1     Running   0          26s   10.244.0.53   w1     <none>           <none>
pod-with-raw-block-volume-8677b97b-x2qh9   1/1     Running   0          26s   10.244.2.46   w2     <none>           <none>
```

</div>
</div>
</div>
</div>
</div>
</div>
</div>

###  7. 進入不同的 Pods 檢視已掛載的 Ceph Block Device

<div class="indent-title-1">

```
$ kubectl exec pod-with-raw-block-volume-8677b97b-2dfj9 -- ls -l /dev/xvda

$ kubectl exec pod-with-raw-block-volume-8677b97b-ft4kw -- ls -l /dev/xvda

$ kubectl exec pod-with-raw-block-volume-8677b97b-sbp6m -- ls -l /dev/xvda
```

3 次執行的結果應該跟以下螢幕輸出類似 : 

```
brw------- 1 root root 252, 0 Dec 19 09:25 /dev/xvda
```

</div>

###  8. 測試使用 XFS 檔案系統 Fotmat Ceph Block Device，並將 /ceph 目錄掛載上去，最後建立 test 空檔案

<div class="indent-title-1">

```
$ kubectl exec -it pod-with-raw-block-volume-8677b97b-m64wt -- bash
pod-with-raw-block-volume-8677b97b-m64wt:/# apk add xfsprogs
pod-with-raw-block-volume-8677b97b-m64wt:/# mkfs.xfs /dev/xvda
pod-with-raw-block-volume-8677b97b-m64wt:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# lsblk /dev/xvda
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
rbd0 252:0    0   1G  0 disk /ceph
pod-with-raw-block-volume-8677b97b-m64wt:/# touch /ceph/test
pod-with-raw-block-volume-8677b97b-m64wt:/# exit
exit
```

</div>

### 9. 進到另外兩個 Pods 檢查是否有看到 test 這個空檔案

<div class="indent-title-1">

```
$ kubectl exec -it pod-with-raw-block-volume-8677b97b-nnpt2 -- bash
pod-with-raw-block-volume-8677b97b-nnpt2:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-nnpt2:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-nnpt2:/# ls -l /ceph/
total 0
-rw-r--r-- 1 root root 0 Dec 19 13:54 test
pod-with-raw-block-volume-8677b97b-nnpt2:/# exit
exit

$ kubectl exec -it pod-with-raw-block-volume-8677b97b-x2qh9 -- bash
pod-with-raw-block-volume-8677b97b-x2qh9:/# mkdir /ceph
pod-with-raw-block-volume-8677b97b-x2qh9:/# mount /dev/xvda /ceph
pod-with-raw-block-volume-8677b97b-x2qh9:/# ls -l /ceph/
total 0
-rw-r--r-- 1 root root 0 Dec 19 13:54 test
pod-with-raw-block-volume-8677b97b-x2qh9:/# exit
exit
```

</div>


### 10. 列出在 kubernetes pool 底下的 RBD images

<div class="indent-title-1">

在 Ceph Node 執行以下命令

```
$ rbd ls kubernetes
```

螢幕輸出 : 

```
csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e
```

</div>

### 11. 查看 RBD Image 的詳細資訊

<div class="indent-title-1">

```
$ rbd info kubernetes/csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e
```

螢幕輸出 : 

```
rbd image 'csi-vol-89e67965-9fb9-47cb-b25b-210268f6d03e':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 13c94844fea3c6
        block_name_prefix: rbd_data.13c94844fea3c6
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Tue Dec 19 21:50:23 2023
        access_timestamp: Tue Dec 19 21:50:23 2023
        modify_timestamp: Tue Dec 19 21:50:23 2023
```

</div>
</div>
</div>
</div>
</div>
</div>
</div>

## 清除測試環境

<div class="indent-title-1">

```!
## 在 Talos 外部管理主機執行以下命令
$ kubectl delete -f raw-block-deployment.yaml,raw-block-pvc-rwx.yaml
$ kubectl delete -f csi-rbd-sc.yaml,secret.yaml
$ kubectl delete -f ~/ceph-csi/deploy/ceph-conf.yaml
$ ./plugin-teardown.sh ~/ceph-csi/deploy/rbd/kubernetes/
$ kubectl label ns csi-ceph pod-security.kubernetes.io/enforce-
$ kubectl get all,configmap,secret
NAME                         DATA   AGE
configmap/kube-root-ca.crt   1      20h
$ kubectl config set-context --current --namespace=default
$ kubectl delete ns csi-ceph
$ cd ~ && rm -r ceph-csi/

## 在 Ceph Node 執行以下命令
$ rbd -p kubernetes ls
$ ceph auth rm client.kubernetes
$ ceph osd pool rm kubernetes kubernetes --yes-i-really-really-mean-it
```

</div>


## 參考文件

- [BLOCK DEVICES AND KUBERNETES - Ceph Docs](https://docs.ceph.com/en/quincy/rbd/rbd-kubernetes/)
- [How to test RBD and CephFS plugins with Kubernetes 1.14+ - Ceph/ceph-csi Github](https://github.com/ceph/ceph-csi/tree/devel/examples)
- [設定 StorageClass (以 Ceph RBD 為例) - 小信豬的原始部落](https://godleon.github.io/blog/Kubernetes/k8s-Config-StorageClass-with-Ceph-RBD/)
- [Pod Security - Talos Docs](https://www.talos.dev/v1.6/kubernetes-guides/configuration/pod-security/)
