# Migration of application pods from Multus to Host and viceVersa

:::danger
This testing is done Pod network to Host network in minikube cluster and the result is similar to multus cluster as well.
:::

:::danger
Testing is done only for CephFS and RBD not for NFS
:::

## Migration from Host to Pod Network

* Deploy ceph cluster and CSI should be using Host networking

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]kuberc get po
NAME                                            READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-5wpbw                          2/2     Running     0          3m21s
csi-cephfsplugin-provisioner-6c84b57bc9-gc4hf   5/5     Running     0          3m20s
csi-rbdplugin-provisioner-66969c54d-kcp7v       5/5     Running     0          3m21s
csi-rbdplugin-tdsls                             2/2     Running     0          3m21s
rook-ceph-exporter-minikube-866f865684-vw5wj    1/1     Running     0          98s
rook-ceph-mds-myfs-a-69c844c8cd-mtm89           1/1     Running     0          104s
rook-ceph-mds-myfs-b-6c9c556bff-r647x           1/1     Running     0          100s
rook-ceph-mgr-a-5f584dfc9d-s8pc8                1/1     Running     0          2m31s
rook-ceph-mon-a-644465bb44-4pkck                1/1     Running     0          3m
rook-ceph-operator-5f9bc6d66d-6ffsn             1/1     Running     0          3m23s
rook-ceph-osd-0-7966cb76b6-qrjcp                1/1     Running     0          119s
rook-ceph-osd-prepare-minikube-9qzs5            0/1     Completed   0          2m9s
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]kuberc get cephcluster
NAME         DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
my-cluster   /var/lib/rook     1          6m17s   Ready   Cluster created successfully   HEALTH_OK              5df76aeb-d3d9-43b4-806e-0bb9441112b5

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]kuberc get po/csi-rbdplugin-tdsls -oyaml |grep -i hostnetwork
  hostNetwork: true
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f examples $]kuberc get po/csi-cephfsplugin-5wpbw -oyaml |grep -i hostnetwork
  hostNetwork: true

* Check csi configmap and see if netNamespacePath is set or not

This is just to ensure everything is good

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get cm rook-ceph-csi-config -oyaml
apiVersion: v1
data:
  csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"","subvolumeGroup":"","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":["kubernetes.io/hostname","topology.kubernetes.io/region","topology.kubernetes.io/zone","topology.rook.io/chassis","topology.rook.io/rack","topology.rook.io/row","topology.rook.io/pdu","topology.rook.io/pod","topology.rook.io/room","topology.rook.io/datacenter"]},"namespace":"rook-ceph"},{"clusterID":"e1026845ad66577abae1d16671b464c8","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"","subvolumeGroup":"csi","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":null},"namespace":"rook-ceph"}]'
kind: ConfigMap
metadata:
  creationTimestamp: "2024-01-23T13:46:16Z"
  name: rook-ceph-csi-config
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: false
    controller: true
    kind: Deployment
    name: rook-ceph-operator
    uid: 6df65e36-c138-42b8-a5c5-6b372b951d99
  resourceVersion: "1379"
  uid: 895fab41-2237-412b-b866-895793d97dbb

* Create CephFS/RBD PVC and mount it to application pods and do I/O Operation

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
cephfs-pvc   Bound    pvc-b021a7b1-d58d-4b89-8514-6e43671e6299   1Gi        RWO            rook-cephfs       23s
rbd-pvc      Bound    pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7   1Gi        RWO            rook-ceph-block   7s
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          2m2s
csirbd-demo-pod      1/1     Running   0          106s

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl exec -it csicephfs-demo-pod -- sh
# df -h
Filesystem                                                                                                          Size  Used Avail Use% Mounted on
overlay                                                                                                              28G  6.0G   21G  23% /
tmpfs                                                                                                                64M     0   64M   0% /dev
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1                                                                                                            28G  6.0G   21G  23% /etc/hosts
shm                                                                                                                  64M     0   64M   0% /dev/shm
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77  1.0G     0  1.0G   0% /var/lib/www/html
tmpfs                                                                                                               3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/asound
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/acpi
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/scsi
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/firmware
# echo testing >/var/lib/www/html/a
# cat /var/lib/www/html/a
testing
# 
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl exec -it csirbd-demo-pod -- sh
# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          28G  6.0G   21G  23% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        28G  6.0G   21G  23% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           2.0G     0  2.0G   0% /proc/asound
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /proc/scsi
tmpfs           2.0G     0  2.0G   0% /sys/firmware
# echo testing > /var/lib/www/html/a
# cat /var/lib/www/html/a
testing
# 

* Now lets move the csi to pod networking by setting `CSI_ENABLE_HOST_NETWORK: "false"` in rook operator configmap and verify we have holder pods gets created

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get configmap/rook-ceph-operator-config -oyaml |grep CSI_ENABLE_HOST_NETWORK
  CSI_ENABLE_HOST_NETWORK: "false"

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get po
NAME                                            READY   STATUS      RESTARTS   AGE
csi-cephfsplugin-holder-my-cluster-lmwff        1/1     Running     0          23s
csi-cephfsplugin-ld5gq                          2/2     Running     0          21s
csi-cephfsplugin-provisioner-6c84b57bc9-gc4hf   5/5     Running     0          15m
csi-rbdplugin-holder-my-cluster-vv8cw           1/1     Running     0          23s
csi-rbdplugin-provisioner-66969c54d-kcp7v       5/5     Running     0          15m
csi-rbdplugin-xdrjh                             2/2     Running     0          21s
rook-ceph-exporter-minikube-866f865684-vw5wj    1/1     Running     0          13m
rook-ceph-mds-myfs-a-69c844c8cd-mtm89           1/1     Running     0          13m
rook-ceph-mds-myfs-b-6c9c556bff-r647x           1/1     Running     0          13m
rook-ceph-mgr-a-5f584dfc9d-s8pc8                1/1     Running     0          14m
rook-ceph-mon-a-644465bb44-4pkck                1/1     Running     0          14m
rook-ceph-operator-5f9bc6d66d-6ffsn             1/1     Running     0          15m
rook-ceph-osd-0-7966cb76b6-qrjcp                1/1     Running     0          13m
rook-ceph-osd-prepare-minikube-9qzs5            0/1     Completed   0          14m
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get ds
NAME                                 DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
csi-cephfsplugin                     1         1         1       1            1           <none>          15m
csi-cephfsplugin-holder-my-cluster   1         1         1       1            1           <none>          27s
csi-rbdplugin                        1         1         1       1            1           <none>          15m
csi-rbdplugin-holder-my-cluster      1         1         1       1            1           <none>          27s
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get po/csi-cephfsplugin-ld5gq -oyaml |grep -i hostnetwork
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get po/csi-rbdplugin-xdrjh -oyaml |grep -i hostnetwork
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get cmrc get cm rook-ceph-csi-config -oyaml
apiVersion: v1
data:
  csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"/var/lib/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/rook-ceph.net.ns","subvolumeGroup":"","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"/var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com/rook-ceph.net.ns","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":["kubernetes.io/hostname","topology.kubernetes.io/region","topology.kubernetes.io/zone","topology.rook.io/chassis","topology.rook.io/rack","topology.rook.io/row","topology.rook.io/pdu","topology.rook.io/pod","topology.rook.io/room","topology.rook.io/datacenter"]},"namespace":"rook-ceph"},{"clusterID":"e1026845ad66577abae1d16671b464c8","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"","subvolumeGroup":"csi","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":null},"namespace":"rook-ceph"}]'
kind: ConfigMap
metadata:
  creationTimestamp: "2024-01-23T13:46:16Z"
  name: rook-ceph-csi-config
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: false
    controller: true
    kind: Deployment
    name: rook-ceph-operator
    uid: 6df65e36-c138-42b8-a5c5-6b372b951d99
  resourceVersion: "1993"
  uid: 895fab41-2237-412b-b866-895793d97dbb

## Migration from Pod to Host Network

* Now the csi pods are moved to pod network lets do i/o and verify we are able to delete and recreate the same app pods without any problem

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csirbd-demo-pod -- sh
# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          28G  6.0G   21G  23% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        28G  6.0G   21G  23% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           2.0G     0  2.0G   0% /proc/asound
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /proc/scsi
tmpfs           2.0G     0  2.0G   0% /sys/firmware
# cat /var/lib/www/html/a
testing
# echo test > /var/lib/www/html/b
# exit
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csicephfs-demo-pod -- sh
# df -h
Filesystem                                                                                                          Size  Used Avail Use% Mounted on
overlay                                                                                                              28G  6.0G   21G  23% /
tmpfs                                                                                                                64M     0   64M   0% /dev
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1                                                                                                            28G  6.0G   21G  23% /etc/hosts
shm                                                                                                                  64M     0   64M   0% /dev/shm
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77  1.0G     0  1.0G   0% /var/lib/www/html
tmpfs                                                                                                               3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/asound
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/acpi
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/scsi
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/firmware
# cat /var/lib/www/html/a
testing
# echo test > /var/lib/www/html/b
# exit

* Lets delete the pod and check if we are able to recreate it

* Verify mount exists on the node

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ mount |grep cephfs
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/pods/263eb6ca-8f61-4857-99bc-2fe649d1cd6e/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/pods/263eb6ca-8f61-4857-99bc-2fe649d1cd6e/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
$ mount |grep rbd
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16)
/dev/rbd0 on /var/lib/kubelet/pods/e313fc91-74d4-4732-94d6-597d3d0ef3c7/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/pods/e313fc91-74d4-4732-94d6-597d3d0ef3c7/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16)
$ 

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          14m
csirbd-demo-pod      1/1     Running   0          13m
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl delete po/csicephfs-demo-pod
pod "csicephfs-demo-pod" deleted
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl delete po/csirbd-demo-pod
pod "csirbd-demo-pod" deleted
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
No resources found in default namespace.

* validate mount exists on the node now

$ mount |grep cephfs
$ mount |grep rbd   
$ 

* Lets recreate the app pods and check if we are using pod networking for csi now

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          7s
csirbd-demo-pod      1/1     Running   0          13s

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc logs po/csi-cephfsplugin-ld5gq -c csi-cephfsplugin
I0123 14:09:06.562980   51254 cephcmds.go:71] ID: 15 Req-ID: 0001-0009-rook-ceph-0000000000000001-9e506c9d-10fc-45e1-9a67-6f1345b279d3 command succeeded: nsenter [--net=/var/lib/kubelet/plugins/rook-ceph.cephfs.csi.ceph.com/rook-ceph.net.ns -- mount -t ceph 10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount -o name=csi-cephfs-node,secretfile=/tmp/csi/keys/keyfile-3725154805,mds_namespace=myfs,_netdev]
I0123 14:09:06.563033   51254 nodeserver.go:252] ID: 15 Req-ID: 0001-0009-rook-ceph-0000000000000001-9e506c9d-10fc-45e1-9a67-6f1345b279d3 cephfs: successfully mounted volume 0001-0009-rook-ceph-0000000000000001-9e506c9d-10fc-45e1-9a67-6f1345b279d3 to /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount
I0123 14:09:06.563113   51254 utils.go:171] ID: 15 Req-ID: 0001-0009-rook-ceph-0000000000000001-9e506c9d-10fc-45e1-9a67-6f1345b279d3 GRPC response: {}

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc logs po/csi-rbdplugin-xdrjh -c csi-rbdplugin
I0123 14:09:06.670447   51263 cephcmds.go:71] ID: 15 Req-ID: 0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 command succeeded: nsenter [--net=/var/lib/kubelet/plugins/rook-ceph.rbd.csi.ceph.com/rook-ceph.net.ns -- rbd --id csi-rbd-node -m 10.106.151.105:6789 --keyfile=***stripped*** map replicapool/csi-vol-30ed5397-b244-402c-b21a-65f3032b3956 --device-type krbd --options noudev]
I0123 14:09:06.670508   51263 nodeserver.go:422] ID: 15 Req-ID: 0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 rbd image: replicapool/csi-vol-30ed5397-b244-402c-b21a-65f3032b3956 was successfully mapped at /dev/rbd0

We can check that cephfs/rbd mount is done on the holder pod now.

* Lets again do the i/o now

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csicephfs-demo-pod -- sh
# cat /var/lib/www/html/a
testing
# echo test > /var/lib/www/html/c
# df -h
Filesystem                                                                                                          Size  Used Avail Use% Mounted on
overlay                                                                                                              28G  6.0G   21G  23% /
tmpfs                                                                                                                64M     0   64M   0% /dev
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1                                                                                                            28G  6.0G   21G  23% /etc/hosts
shm                                                                                                                  64M     0   64M   0% /dev/shm
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77  1.0G     0  1.0G   0% /var/lib/www/html
tmpfs                                                                                                               3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/asound
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/acpi
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/scsi
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/firmware
# 
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csirbd-demo-pod -- sh
# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          28G  6.0G   21G  23% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        28G  6.0G   21G  23% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           2.0G     0  2.0G   0% /proc/asound
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /proc/scsi
tmpfs           2.0G     0  2.0G   0% /sys/firmware
# cat /var/lib/www/html/a
testing
# echo test > /var/lib/www/html/c
# 

* Now lets move the csi to host network

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get configmap/rook-ceph-operator-config -oyaml |grep CSI_ENABLE_HOST_NETWORK
CSI_ENABLE_HOST_NETWORK: "true"

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get po/csi-cephfsplugin-265j9 -oyaml |grep -i hostnetwork
  hostNetwork: true
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get po/csi-rbdplugin-hfbp6 -oyaml |grep -i hostnetwork
  hostNetwork: true

* There is a small bug in Rook if a csi pod is moved to host network, the netnamespace path is not removed from csi configmap, for now lets remove it manually by editing csi configmap

🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc edit cm rook-ceph-csi-config
configmap/rook-ceph-csi-config edited
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kuberc get cm rook-ceph-csi-config -oyaml
apiVersion: v1
data:
  csi-cluster-config-json: '[{"clusterID":"rook-ceph","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"","subvolumeGroup":"","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":["kubernetes.io/hostname","topology.kubernetes.io/region","topology.kubernetes.io/zone","topology.rook.io/chassis","topology.rook.io/rack","topology.rook.io/row","topology.rook.io/pdu","topology.rook.io/pod","topology.rook.io/room","topology.rook.io/datacenter"]},"namespace":"rook-ceph"},{"clusterID":"e1026845ad66577abae1d16671b464c8","monitors":["10.106.151.105:6789"],"cephFS":{"netNamespaceFilePath":"","subvolumeGroup":"csi","kernelMountOptions":"","fuseMountOptions":""},"rbd":{"netNamespaceFilePath":"","radosNamespace":""},"nfs":{"netNamespaceFilePath":""},"readAffinity":{"enabled":false,"crushLocationLabels":null},"namespace":"rook-ceph"}]'
kind: ConfigMap
metadata:
  creationTimestamp: "2024-01-23T13:46:16Z"
  name: rook-ceph-csi-config
  namespace: rook-ceph
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: false
    controller: true
    kind: Deployment
    name: rook-ceph-operator
    uid: 6df65e36-c138-42b8-a5c5-6b372b951d99
  resourceVersion: "2778"
  uid: 895fab41-2237-412b-b866-895793d97dbb

* Now lets try to access the existing applications

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]minikube ssh
mou                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ mount |grep cephfs
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/pods/4123eb59-a6af-451f-ab07-e2dd16446bfe/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/pods/4123eb59-a6af-451f-ab07-e2dd16446bfe/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
$ mount |grep rbd   
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16)
/dev/rbd0 on /var/lib/kubelet/pods/7b5448a2-7e22-4548-a35e-b4d4acd992ca/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/pods/7b5448a2-7e22-4548-a35e-b4d4acd992ca/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16)
$ 

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csirbd-demo-pod -- sh
# cat /var/lib/www/html/c
test
# echo again > /var/lib/www/html/d
# df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          28G  6.0G   21G  23% /
tmpfs            64M     0   64M   0% /dev
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1        28G  6.0G   21G  23% /etc/hosts
shm              64M     0   64M   0% /dev/shm
/dev/rbd0       976M  2.6M  958M   1% /var/lib/www/html
tmpfs           3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           2.0G     0  2.0G   0% /proc/asound
tmpfs           2.0G     0  2.0G   0% /proc/acpi
tmpfs           2.0G     0  2.0G   0% /proc/scsi
tmpfs           2.0G     0  2.0G   0% /sys/firmware
# exit
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csicephfs-demo-pod -- sh
# df -h
Filesystem                                                                                                          Size  Used Avail Use% Mounted on
overlay                                                                                                              28G  6.0G   21G  23% /
tmpfs                                                                                                                64M     0   64M   0% /dev
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/vda1                                                                                                            28G  6.0G   21G  23% /etc/hosts
shm                                                                                                                  64M     0   64M   0% /dev/shm
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77  1.0G     0  1.0G   0% /var/lib/www/html
tmpfs                                                                                                               3.9G   12K  3.9G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/asound
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/acpi
tmpfs                                                                                                               2.0G     0  2.0G   0% /proc/scsi
tmpfs                                                                                                               2.0G     0  2.0G   0% /sys/firmware
# cat /var/lib/www/html/c
test
# echo again > /var/lib/www/html/d
# 

* Now lets delete the application pods and verify the mount exists on the host

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          12m
csirbd-demo-pod      1/1     Running   0          12m
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl deletetl delete po/csirbd-demo-pod
pod "csirbd-demo-pod" deleted
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl deletetl delete po/csicephfs-demo-pod
pod "csicephfs-demo-pod" deleted
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
No resources found in default namespace.
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]minikube ssh
                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ mount |grep cephfs
$ mount |grep rbd
$ ll /dev/r 
random  rfkill  rtc     rtc0    
$ ll /dev/r
random  rfkill  rtc     rtc0    
$ ll /dev/r
-bash: ll: command not found
$ 

* lets recreate the application and see

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl createtl create -f ../cephfs/pod.yaml 
pod/csicephfs-demo-pod created
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl createtl create -f pod.yaml 
pod/csirbd-demo-pod created
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl get po
NAME                 READY   STATUS    RESTARTS   AGE
csicephfs-demo-pod   1/1     Running   0          31s
csirbd-demo-pod      1/1     Running   0          29s
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csicephfs-demo-pod -- sh
# cat /var/lib/www/html/d
again
# exit
[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]kubectl execectl exec -it csirbd-demo-pod -- sh
# cat /var/lib/www/html/d
again
# 

[🎩︎]mrajanna@li-2cfbef4c-22d9-11b2-a85c-a3e4a93c405f rbd $]minikube ssh
mou                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

mou$ mount |grep cephfs
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.cephfs.csi.ceph.com/de8695c15b0729676ef3f47ec8b12c4b65a70d1a1e7ac7a4531294fe30eb3bbb/globalmount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /var/lib/kubelet/pods/0b9ee432-bb08-442e-aa9e-4b066d6b0527/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs,_netdev)
10.106.151.105:6789:/volumes/csi/csi-vol-9e506c9d-10fc-45e1-9a67-6f1345b279d3/e61b8727-311e-42a6-83cb-07fff99f7c77 on /mnt/vda1/var/lib/kubelet/pods/0b9ee432-bb08-442e-aa9e-4b066d6b0527/volumes/kubernetes.io~csi/pvc-b021a7b1-d58d-4b89-8514-6e43671e6299/mount type ceph (rw,relatime,name=csi-cephfs-node,secret=<hidden>,fsid=00000000-0000-0000-0000-000000000000,acl,mds_namespace=myfs)
$ mount |grep rbd
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/7d0b17024664d12a9a6c112a522996ed73f68349b1b2ea9f8426e2abf852623d/globalmount/0001-0009-rook-ceph-0000000000000002-30ed5397-b244-402c-b21a-65f3032b3956 type ext4 (rw,relatime,stripe=16)
/dev/rbd0 on /var/lib/kubelet/pods/7f14f942-39ff-4a0b-84fb-f88cef14017c/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16,_netdev)
/dev/rbd0 on /mnt/vda1/var/lib/kubelet/pods/7f14f942-39ff-4a0b-84fb-f88cef14017c/volumes/kubernetes.io~csi/pvc-b27dac9f-0ce0-47ab-b1e8-7e0fd1452bf7/mount type ext4 (rw,relatime,stripe=16)
$