---
# System prepended metadata

title: 'CephCSI part 2: logs'

---

# CephCSI part 2: logs

pods:
```
csi-addons-controller-manager-6c57bc4f4f-hscb6                    2/2     Running             0          20m
csi-cephfsplugin-dkcnl                                            2/2     Running             0          16m
csi-cephfsplugin-mvfwp                                            2/2     Running             0          16m
csi-cephfsplugin-p5glz                                            2/2     Running             0          16m
csi-cephfsplugin-provisioner-7b9dfb49b9-ssln5                     5/5     Running             0          16m
csi-cephfsplugin-provisioner-7b9dfb49b9-xvw9z                     5/5     Running             0          16m
csi-rbdplugin-674kd                                               3/3     Running             0          16m
csi-rbdplugin-provisioner-5d8b9d46f9-jzjhw                        6/6     Running             0          16m
csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd                        6/6     Running             0          16m
csi-rbdplugin-rpjzh                                               3/3     Running             0          16m
csi-rbdplugin-xwd5v                                               3/3     Running             0          16m
noobaa-core-0                                                     0/1     Pending             0          10m
noobaa-db-pg-0                                                    0/1     Pending             0          10m
noobaa-operator-5c54457684-pts27                                  1/1     Running             0          20m
ocs-metrics-exporter-669656c4d-8rncd                              1/1     Running             0          20m
ocs-operator-869864fc87-hzsl9                                     1/1     Running             0          20m
odf-console-6f7f8b49f6-8mhq6                                      1/1     Running             0          11m
odf-operator-controller-manager-676dc95dff-gz7jc                  2/2     Running             0          11m
rook-ceph-crashcollector-3a237eefc2aa94160e267ef4e49efd2f-86cs5   1/1     Running             0          11m
rook-ceph-crashcollector-be82a8d9410eaf334ad97fb27b2286d9-5qghx   1/1     Running             0          11m
rook-ceph-crashcollector-fc71fffeafcb84aa030f611eb1be8737-49dlb   1/1     Running             0          11m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-747f97c878vk4   2/2     Running             0          11m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-7578f8c7826w8   2/2     Running             0          11m
rook-ceph-mgr-a-69b7c9c584-zf8mq                                  2/2     Running             0          11m
rook-ceph-mon-a-6749c6cc-6td6s                                    2/2     Running             0          15m
rook-ceph-mon-b-7b84589967-f4zb9                                  2/2     Running             0          13m
rook-ceph-mon-c-6989b7c6f7-nns65                                  2/2     Running             0          12m
rook-ceph-operator-868f888df8-bhpbg                               1/1     Running             0          20m
rook-ceph-osd-0-7c8658dc49-gvmw5                                  2/2     Running             0          11m
rook-ceph-osd-1-7449445749-9mpjn                                  2/2     Running             0          11m
rook-ceph-osd-2-69f66db75-wwdrm                                   2/2     Running             0          11m
rook-ceph-osd-prepare-040c9282141fc0dcb2985f28c1915992-k8m2s      0/1     Completed           0          11m
rook-ceph-osd-prepare-30d09b72f83d59daa01c52015db63ff3-7fs6q      0/1     Completed           0          11m
rook-ceph-osd-prepare-f8bb2aedd5676bc25e0cc64d68d88847-fhf95      0/1     Completed           0          11m
```


storageclass:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes
  creationTimestamp: "2023-06-08T10:42:39Z"
  name: ocs-storagecluster-ceph-rbd
  resourceVersion: "68643"
  uid: fa834175-bd4c-4411-883c-f9e3de713f96
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
  imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
  imageFormat: "2"
  pool: ocs-storagecluster-cephblockpool
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```

snapshotclass:
```yaml
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: openshift-storage.rbd.csi.ceph.com
kind: VolumeSnapshotClass
metadata:
  creationTimestamp: "2023-06-08T10:42:39Z"
  generation: 1
  name: ocs-storagecluster-rbdplugin-snapclass
  resourceVersion: "68646"
  uid: e2e7737b-f4c0-4048-8d75-23bb7c536d67
parameters:
  clusterID: openshift-storage
  csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
```

### fresh-pvc:
```yaml
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
      volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    creationTimestamp: "2023-06-08T10:50:47Z"
    finalizers:
    - kubernetes.io/pvc-protection
    - provisioner.storage.kubernetes.io/cloning-protection
    name: fresh-pvc
    namespace: openshift-storage
    resourceVersion: "73583"
    uid: e270178a-ebb0-48ec-82c9-345994b96a62
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: ocs-storagecluster-ceph-rbd
    volumeMode: Filesystem
    volumeName: pvc-e270178a-ebb0-48ec-82c9-345994b96a62
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
```

pvc-describe:
```
rakshith:ceph-csi$ k describe pvc fresh-pvc 
Name:          fresh-pvc
Namespace:     openshift-storage
StorageClass:  ocs-storagecluster-ceph-rbd
Status:        Bound
Volume:        pvc-e270178a-ebb0-48ec-82c9-345994b96a62
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection provisioner.storage.kubernetes.io/cloning-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                 Age                From                                                                                                                Message
  ----    ------                 ----               ----                                                                                                                -------
  Normal  ExternalProvisioning   13m (x2 over 13m)  persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
  Normal  Provisioning           13m                openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13  External provisioner is provisioning volume for claim "openshift-storage/fresh-pvc"
  Normal  ProvisioningSucceeded  13m                openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13  Successfully provisioned volume pvc-e270178a-ebb0-48ec-82c9-345994b96a62
```

create
```bash
I0608 10:50:47.841082       1 utils.go:195] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:50:47.841274       1 utils.go:206] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e270178a-ebb0-48ec-82c9-345994b96a62","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-e270178a-ebb0-48ec-82c9-345994b96a62","csi.storage.k8s.io/pvc/name":"fresh-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0608 10:50:47.841414       1 rbd_util.go:1279] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 setting disableInUseChecks: false image features: [layering fast-diff deep-flatten exclusive-lock object-map] mounter: rbd
I0608 10:50:47.842619       1 omap.go:88] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:50:47.848661       1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-e270178a-ebb0-48ec-82c9-345994b96a62:f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d])
I0608 10:50:47.851169       1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage])
I0608 10:50:47.851190       1 rbd_journal.go:487] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 generated Volume ID (0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d) and image name (csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d) for request name (pvc-e270178a-ebb0-48ec-82c9-345994b96a62)
I0608 10:50:47.851262       1 rbd_util.go:414] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 rbd: create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d size 10240M (features: [layering fast-diff deep-flatten exclusive-lock object-map]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:50:47.851289       1 rbd_util.go:1526] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 setting image options on ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 10:50:47.870392       1 controllerserver.go:730] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 created image ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d backed for request name pvc-e270178a-ebb0-48ec-82c9-345994b96a62
I0608 10:50:47.883744       1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256])
I0608 10:50:47.883853       1 utils.go:212] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC response: {"volume":{"capacity_bytes":10737418240,"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}
```

delete
```bash
I0608 11:18:03.017306       1 utils.go:195] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:18:03.017460       1 utils.go:206] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}
I0608 11:18:03.019659       1 omap.go:88] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
I0608 11:18:03.053682       1 rbd_util.go:619] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: delete csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:18:03.057715       1 controllerserver.go:1004] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d deleting image csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 11:18:03.057733       1 rbd_util.go:619] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: delete csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:18:03.093592       1 rbd_util.go:654] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d" with id "374949d1a256" from trash
I0608 11:18:03.100978       1 rbd_util.go:678] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d" with id "374949d1a256" to trash
I0608 11:18:03.106250       1 omap.go:125] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-e270178a-ebb0-48ec-82c9-345994b96a62]
I0608 11:18:03.106335       1 utils.go:212] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC response: {}

```

### snapshot:
```yaml
apiVersion: v1
items:
- apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshot
  metadata:
    annotations:
      snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
      snapshot.storage.kubernetes.io/pvc-storage-class: ocs-storagecluster-ceph-rbd
      snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
    creationTimestamp: "2023-06-08T10:51:06Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
    generation: 1
    name: fresh-pvc-snapshot
    namespace: openshift-storage
    resourceVersion: "73112"
    uid: 84f627ad-1d04-4eeb-b631-6a2e6a9196ac
  spec:
    source:
      persistentVolumeClaimName: fresh-pvc
    volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass
  status:
    boundVolumeSnapshotContentName: snapcontent-84f627ad-1d04-4eeb-b631-6a2e6a9196ac
    creationTime: "2023-06-08T10:51:07Z"
    readyToUse: true
    restoreSize: 10Gi
kind: List
metadata:
  resourceVersion: ""
```

describe
```
rakshith:ceph-csi$ k describe vs fresh-pvc-snapshot 
Name:         fresh-pvc-snapshot
Namespace:    openshift-storage
Labels:       <none>
Annotations:  snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
              snapshot.storage.kubernetes.io/pvc-storage-class: ocs-storagecluster-ceph-rbd
              snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2023-06-08T11:16:26Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:snapshot.storage.kubernetes.io/pvc-access-modes:
          f:snapshot.storage.kubernetes.io/pvc-storage-class:
          f:snapshot.storage.kubernetes.io/pvc-volume-mode:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:      Mozilla
    Operation:    Update
    Time:         2023-06-08T11:16:26Z
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection":
          v:"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection":
    Manager:      snapshot-controller
    Operation:    Update
    Time:         2023-06-08T11:16:26Z
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:boundVolumeSnapshotContentName:
        f:creationTime:
        f:readyToUse:
        f:restoreSize:
    Manager:         snapshot-controller
    Operation:       Update
    Subresource:     status
    Time:            2023-06-08T11:16:28Z
  Resource Version:  86077
  UID:               0b406f0b-301d-4eaa-87d8-ad86d3161def
Spec:
  Source:
    Persistent Volume Claim Name:  fresh-pvc
  Volume Snapshot Class Name:      ocs-storagecluster-rbdplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-0b406f0b-301d-4eaa-87d8-ad86d3161def
  Creation Time:                       2023-06-08T11:16:27Z
  Ready To Use:                        true
  Restore Size:                        10Gi
Events:
  Type    Reason            Age   From                 Message
  ----    ------            ----  ----                 -------
  Normal  CreatingSnapshot  16s   snapshot-controller  Waiting for a snapshot openshift-storage/fresh-pvc-snapshot to be created by the CSI driver.
  Normal  SnapshotCreated   14s   snapshot-controller  Snapshot openshift-storage/fresh-pvc-snapshot was successfully created by the CSI driver.
  Normal  SnapshotReady     14s   snapshot-controller  Snapshot openshift-storage/fresh-pvc-snapshot is ready to use.
```

create
```bash
rakshith:ceph-csi$ cat csi.logs | grep "ID: 23"
I0608 10:51:06.758527       1 utils.go:195] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC call: /csi.v1.Controller/CreateSnapshot
I0608 10:51:06.758667       1 utils.go:206] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC request: {"name":"snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/volumesnapshot/name":"fresh-pvc-snapshot","csi.storage.k8s.io/volumesnapshot/namespace":"openshift-storage","csi.storage.k8s.io/volumesnapshotcontent/name":"snapcontent-84f627ad-1d04-4eeb-b631-6a2e6a9196ac"},"secrets":"***stripped***","source_volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}
I0608 10:51:06.760324       1 omap.go:88] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
E0608 10:51:06.772963       1 omap.go:79] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac omap not found (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): rados: ret=-2, No such file or directory
I0608 10:51:06.790147       1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): map[csi.snap.snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac:9db2818a-f217-43b1-b62b-48d3213e77fc])
I0608 10:51:06.792903       1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage])
I0608 10:51:06.792922       1 rbd_journal.go:407] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac generated Volume ID (0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc) and image name (csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc) for request name (snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac)
I0608 10:51:06.792965       1 rbd_util.go:1384] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.220042       1 rbd_util.go:1422] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: clone ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc (features: [layering deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.220077       1 rbd_util.go:1526] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac setting image options on ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:07.299811       1 rbd_util.go:1397] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.326191       1 rbd_util.go:1384] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:08.253443       1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da])
I0608 10:51:08.291243       1 rbd_util.go:794] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac clone depth is (1), configured softlimit (4) and hardlimit (8) for ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:08.388990       1 utils.go:212] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC response: {"snapshot":{"creation_time":{"nanos":242609242,"seconds":1686221467},"ready_to_use":true,"size_bytes":10737418240,"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc","source_volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}
```

delete
```bash
I0608 10:52:07.309288       1 utils.go:212] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}}
I0608 11:06:04.526310       1 utils.go:195] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC call: /csi.v1.Controller/DeleteSnapshot
I0608 11:06:04.526461       1 utils.go:206] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC request: {"secrets":"***stripped***","snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}
I0608 11:06:04.528007       1 omap.go:88] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 11:06:04.554944       1 controllerserver.go:1440] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc deleting cloned rbd volume csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 11:06:04.554999       1 rbd_util.go:1397] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: snap rm ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 11:06:04.584522       1 rbd_util.go:619] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: delete csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:06:04.618899       1 rbd_util.go:654] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc" with id "37492d2e39da" from trash
I0608 11:06:04.627503       1 rbd_util.go:678] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc" with id "37492d2e39da" to trash
I0608 11:06:04.632759       1 omap.go:125] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): [csi.snap.snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac]
I0608 11:06:04.632819       1 utils.go:212] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC response: {}
```

### restore pvc:
```yaml
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
      volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    creationTimestamp: "2023-06-08T10:51:33Z"
    finalizers:
    - kubernetes.io/pvc-protection
    name: restore-pvc
    namespace: openshift-storage
    resourceVersion: "73315"
    uid: 28b5c76e-9313-4df1-b2bb-88229b42e410
  spec:
    accessModes:
    - ReadWriteOnce
    dataSource:
      apiGroup: snapshot.storage.k8s.io
      kind: VolumeSnapshot
      name: fresh-pvc-snapshot
    dataSourceRef:
      apiGroup: snapshot.storage.k8s.io
      kind: VolumeSnapshot
      name: fresh-pvc-snapshot
    resources:
      requests:
        storage: 10Gi
    storageClassName: ocs-storagecluster-ceph-rbd
    volumeMode: Filesystem
    volumeName: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
```

create
```bash
I0608 10:51:33.979780       1 utils.go:195] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:51:33.979934       1 utils.go:206] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-28b5c76e-9313-4df1-b2bb-88229b42e410","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-28b5c76e-9313-4df1-b2bb-88229b42e410","csi.storage.k8s.io/pvc/name":"restore-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}}}}
I0608 10:51:33.980065       1 rbd_util.go:1279] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 setting disableInUseChecks: false image features: [exclusive-lock object-map layering fast-diff deep-flatten] mounter: rbd
I0608 10:51:33.981456       1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 10:51:34.008805       1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:51:34.020156       1 rbd_util.go:794] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 clone depth is (0), configured softlimit (3) and hardlimit (7) for ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 10:51:34.026038       1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-28b5c76e-9313-4df1-b2bb-88229b42e410:d1ca5033-4fea-4c6c-adf9-a44800007426])
I0608 10:51:34.029051       1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imagename:csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 csi.volname:pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 csi.volume.owner:openshift-storage])
I0608 10:51:34.029073       1 rbd_journal.go:487] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 generated Volume ID (0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426) and image name (csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426) for request name (pvc-28b5c76e-9313-4df1-b2bb-88229b42e410)
I0608 10:51:34.029645       1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 10:51:34.052791       1 rbd_util.go:1422] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 rbd: clone ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 (features: [deep-flatten layering fast-diff object-map exclusive-lock]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:34.052814       1 rbd_util.go:1526] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 setting image options on ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426
I0608 10:51:34.288757       1 controllerserver.go:665] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 create volume ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 from snapshot ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:34.288781       1 controllerserver.go:730] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 created image ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 backed for request name pvc-28b5c76e-9313-4df1-b2bb-88229b42e410
I0608 10:51:34.326299       1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imageid:37492d08a553])
I0608 10:51:34.326455       1 utils.go:212] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Snapshot":{"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426"}}
```

delete
```bash
I0608 11:48:17.278911       1 utils.go:195] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:48:17.278993       1 utils.go:206] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426"}
I0608 11:48:17.282247       1 omap.go:88] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imageid:37492d08a553 csi.imagename:csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 csi.volname:pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 csi.volume.owner:openshift-storage]
I0608 11:48:17.401805       1 rbd_util.go:619] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: delete csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:48:17.405719       1 controllerserver.go:1004] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 deleting image csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426
I0608 11:48:17.405732       1 rbd_util.go:619] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: delete csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:48:17.453235       1 rbd_util.go:654] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426" with id "37492d08a553" from trash
I0608 11:48:17.461364       1 rbd_util.go:678] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426" with id "37492d08a553" to trash
I0608 11:48:17.466728       1 omap.go:125] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-28b5c76e-9313-4df1-b2bb-88229b42e410]
I0608 11:48:17.466801       1 utils.go:212] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC response: {}
```

### clone pvc:
```yaml
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
      volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
    creationTimestamp: "2023-06-08T10:52:05Z"
    finalizers:
    - kubernetes.io/pvc-protection
    name: clone-pvc
    namespace: openshift-storage
    resourceVersion: "73600"
    uid: 75c0fbe3-271b-4486-a22c-0110b13234da
  spec:
    accessModes:
    - ReadWriteOnce
    dataSource:
      apiGroup: ""
      kind: PersistentVolumeClaim
      name: fresh-pvc
    dataSourceRef:
      apiGroup: ""
      kind: PersistentVolumeClaim
      name: fresh-pvc
    resources:
      requests:
        storage: 10Gi
    storageClassName: ocs-storagecluster-ceph-rbd
    volumeMode: Filesystem
    volumeName: pvc-75c0fbe3-271b-4486-a22c-0110b13234da
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
```

create
```bash
I0608 10:52:05.952793       1 utils.go:195] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:52:05.952995       1 utils.go:206] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-75c0fbe3-271b-4486-a22c-0110b13234da","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-75c0fbe3-271b-4486-a22c-0110b13234da","csi.storage.k8s.io/pvc/name":"clone-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}}}
I0608 10:52:05.953138       1 rbd_util.go:1279] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting disableInUseChecks: false image features: [object-map exclusive-lock deep-flatten layering fast-diff] mounter: rbd
I0608 10:52:05.954561       1 omap.go:88] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
I0608 10:52:05.968432       1 omap.go:88] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:52:06.001089       1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-75c0fbe3-271b-4486-a22c-0110b13234da:cae7f13f-3e63-46ef-a4b6-8809b3b5a87b])
I0608 10:52:06.003841       1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage])
I0608 10:52:06.003861       1 rbd_journal.go:487] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da generated Volume ID (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) and image name (csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) for request name (pvc-75c0fbe3-271b-4486-a22c-0110b13234da)
I0608 10:52:06.003922       1 rbd_util.go:1384] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap create ocs-storagecluster-cephblockpool/@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon 
I0608 10:52:06.157382       1 rbd_util.go:1422] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: clone ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp (features: [layering deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:52:06.157409       1 rbd_util.go:1526] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting image options on ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp
I0608 10:52:06.215467       1 rbd_util.go:1397] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon 
I0608 10:52:06.306500       1 rbd_util.go:1384] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap create ocs-storagecluster-cephblockpool/@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon 
I0608 10:52:07.164524       1 rbd_util.go:1422] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: clone ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b (features: [exclusive-lock object-map layering fast-diff deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:52:07.164554       1 rbd_util.go:1526] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting image options on ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 10:52:07.254173       1 rbd_util.go:1397] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon 
I0608 10:52:07.309134       1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858])
I0608 10:52:07.309288       1 utils.go:212] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},
```

delete
```bash
I0608 11:49:33.777904       1 utils.go:195] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:49:33.777979       1 utils.go:206] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:49:33.779734       1 omap.go:88] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:49:33.870623       1 rbd_util.go:619] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: delete csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:49:33.913873       1 rbd_util.go:654] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp" with id "3749b014585d" from trash
I0608 11:49:33.920917       1 rbd_util.go:678] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp" with id "3749b014585d" to trash
I0608 11:49:33.920937       1 controllerserver.go:1004] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b deleting image csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:49:33.920945       1 rbd_util.go:619] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: delete csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:49:33.964784       1 rbd_util.go:654] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b" with id "374935432858" from trash
I0608 11:49:33.971440       1 rbd_util.go:678] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b" with id "374935432858" to trash
I0608 11:49:33.976466       1 omap.go:125] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-75c0fbe3-271b-4486-a22c-0110b13234da]
I0608 11:49:33.976539       1 utils.go:212] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}

```

### Expansion:
```bash
rakshith:ceph-csi$ k describe pvc clone-pvc 
Name:          clone-pvc
Namespace:     openshift-storage
StorageClass:  ocs-storagecluster-ceph-rbd
Status:        Bound
Volume:        pvc-75c0fbe3-271b-4486-a22c-0110b13234da
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      15Gi
Access Modes:  RWO
VolumeMode:    Filesystem
DataSource:
  APIGroup:  
  Kind:      PersistentVolumeClaim
  Name:      fresh-pvc
Used By:     csirbd-demo-pod
Events:
  Type     Reason                      Age                From                                                                                                                Message
  ----     ------                      ----               ----                                                                                                                -------
  Normal   Provisioning                28m                openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13  External provisioner is provisioning volume for claim "openshift-storage/clone-pvc"
  Normal   ExternalProvisioning        28m (x2 over 28m)  persistentvolume-controller                                                                                         waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
  Normal   ProvisioningSucceeded       28m                openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13  Successfully provisioned volume pvc-75c0fbe3-271b-4486-a22c-0110b13234da
  Warning  ExternalExpanding           118s               volume_expand                                                                                                       Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
  Normal   Resizing                    118s               external-resizer openshift-storage.rbd.csi.ceph.com                                                                 External resizer is resizing volume pvc-75c0fbe3-271b-4486-a22c-0110b13234da
  Normal   FileSystemResizeRequired    118s               external-resizer openshift-storage.rbd.csi.ceph.com                                                                 Require file system resize of volume on node
  Normal   FileSystemResizeSuccessful  60s (x2 over 60s)  kubelet                                                                                                             MountVolume.NodeExpandVolume succeeded for volume "pvc-75c0fbe3-271b-4486-a22c-0110b13234da" ip-10-0-221-171.us-west-2.compute.internal
```

controller logs
```bash
0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Controller/ControllerExpandVolume
I0608 11:18:55.182571       1 utils.go:206] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"capacity_range":{"required_bytes":16106127360},"secrets":"***stripped***","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:18:55.184281       1 omap.go:88] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:18:55.216929       1 controllerserver.go:1576] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd volume ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b size is 10737418240,resizing to 16106127360
I0608 11:18:55.267338       1 utils.go:212] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {"capacity_bytes":16106127360,"node_expansion_required":true}
```

node logs
```bash
I0608 11:19:53.222832  116821 utils.go:195] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeExpandVolume
I0608 11:19:53.222981  116821 utils.go:206] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"capacity_range":{"required_bytes":16106127360},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","volume_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount"}
I0608 11:19:53.240196  116821 cephcmds.go:105] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [device list --format=json --device-type krbd]
I0608 11:19:53.240267  116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.241607  116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.241631  116821 resizefs_linux.go:56] ResizeFS.Resize - Expanding mounted volume /dev/rbd0
I0608 11:19:53.243818  116821 resizefs_linux.go:71] Device /dev/rbd0 resized successfully
I0608 11:19:53.243874  116821 utils.go:212] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```

## csi pods + example:
```
rakshith:ceph-csi$ k get po -o wide | grep csi
csi-addons-controller-manager-6c57bc4f4f-hscb6                    2/2     Running     0          21m    10.128.2.22    ip-10-0-156-223.us-west-2.compute.internal   <none>           <none>
csi-cephfsplugin-dkcnl                                            2/2     Running     0          17m    10.0.167.49    ip-10-0-167-49.us-west-2.compute.internal    <none>           <none>
csi-cephfsplugin-mvfwp                                            2/2     Running     0          17m    10.0.221.171   ip-10-0-221-171.us-west-2.compute.internal   <none>           <none>
csi-cephfsplugin-p5glz                                            2/2     Running     0          17m    10.0.156.223   ip-10-0-156-223.us-west-2.compute.internal   <none>           <none>
csi-cephfsplugin-provisioner-7b9dfb49b9-ssln5                     5/5     Running     0          17m    10.129.2.19    ip-10-0-167-49.us-west-2.compute.internal    <none>           <none>
csi-cephfsplugin-provisioner-7b9dfb49b9-xvw9z                     5/5     Running     0          17m    10.131.0.33    ip-10-0-221-171.us-west-2.compute.internal   <none>           <none>
csi-rbdplugin-674kd                                               3/3     Running     0          17m    10.0.156.223   ip-10-0-156-223.us-west-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-5d8b9d46f9-jzjhw                        6/6     Running     0          17m    10.128.2.24    ip-10-0-156-223.us-west-2.compute.internal   <none>           <none>
csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd                        6/6     Running     0          17m    10.131.0.32    ip-10-0-221-171.us-west-2.compute.internal   <none>           <none>
csi-rbdplugin-rpjzh                                               3/3     Running     0          17m    10.0.221.171   ip-10-0-221-171.us-west-2.compute.internal   <none>           <none>
csi-rbdplugin-xwd5v                                               3/3     Running     0          17m    10.0.167.49    ip-10-0-167-49.us-west-2.compute.internal    <none>           <none>
csirbd-demo-pod                                                   1/1     Running     0          119s   10.128.2.33    ip-10-0-156-223.us-west-2.compute.internal   <none>           <none>
```

### NodeStageVolume:
```bash
I0608 11:19:52.653283  116821 utils.go:195] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeStageVolume
I0608 11:19:52.653450  116821 utils.go:206] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:19:52.653642  116821 rbd_util.go:1279] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b setting disableInUseChecks: false image features: [layering fast-diff exclusive-lock object-map deep-flatten] mounter: rbd
I0608 11:19:52.669071  116821 omap.go:88] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:19:52.709565  116821 rbd_util.go:346] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b checking for ImageFeatures: [layering fast-diff operations exclusive-lock object-map deep-flatten]
I0608 11:19:52.727006  116821 cephcmds.go:105] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [device list --format=json --device-type krbd]
I0608 11:19:52.759598  116821 rbd_attach.go:420] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: map mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 11:19:52.863390  116821 cephcmds.go:105] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [--id csi-rbd-node -m 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789 --keyfile=***stripped*** map ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b --device-type krbd --options noudev]
I0608 11:19:52.863431  116821 nodeserver.go:414] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd image: ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b was successfully mapped at /dev/rbd0
I0608 11:19:52.863503  116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:52.867986  116821 mount_linux.go:566] Output: ""
I0608 11:19:53.126834  116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.142597  116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.142614  116821 mount_linux.go:452] Checking for issues with fsck on disk: /dev/rbd0
I0608 11:19:53.158110  116821 mount_linux.go:553] Attempting to mount disk /dev/rbd0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:19:53.158136  116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o _netdev,defaults /dev/rbd0 /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:19:53.199021  116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.203315  116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.203329  116821 resizefs_linux.go:124] ResizeFs.needResize - checking mounted volume /dev/rbd0
I0608 11:19:53.204851  116821 resizefs_linux.go:128] Ext size: filesystem size=16106127360, block size=4096
I0608 11:19:53.204863  116821 resizefs_linux.go:143] Volume /dev/rbd0: device size=16106127360, filesystem size=16106127360, block size=4096
I0608 11:19:53.204885  116821 nodeserver.go:374] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully mounted volume 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b to stagingTargetPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:19:53.204986  116821 utils.go:212] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```

### NodePublishVolume:
```bash
I0608 11:19:53.211124  116821 utils.go:195] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodePublishVolume
I0608 11:19:53.211277  116821 utils.go:206] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","target_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:19:53.211353  116821 nodeserver.go:802] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b target /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
isBlock false
fstype ext4
stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
readonly false
mountflags [bind _netdev]
I0608 11:19:53.211378  116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount)
I0608 11:19:53.213451  116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o bind,remount,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount)
I0608 11:19:53.214580  116821 nodeserver.go:700] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully mounted stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b to targetPath /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:19:53.214608  116821 utils.go:212] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```

### NodeUnpublishVolume:
```bash
I0608 11:25:27.079341  116821 utils.go:195] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0608 11:25:27.079441  116821 utils.go:206] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"target_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:25:27.079504  116821 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:25:27.083138  116821 nodeserver.go:888] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully unbound volume 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b from /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:25:27.083183  116821 utils.go:212] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```

### NodeUnstageVolume:
```bash
I0608 11:25:27.187622  116821 utils.go:195] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeUnstageVolume
I0608 11:25:27.187741  116821 utils.go:206] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:25:27.187837  116821 mount_linux.go:361] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:25:27.210704  116821 nodeserver.go:944] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b successfully unmounted volume (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) from staging path (/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:25:27.260510  116821 cephcmds.go:105] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [unmap ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b --device-type krbd --options noudev]
I0608 11:25:27.260535  116821 nodeserver.go:1002] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b successfully unmapped volume (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:25:27.260636  116821 utils.go:212] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```

----


# Reclaimspace Operation

### workflow
```bash
rakshith:kubernetes-csi-addons$ kubectl annotate pvc clone-pvc "reclaimspace.csiaddons.openshift.io/schedule=*/2 * * * *"
persistentvolumeclaim/clone-pvc annotated
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io 
NAME                   SCHEDULE      SUSPEND   ACTIVE   LASTSCHEDULE   AGE
clone-pvc-1686223828   */2 * * * *                                     12s
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io -w
NAME                              NAMESPACE           AGE   RETRIES   RESULT
clone-pvc-1686223828-1686223920   openshift-storage   0s              
clone-pvc-1686223828-1686223920   openshift-storage   0s              Succeeded
rakshith:kubernetes-csi-addons$ kubectl annotate pvc clone-pvc "reclaimspace.csiaddons.openshift.io/schedule-"
persistentvolumeclaim/clone-pvc annotated
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io 
No resources found in openshift-storage namespace.
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io 
No resources found in openshift-storage namespace.
```

### Annotation

```bash
2023-06-08T11:30:28.491Z	INFO	Successfully created reclaimSpaceCronJob	{"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"clone-pvc","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc", "reconcileID": "c887e64e-ab3c-4e60-bf2f-7df8649a8731", "Schedule": "*/2 * * * *", "ReclaimSpaceCronJobName": "clone-pvc-1686223828"}
```

### ReclaimSpaceCronJob

```bash
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io clone-pvc-1686223828 -o yaml
apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceCronJob
metadata:
  creationTimestamp: "2023-06-08T11:30:28Z"
  generation: 1
  name: clone-pvc-1686223828
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: v1
    blockOwnerDeletion: true
    controller: true
    kind: PersistentVolumeClaim
    name: clone-pvc
    uid: 75c0fbe3-271b-4486-a22c-0110b13234da
  resourceVersion: "94922"
  uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    metadata: {}
    spec:
      backOffLimit: 6
      retryDeadlineSeconds: 600
      target:
        persistentVolumeClaim: clone-pvc
  schedule: '*/2 * * * *'
  successfulJobsHistoryLimit: 3
status:
  lastScheduleTime: "2023-06-08T11:34:00Z"
  lastSuccessfulTime: "2023-06-08T11:34:00Z"
```


```bash
2023-06-08T11:34:00.008Z	INFO	Successfully created reclaimSpaceJob for reclaimSpaceCronJob run	{"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"clone-pvc-1686223828","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828", "reconcileID": "bfe9b421-c797-45b5-b82e-301ddb007f36", "now": "2023-06-08T11:34:00.004Z", "nextRun": "2023-06-08T11:36:00.000Z", "currentRun": "2023-06-08T11:34:00.000Z", "reclaimSpacejob": "clone-pvc-1686223828-1686224040"}
```

### ReclaimSpaceJob

```bash
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io -o yaml 
apiVersion: v1
items:
- apiVersion: csiaddons.openshift.io/v1alpha1
  kind: ReclaimSpaceJob
  metadata:
    annotations:
      csiaddons.openshift.io/scheduled-at: "2023-06-08T11:32:00Z"
    creationTimestamp: "2023-06-08T11:32:00Z"
    generation: 1
    name: clone-pvc-1686223828-1686223920
    namespace: openshift-storage
    ownerReferences:
    - apiVersion: csiaddons.openshift.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: ReclaimSpaceCronJob
      name: clone-pvc-1686223828
      uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
    resourceVersion: "93904"
    uid: 5f770abf-af1d-499d-9569-98ff6414fbf9
  spec:
    backOffLimit: 6
    retryDeadlineSeconds: 600
    target:
      persistentVolumeClaim: clone-pvc
  status:
    completionTime: "2023-06-08T11:32:00Z"
    message: Reclaim Space operation successfully completed.
    result: Succeeded
    startTime: "2023-06-08T11:32:00Z"
- apiVersion: csiaddons.openshift.io/v1alpha1
  kind: ReclaimSpaceJob
  metadata:
    annotations:
      csiaddons.openshift.io/scheduled-at: "2023-06-08T11:34:00Z"
    creationTimestamp: "2023-06-08T11:34:00Z"
    generation: 1
    name: clone-pvc-1686223828-1686224040
    namespace: openshift-storage
    ownerReferences:
    - apiVersion: csiaddons.openshift.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: ReclaimSpaceCronJob
      name: clone-pvc-1686223828
      uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
    resourceVersion: "94921"
    uid: a34b147c-fe12-4415-88a1-0adc389fcb76
  spec:
    backOffLimit: 6
    retryDeadlineSeconds: 600
    target:
      persistentVolumeClaim: clone-pvc
  status:
    completionTime: "2023-06-08T11:34:00Z"
    message: Reclaim Space operation successfully completed.
    result: Succeeded
    startTime: "2023-06-08T11:34:00Z"
kind: List
metadata:
  resourceVersion: ""
```

```bash
2023-06-08T11:32:00.472Z	INFO	No upcoming scheduled times, requeue with delay till next run	{"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"clone-pvc-1686223828","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828", "reconcileID": "ed5e6231-b487-489e-99a4-b9f4b7e7f278", "now": "2023-06-08T11:32:00.472Z", "nextRun": "2023-06-08T11:34:00.000Z"}
```

#### ReclaimSpace from CSI-Addons controller
```bash
2023-06-08T11:32:00.212Z	INFO	Making node reclaim space request	{"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh"}
2023-06-08T11:32:00.254Z	INFO	Making controller reclaim space request	{"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh", "controllerClient": "openshift-storage/csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd"}
2023-06-08T11:32:00.458Z	INFO	Successfully completed reclaim space operation	{"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh", "controllerClient": "openshift-storage/csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd"}
```

#### ControllerReclaimSpace
```bash
I0608 11:36:00.033207       1 utils.go:195] ID: 21 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0608 11:36:00.033301       1 utils.go:206] ID: 21 GRPC request: {"parameters":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:36:00.036733       1 omap.go:88] ID: 21 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:36:00.180510       1 utils.go:212] ID: 21 GRPC response: {}
```

#### NodeReclaimSpace
```bash
I0608 11:36:00.022006  116821 utils.go:195] ID: 53 GRPC call: /reclaimspace.ReclaimSpaceNode/NodeReclaimSpace
I0608 11:36:00.022109  116821 utils.go:206] ID: 53 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:36:00.023829  116821 cephcmds.go:105] ID: 53 command succeeded: fstrim [/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b]
I0608 11:36:00.023859  116821 utils.go:212] ID: 53 GRPC response: {}
```

