# CephCSI part 2: logs
pods:
```
csi-addons-controller-manager-6c57bc4f4f-hscb6 2/2 Running 0 20m
csi-cephfsplugin-dkcnl 2/2 Running 0 16m
csi-cephfsplugin-mvfwp 2/2 Running 0 16m
csi-cephfsplugin-p5glz 2/2 Running 0 16m
csi-cephfsplugin-provisioner-7b9dfb49b9-ssln5 5/5 Running 0 16m
csi-cephfsplugin-provisioner-7b9dfb49b9-xvw9z 5/5 Running 0 16m
csi-rbdplugin-674kd 3/3 Running 0 16m
csi-rbdplugin-provisioner-5d8b9d46f9-jzjhw 6/6 Running 0 16m
csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd 6/6 Running 0 16m
csi-rbdplugin-rpjzh 3/3 Running 0 16m
csi-rbdplugin-xwd5v 3/3 Running 0 16m
noobaa-core-0 0/1 Pending 0 10m
noobaa-db-pg-0 0/1 Pending 0 10m
noobaa-operator-5c54457684-pts27 1/1 Running 0 20m
ocs-metrics-exporter-669656c4d-8rncd 1/1 Running 0 20m
ocs-operator-869864fc87-hzsl9 1/1 Running 0 20m
odf-console-6f7f8b49f6-8mhq6 1/1 Running 0 11m
odf-operator-controller-manager-676dc95dff-gz7jc 2/2 Running 0 11m
rook-ceph-crashcollector-3a237eefc2aa94160e267ef4e49efd2f-86cs5 1/1 Running 0 11m
rook-ceph-crashcollector-be82a8d9410eaf334ad97fb27b2286d9-5qghx 1/1 Running 0 11m
rook-ceph-crashcollector-fc71fffeafcb84aa030f611eb1be8737-49dlb 1/1 Running 0 11m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-747f97c878vk4 2/2 Running 0 11m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-7578f8c7826w8 2/2 Running 0 11m
rook-ceph-mgr-a-69b7c9c584-zf8mq 2/2 Running 0 11m
rook-ceph-mon-a-6749c6cc-6td6s 2/2 Running 0 15m
rook-ceph-mon-b-7b84589967-f4zb9 2/2 Running 0 13m
rook-ceph-mon-c-6989b7c6f7-nns65 2/2 Running 0 12m
rook-ceph-operator-868f888df8-bhpbg 1/1 Running 0 20m
rook-ceph-osd-0-7c8658dc49-gvmw5 2/2 Running 0 11m
rook-ceph-osd-1-7449445749-9mpjn 2/2 Running 0 11m
rook-ceph-osd-2-69f66db75-wwdrm 2/2 Running 0 11m
rook-ceph-osd-prepare-040c9282141fc0dcb2985f28c1915992-k8m2s 0/1 Completed 0 11m
rook-ceph-osd-prepare-30d09b72f83d59daa01c52015db63ff3-7fs6q 0/1 Completed 0 11m
rook-ceph-osd-prepare-f8bb2aedd5676bc25e0cc64d68d88847-fhf95 0/1 Completed 0 11m
```
storageclass:
```yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
description: Provides RWO Filesystem volumes, and RWO and RWX Block volumes
creationTimestamp: "2023-06-08T10:42:39Z"
name: ocs-storagecluster-ceph-rbd
resourceVersion: "68643"
uid: fa834175-bd4c-4411-883c-f9e3de713f96
parameters:
clusterID: openshift-storage
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
imageFeatures: layering,deep-flatten,exclusive-lock,object-map,fast-diff
imageFormat: "2"
pool: ocs-storagecluster-cephblockpool
provisioner: openshift-storage.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
```
snapshotclass:
```yaml
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: openshift-storage.rbd.csi.ceph.com
kind: VolumeSnapshotClass
metadata:
creationTimestamp: "2023-06-08T10:42:39Z"
generation: 1
name: ocs-storagecluster-rbdplugin-snapclass
resourceVersion: "68646"
uid: e2e7737b-f4c0-4048-8d75-23bb7c536d67
parameters:
clusterID: openshift-storage
csi.storage.k8s.io/snapshotter-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/snapshotter-secret-namespace: openshift-storage
```
### fresh-pvc:
```yaml
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
creationTimestamp: "2023-06-08T10:50:47Z"
finalizers:
- kubernetes.io/pvc-protection
- provisioner.storage.kubernetes.io/cloning-protection
name: fresh-pvc
namespace: openshift-storage
resourceVersion: "73583"
uid: e270178a-ebb0-48ec-82c9-345994b96a62
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Filesystem
volumeName: pvc-e270178a-ebb0-48ec-82c9-345994b96a62
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
```
pvc-describe:
```
rakshith:ceph-csi$ k describe pvc fresh-pvc
Name: fresh-pvc
Namespace: openshift-storage
StorageClass: ocs-storagecluster-ceph-rbd
Status: Bound
Volume: pvc-e270178a-ebb0-48ec-82c9-345994b96a62
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection provisioner.storage.kubernetes.io/cloning-protection]
Capacity: 10Gi
Access Modes: RWO
VolumeMode: Filesystem
Used By: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 13m (x2 over 13m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
Normal Provisioning 13m openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13 External provisioner is provisioning volume for claim "openshift-storage/fresh-pvc"
Normal ProvisioningSucceeded 13m openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13 Successfully provisioned volume pvc-e270178a-ebb0-48ec-82c9-345994b96a62
```
create
```bash
I0608 10:50:47.841082 1 utils.go:195] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:50:47.841274 1 utils.go:206] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-e270178a-ebb0-48ec-82c9-345994b96a62","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-e270178a-ebb0-48ec-82c9-345994b96a62","csi.storage.k8s.io/pvc/name":"fresh-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0608 10:50:47.841414 1 rbd_util.go:1279] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 setting disableInUseChecks: false image features: [layering fast-diff deep-flatten exclusive-lock object-map] mounter: rbd
I0608 10:50:47.842619 1 omap.go:88] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:50:47.848661 1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-e270178a-ebb0-48ec-82c9-345994b96a62:f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d])
I0608 10:50:47.851169 1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage])
I0608 10:50:47.851190 1 rbd_journal.go:487] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 generated Volume ID (0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d) and image name (csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d) for request name (pvc-e270178a-ebb0-48ec-82c9-345994b96a62)
I0608 10:50:47.851262 1 rbd_util.go:414] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 rbd: create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d size 10240M (features: [layering fast-diff deep-flatten exclusive-lock object-map]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:50:47.851289 1 rbd_util.go:1526] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 setting image options on ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 10:50:47.870392 1 controllerserver.go:730] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 created image ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d backed for request name pvc-e270178a-ebb0-48ec-82c9-345994b96a62
I0608 10:50:47.883744 1 omap.go:158] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256])
I0608 10:50:47.883853 1 utils.go:212] ID: 21 Req-ID: pvc-e270178a-ebb0-48ec-82c9-345994b96a62 GRPC response: {"volume":{"capacity_bytes":10737418240,"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}
```
delete
```bash
I0608 11:18:03.017306 1 utils.go:195] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:18:03.017460 1 utils.go:206] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}
I0608 11:18:03.019659 1 omap.go:88] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
I0608 11:18:03.053682 1 rbd_util.go:619] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: delete csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:18:03.057715 1 controllerserver.go:1004] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d deleting image csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 11:18:03.057733 1 rbd_util.go:619] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: delete csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:18:03.093592 1 rbd_util.go:654] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d" with id "374949d1a256" from trash
I0608 11:18:03.100978 1 rbd_util.go:678] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d" with id "374949d1a256" to trash
I0608 11:18:03.106250 1 omap.go:125] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-e270178a-ebb0-48ec-82c9-345994b96a62]
I0608 11:18:03.106335 1 utils.go:212] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d GRPC response: {}
```
### snapshot:
```yaml
apiVersion: v1
items:
- apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
annotations:
snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
snapshot.storage.kubernetes.io/pvc-storage-class: ocs-storagecluster-ceph-rbd
snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
creationTimestamp: "2023-06-08T10:51:06Z"
finalizers:
- snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
- snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
generation: 1
name: fresh-pvc-snapshot
namespace: openshift-storage
resourceVersion: "73112"
uid: 84f627ad-1d04-4eeb-b631-6a2e6a9196ac
spec:
source:
persistentVolumeClaimName: fresh-pvc
volumeSnapshotClassName: ocs-storagecluster-rbdplugin-snapclass
status:
boundVolumeSnapshotContentName: snapcontent-84f627ad-1d04-4eeb-b631-6a2e6a9196ac
creationTime: "2023-06-08T10:51:07Z"
readyToUse: true
restoreSize: 10Gi
kind: List
metadata:
resourceVersion: ""
```
describe
```
rakshith:ceph-csi$ k describe vs fresh-pvc-snapshot
Name: fresh-pvc-snapshot
Namespace: openshift-storage
Labels: <none>
Annotations: snapshot.storage.kubernetes.io/pvc-access-modes: ReadWriteOnce
snapshot.storage.kubernetes.io/pvc-storage-class: ocs-storagecluster-ceph-rbd
snapshot.storage.kubernetes.io/pvc-volume-mode: Filesystem
API Version: snapshot.storage.k8s.io/v1
Kind: VolumeSnapshot
Metadata:
Creation Timestamp: 2023-06-08T11:16:26Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
Generation: 1
Managed Fields:
API Version: snapshot.storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:snapshot.storage.kubernetes.io/pvc-access-modes:
f:snapshot.storage.kubernetes.io/pvc-storage-class:
f:snapshot.storage.kubernetes.io/pvc-volume-mode:
f:spec:
.:
f:source:
.:
f:persistentVolumeClaimName:
f:volumeSnapshotClassName:
Manager: Mozilla
Operation: Update
Time: 2023-06-08T11:16:26Z
API Version: snapshot.storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection":
v:"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection":
Manager: snapshot-controller
Operation: Update
Time: 2023-06-08T11:16:26Z
API Version: snapshot.storage.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:boundVolumeSnapshotContentName:
f:creationTime:
f:readyToUse:
f:restoreSize:
Manager: snapshot-controller
Operation: Update
Subresource: status
Time: 2023-06-08T11:16:28Z
Resource Version: 86077
UID: 0b406f0b-301d-4eaa-87d8-ad86d3161def
Spec:
Source:
Persistent Volume Claim Name: fresh-pvc
Volume Snapshot Class Name: ocs-storagecluster-rbdplugin-snapclass
Status:
Bound Volume Snapshot Content Name: snapcontent-0b406f0b-301d-4eaa-87d8-ad86d3161def
Creation Time: 2023-06-08T11:16:27Z
Ready To Use: true
Restore Size: 10Gi
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CreatingSnapshot 16s snapshot-controller Waiting for a snapshot openshift-storage/fresh-pvc-snapshot to be created by the CSI driver.
Normal SnapshotCreated 14s snapshot-controller Snapshot openshift-storage/fresh-pvc-snapshot was successfully created by the CSI driver.
Normal SnapshotReady 14s snapshot-controller Snapshot openshift-storage/fresh-pvc-snapshot is ready to use.
```
create
```bash
rakshith:ceph-csi$ cat csi.logs | grep "ID: 23"
I0608 10:51:06.758527 1 utils.go:195] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC call: /csi.v1.Controller/CreateSnapshot
I0608 10:51:06.758667 1 utils.go:206] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC request: {"name":"snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/volumesnapshot/name":"fresh-pvc-snapshot","csi.storage.k8s.io/volumesnapshot/namespace":"openshift-storage","csi.storage.k8s.io/volumesnapshotcontent/name":"snapcontent-84f627ad-1d04-4eeb-b631-6a2e6a9196ac"},"secrets":"***stripped***","source_volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}
I0608 10:51:06.760324 1 omap.go:88] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
E0608 10:51:06.772963 1 omap.go:79] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac omap not found (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): rados: ret=-2, No such file or directory
I0608 10:51:06.790147 1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): map[csi.snap.snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac:9db2818a-f217-43b1-b62b-48d3213e77fc])
I0608 10:51:06.792903 1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage])
I0608 10:51:06.792922 1 rbd_journal.go:407] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac generated Volume ID (0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc) and image name (csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc) for request name (snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac)
I0608 10:51:06.792965 1 rbd_util.go:1384] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.220042 1 rbd_util.go:1422] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: clone ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc (features: [layering deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.220077 1 rbd_util.go:1526] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac setting image options on ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:07.299811 1 rbd_util.go:1397] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:07.326191 1 rbd_util.go:1384] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac rbd: snap create ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:08.253443 1 omap.go:158] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da])
I0608 10:51:08.291243 1 rbd_util.go:794] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac clone depth is (1), configured softlimit (4) and hardlimit (8) for ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:08.388990 1 utils.go:212] ID: 23 Req-ID: snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac GRPC response: {"snapshot":{"creation_time":{"nanos":242609242,"seconds":1686221467},"ready_to_use":true,"size_bytes":10737418240,"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc","source_volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}
```
delete
```bash
I0608 10:52:07.309288 1 utils.go:212] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}}
I0608 11:06:04.526310 1 utils.go:195] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC call: /csi.v1.Controller/DeleteSnapshot
I0608 11:06:04.526461 1 utils.go:206] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC request: {"secrets":"***stripped***","snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}
I0608 11:06:04.528007 1 omap.go:88] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 11:06:04.554944 1 controllerserver.go:1440] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc deleting cloned rbd volume csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 11:06:04.554999 1 rbd_util.go:1397] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: snap rm ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 11:06:04.584522 1 rbd_util.go:619] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: delete csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:06:04.618899 1 rbd_util.go:654] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc" with id "37492d2e39da" from trash
I0608 11:06:04.627503 1 rbd_util.go:678] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc" with id "37492d2e39da" to trash
I0608 11:06:04.632759 1 omap.go:125] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snaps.default"): [csi.snap.snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac]
I0608 11:06:04.632819 1 utils.go:212] ID: 26 Req-ID: 0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc GRPC response: {}
```
### restore pvc:
```yaml
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
creationTimestamp: "2023-06-08T10:51:33Z"
finalizers:
- kubernetes.io/pvc-protection
name: restore-pvc
namespace: openshift-storage
resourceVersion: "73315"
uid: 28b5c76e-9313-4df1-b2bb-88229b42e410
spec:
accessModes:
- ReadWriteOnce
dataSource:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: fresh-pvc-snapshot
dataSourceRef:
apiGroup: snapshot.storage.k8s.io
kind: VolumeSnapshot
name: fresh-pvc-snapshot
resources:
requests:
storage: 10Gi
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Filesystem
volumeName: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
```
create
```bash
I0608 10:51:33.979780 1 utils.go:195] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:51:33.979934 1 utils.go:206] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-28b5c76e-9313-4df1-b2bb-88229b42e410","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-28b5c76e-9313-4df1-b2bb-88229b42e410","csi.storage.k8s.io/pvc/name":"restore-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}}}}
I0608 10:51:33.980065 1 rbd_util.go:1279] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 setting disableInUseChecks: false image features: [exclusive-lock object-map layering fast-diff deep-flatten] mounter: rbd
I0608 10:51:33.981456 1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 10:51:34.008805 1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:51:34.020156 1 rbd_util.go:794] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 clone depth is (0), configured softlimit (3) and hardlimit (7) for ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d
I0608 10:51:34.026038 1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-28b5c76e-9313-4df1-b2bb-88229b42e410:d1ca5033-4fea-4c6c-adf9-a44800007426])
I0608 10:51:34.029051 1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imagename:csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 csi.volname:pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 csi.volume.owner:openshift-storage])
I0608 10:51:34.029073 1 rbd_journal.go:487] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 generated Volume ID (0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426) and image name (csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426) for request name (pvc-28b5c76e-9313-4df1-b2bb-88229b42e410)
I0608 10:51:34.029645 1 omap.go:88] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.snap.9db2818a-f217-43b1-b62b-48d3213e77fc"): map[csi.imageid:37492d2e39da csi.imagename:csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc csi.snapname:snapshot-84f627ad-1d04-4eeb-b631-6a2e6a9196ac csi.source:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volume.owner:openshift-storage]
I0608 10:51:34.052791 1 rbd_util.go:1422] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 rbd: clone ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 (features: [deep-flatten layering fast-diff object-map exclusive-lock]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:51:34.052814 1 rbd_util.go:1526] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 setting image options on ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426
I0608 10:51:34.288757 1 controllerserver.go:665] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 create volume ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 from snapshot ocs-storagecluster-cephblockpool/csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc@csi-snap-9db2818a-f217-43b1-b62b-48d3213e77fc
I0608 10:51:34.288781 1 controllerserver.go:730] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 created image ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 backed for request name pvc-28b5c76e-9313-4df1-b2bb-88229b42e410
I0608 10:51:34.326299 1 omap.go:158] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imageid:37492d08a553])
I0608 10:51:34.326455 1 utils.go:212] ID: 24 Req-ID: pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Snapshot":{"snapshot_id":"0001-0011-openshift-storage-0000000000000001-9db2818a-f217-43b1-b62b-48d3213e77fc"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},"volume_id":"0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426"}}
```
delete
```bash
I0608 11:48:17.278911 1 utils.go:195] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:48:17.278993 1 utils.go:206] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426"}
I0608 11:48:17.282247 1 omap.go:88] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.d1ca5033-4fea-4c6c-adf9-a44800007426"): map[csi.imageid:37492d08a553 csi.imagename:csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 csi.volname:pvc-28b5c76e-9313-4df1-b2bb-88229b42e410 csi.volume.owner:openshift-storage]
I0608 11:48:17.401805 1 rbd_util.go:619] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: delete csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:48:17.405719 1 controllerserver.go:1004] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 deleting image csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426
I0608 11:48:17.405732 1 rbd_util.go:619] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: delete csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426 using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:48:17.453235 1 rbd_util.go:654] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426" with id "37492d08a553" from trash
I0608 11:48:17.461364 1 rbd_util.go:678] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-d1ca5033-4fea-4c6c-adf9-a44800007426" with id "37492d08a553" to trash
I0608 11:48:17.466728 1 omap.go:125] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-28b5c76e-9313-4df1-b2bb-88229b42e410]
I0608 11:48:17.466801 1 utils.go:212] ID: 36 Req-ID: 0001-0011-openshift-storage-0000000000000001-d1ca5033-4fea-4c6c-adf9-a44800007426 GRPC response: {}
```
### clone pvc:
```yaml
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
creationTimestamp: "2023-06-08T10:52:05Z"
finalizers:
- kubernetes.io/pvc-protection
name: clone-pvc
namespace: openshift-storage
resourceVersion: "73600"
uid: 75c0fbe3-271b-4486-a22c-0110b13234da
spec:
accessModes:
- ReadWriteOnce
dataSource:
apiGroup: ""
kind: PersistentVolumeClaim
name: fresh-pvc
dataSourceRef:
apiGroup: ""
kind: PersistentVolumeClaim
name: fresh-pvc
resources:
requests:
storage: 10Gi
storageClassName: ocs-storagecluster-ceph-rbd
volumeMode: Filesystem
volumeName: pvc-75c0fbe3-271b-4486-a22c-0110b13234da
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
phase: Bound
```
create
```bash
I0608 10:52:05.952793 1 utils.go:195] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC call: /csi.v1.Controller/CreateVolume
I0608 10:52:05.952995 1 utils.go:206] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC request: {"capacity_range":{"required_bytes":10737418240},"name":"pvc-75c0fbe3-271b-4486-a22c-0110b13234da","parameters":{"clusterID":"openshift-storage","csi.storage.k8s.io/pv/name":"pvc-75c0fbe3-271b-4486-a22c-0110b13234da","csi.storage.k8s.io/pvc/name":"clone-pvc","csi.storage.k8s.io/pvc/namespace":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","pool":"ocs-storagecluster-cephblockpool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}],"volume_content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}}}
I0608 10:52:05.953138 1 rbd_util.go:1279] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting disableInUseChecks: false image features: [object-map exclusive-lock deep-flatten layering fast-diff] mounter: rbd
I0608 10:52:05.954561 1 omap.go:88] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"): map[csi.imageid:374949d1a256 csi.imagename:csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d csi.volname:pvc-e270178a-ebb0-48ec-82c9-345994b96a62 csi.volume.owner:openshift-storage]
I0608 10:52:05.968432 1 omap.go:88] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[]
I0608 10:52:06.001089 1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): map[csi.volume.pvc-75c0fbe3-271b-4486-a22c-0110b13234da:cae7f13f-3e63-46ef-a4b6-8809b3b5a87b])
I0608 10:52:06.003841 1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage])
I0608 10:52:06.003861 1 rbd_journal.go:487] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da generated Volume ID (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) and image name (csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) for request name (pvc-75c0fbe3-271b-4486-a22c-0110b13234da)
I0608 10:52:06.003922 1 rbd_util.go:1384] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap create ocs-storagecluster-cephblockpool/@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon
I0608 10:52:06.157382 1 rbd_util.go:1422] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: clone ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp (features: [layering deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:52:06.157409 1 rbd_util.go:1526] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting image options on ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp
I0608 10:52:06.215467 1 rbd_util.go:1397] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon
I0608 10:52:06.306500 1 rbd_util.go:1384] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap create ocs-storagecluster-cephblockpool/@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon
I0608 10:52:07.164524 1 rbd_util.go:1422] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: clone ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b (features: [exclusive-lock object-map layering fast-diff deep-flatten]) using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 10:52:07.164554 1 rbd_util.go:1526] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da setting image options on ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 10:52:07.254173 1 rbd_util.go:1397] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da rbd: snap rm ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp@csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon
I0608 10:52:07.309134 1 omap.go:158] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da set omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858])
I0608 10:52:07.309288 1 utils.go:212] ID: 25 Req-ID: pvc-75c0fbe3-271b-4486-a22c-0110b13234da GRPC response: {"volume":{"capacity_bytes":10737418240,"content_source":{"Type":{"Volume":{"volume_id":"0001-0011-openshift-storage-0000000000000001-f3e7cae1-0d25-470b-9d7e-b8eb4a1ffd3d"}}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool"},
```
delete
```bash
I0608 11:49:33.777904 1 utils.go:195] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Controller/DeleteVolume
I0608 11:49:33.777979 1 utils.go:206] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:49:33.779734 1 omap.go:88] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:49:33.870623 1 rbd_util.go:619] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: delete csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:49:33.913873 1 rbd_util.go:654] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp" with id "3749b014585d" from trash
I0608 11:49:33.920917 1 rbd_util.go:678] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b-temp" with id "3749b014585d" to trash
I0608 11:49:33.920937 1 controllerserver.go:1004] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b deleting image csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:49:33.920945 1 rbd_util.go:619] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: delete csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b using mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789, pool ocs-storagecluster-cephblockpool
I0608 11:49:33.964784 1 rbd_util.go:654] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: adding task to remove image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b" with id "374935432858" from trash
I0608 11:49:33.971440 1 rbd_util.go:678] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully added task to move image "ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b" with id "374935432858" to trash
I0608 11:49:33.976466 1 omap.go:125] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b removed omap keys (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volumes.default"): [csi.volume.pvc-75c0fbe3-271b-4486-a22c-0110b13234da]
I0608 11:49:33.976539 1 utils.go:212] ID: 37 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
### Expansion:
```bash
rakshith:ceph-csi$ k describe pvc clone-pvc
Name: clone-pvc
Namespace: openshift-storage
StorageClass: ocs-storagecluster-ceph-rbd
Status: Bound
Volume: pvc-75c0fbe3-271b-4486-a22c-0110b13234da
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: openshift-storage.rbd.csi.ceph.com
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 15Gi
Access Modes: RWO
VolumeMode: Filesystem
DataSource:
APIGroup:
Kind: PersistentVolumeClaim
Name: fresh-pvc
Used By: csirbd-demo-pod
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Provisioning 28m openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13 External provisioner is provisioning volume for claim "openshift-storage/clone-pvc"
Normal ExternalProvisioning 28m (x2 over 28m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.rbd.csi.ceph.com" or manually created by system administrator
Normal ProvisioningSucceeded 28m openshift-storage.rbd.csi.ceph.com_csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd_04e2e6bf-2ed7-40f2-9bf6-39a5c20fde13 Successfully provisioned volume pvc-75c0fbe3-271b-4486-a22c-0110b13234da
Warning ExternalExpanding 118s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC.
Normal Resizing 118s external-resizer openshift-storage.rbd.csi.ceph.com External resizer is resizing volume pvc-75c0fbe3-271b-4486-a22c-0110b13234da
Normal FileSystemResizeRequired 118s external-resizer openshift-storage.rbd.csi.ceph.com Require file system resize of volume on node
Normal FileSystemResizeSuccessful 60s (x2 over 60s) kubelet MountVolume.NodeExpandVolume succeeded for volume "pvc-75c0fbe3-271b-4486-a22c-0110b13234da" ip-10-0-221-171.us-west-2.compute.internal
```
controller logs
```bash
0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Controller/ControllerExpandVolume
I0608 11:18:55.182571 1 utils.go:206] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"capacity_range":{"required_bytes":16106127360},"secrets":"***stripped***","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:18:55.184281 1 omap.go:88] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:18:55.216929 1 controllerserver.go:1576] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd volume ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b size is 10737418240,resizing to 16106127360
I0608 11:18:55.267338 1 utils.go:212] ID: 32 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {"capacity_bytes":16106127360,"node_expansion_required":true}
```
node logs
```bash
I0608 11:19:53.222832 116821 utils.go:195] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeExpandVolume
I0608 11:19:53.222981 116821 utils.go:206] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"capacity_range":{"required_bytes":16106127360},"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","volume_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount"}
I0608 11:19:53.240196 116821 cephcmds.go:105] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [device list --format=json --device-type krbd]
I0608 11:19:53.240267 116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.241607 116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.241631 116821 resizefs_linux.go:56] ResizeFS.Resize - Expanding mounted volume /dev/rbd0
I0608 11:19:53.243818 116821 resizefs_linux.go:71] Device /dev/rbd0 resized successfully
I0608 11:19:53.243874 116821 utils.go:212] ID: 16 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
## csi pods + example:
```
rakshith:ceph-csi$ k get po -o wide | grep csi
csi-addons-controller-manager-6c57bc4f4f-hscb6 2/2 Running 0 21m 10.128.2.22 ip-10-0-156-223.us-west-2.compute.internal <none> <none>
csi-cephfsplugin-dkcnl 2/2 Running 0 17m 10.0.167.49 ip-10-0-167-49.us-west-2.compute.internal <none> <none>
csi-cephfsplugin-mvfwp 2/2 Running 0 17m 10.0.221.171 ip-10-0-221-171.us-west-2.compute.internal <none> <none>
csi-cephfsplugin-p5glz 2/2 Running 0 17m 10.0.156.223 ip-10-0-156-223.us-west-2.compute.internal <none> <none>
csi-cephfsplugin-provisioner-7b9dfb49b9-ssln5 5/5 Running 0 17m 10.129.2.19 ip-10-0-167-49.us-west-2.compute.internal <none> <none>
csi-cephfsplugin-provisioner-7b9dfb49b9-xvw9z 5/5 Running 0 17m 10.131.0.33 ip-10-0-221-171.us-west-2.compute.internal <none> <none>
csi-rbdplugin-674kd 3/3 Running 0 17m 10.0.156.223 ip-10-0-156-223.us-west-2.compute.internal <none> <none>
csi-rbdplugin-provisioner-5d8b9d46f9-jzjhw 6/6 Running 0 17m 10.128.2.24 ip-10-0-156-223.us-west-2.compute.internal <none> <none>
csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd 6/6 Running 0 17m 10.131.0.32 ip-10-0-221-171.us-west-2.compute.internal <none> <none>
csi-rbdplugin-rpjzh 3/3 Running 0 17m 10.0.221.171 ip-10-0-221-171.us-west-2.compute.internal <none> <none>
csi-rbdplugin-xwd5v 3/3 Running 0 17m 10.0.167.49 ip-10-0-167-49.us-west-2.compute.internal <none> <none>
csirbd-demo-pod 1/1 Running 0 119s 10.128.2.33 ip-10-0-156-223.us-west-2.compute.internal <none> <none>
```
### NodeStageVolume:
```bash
I0608 11:19:52.653283 116821 utils.go:195] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeStageVolume
I0608 11:19:52.653450 116821 utils.go:206] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:19:52.653642 116821 rbd_util.go:1279] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b setting disableInUseChecks: false image features: [layering fast-diff exclusive-lock object-map deep-flatten] mounter: rbd
I0608 11:19:52.669071 116821 omap.go:88] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:19:52.709565 116821 rbd_util.go:346] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b checking for ImageFeatures: [layering fast-diff operations exclusive-lock object-map deep-flatten]
I0608 11:19:52.727006 116821 cephcmds.go:105] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [device list --format=json --device-type krbd]
I0608 11:19:52.759598 116821 rbd_attach.go:420] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: map mon 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789
I0608 11:19:52.863390 116821 cephcmds.go:105] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [--id csi-rbd-node -m 172.30.13.178:6789,172.30.185.10:6789,172.30.139.27:6789 --keyfile=***stripped*** map ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b --device-type krbd --options noudev]
I0608 11:19:52.863431 116821 nodeserver.go:414] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd image: ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b was successfully mapped at /dev/rbd0
I0608 11:19:52.863503 116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:52.867986 116821 mount_linux.go:566] Output: ""
I0608 11:19:53.126834 116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.142597 116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.142614 116821 mount_linux.go:452] Checking for issues with fsck on disk: /dev/rbd0
I0608 11:19:53.158110 116821 mount_linux.go:553] Attempting to mount disk /dev/rbd0 in ext4 format at /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:19:53.158136 116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o _netdev,defaults /dev/rbd0 /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:19:53.199021 116821 mount_linux.go:563] Attempting to determine if disk "/dev/rbd0" is formatted using blkid with args: ([-p -s TYPE -s PTTYPE -o export /dev/rbd0])
I0608 11:19:53.203315 116821 mount_linux.go:566] Output: "DEVNAME=/dev/rbd0\nTYPE=ext4\n"
I0608 11:19:53.203329 116821 resizefs_linux.go:124] ResizeFs.needResize - checking mounted volume /dev/rbd0
I0608 11:19:53.204851 116821 resizefs_linux.go:128] Ext size: filesystem size=16106127360, block size=4096
I0608 11:19:53.204863 116821 resizefs_linux.go:143] Volume /dev/rbd0: device size=16106127360, filesystem size=16106127360, block size=4096
I0608 11:19:53.204885 116821 nodeserver.go:374] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully mounted volume 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b to stagingTargetPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:19:53.204986 116821 utils.go:212] ID: 9 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
### NodePublishVolume:
```bash
I0608 11:19:53.211124 116821 utils.go:195] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodePublishVolume
I0608 11:19:53.211277 116821 utils.go:206] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","target_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":7}},"volume_context":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:19:53.211353 116821 nodeserver.go:802] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b target /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
isBlock false
fstype ext4
stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
readonly false
mountflags [bind _netdev]
I0608 11:19:53.211378 116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o bind,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount)
I0608 11:19:53.213451 116821 mount_linux.go:219] Mounting cmd (mount) with arguments (-t ext4 -o bind,remount,_netdev /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount)
I0608 11:19:53.214580 116821 nodeserver.go:700] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully mounted stagingPath /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b to targetPath /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:19:53.214608 116821 utils.go:212] ID: 13 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
### NodeUnpublishVolume:
```bash
I0608 11:25:27.079341 116821 utils.go:195] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0608 11:25:27.079441 116821 utils.go:206] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"target_path":"/var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:25:27.079504 116821 mount_linux.go:361] Unmounting /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:25:27.083138 116821 nodeserver.go:888] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b rbd: successfully unbound volume 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b from /var/lib/kubelet/pods/e13aafbf-023c-4fda-ac15-e8e80fa1f2c6/volumes/kubernetes.io~csi/pvc-75c0fbe3-271b-4486-a22c-0110b13234da/mount
I0608 11:25:27.083183 116821 utils.go:212] ID: 28 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
### NodeUnstageVolume:
```bash
I0608 11:25:27.187622 116821 utils.go:195] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC call: /csi.v1.Node/NodeUnstageVolume
I0608 11:25:27.187741 116821 utils.go:206] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC request: {"staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:25:27.187837 116821 mount_linux.go:361] Unmounting /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b
I0608 11:25:27.210704 116821 nodeserver.go:944] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b successfully unmounted volume (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b) from staging path (/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:25:27.260510 116821 cephcmds.go:105] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b command succeeded: rbd [unmap ocs-storagecluster-cephblockpool/csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b --device-type krbd --options noudev]
I0608 11:25:27.260535 116821 nodeserver.go:1002] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b successfully unmapped volume (0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b)
I0608 11:25:27.260636 116821 utils.go:212] ID: 30 Req-ID: 0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b GRPC response: {}
```
----
# Reclaimspace Operation
### workflow
```bash
rakshith:kubernetes-csi-addons$ kubectl annotate pvc clone-pvc "reclaimspace.csiaddons.openshift.io/schedule=*/2 * * * *"
persistentvolumeclaim/clone-pvc annotated
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io
NAME SCHEDULE SUSPEND ACTIVE LASTSCHEDULE AGE
clone-pvc-1686223828 */2 * * * * 12s
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io -w
NAME NAMESPACE AGE RETRIES RESULT
clone-pvc-1686223828-1686223920 openshift-storage 0s
clone-pvc-1686223828-1686223920 openshift-storage 0s Succeeded
rakshith:kubernetes-csi-addons$ kubectl annotate pvc clone-pvc "reclaimspace.csiaddons.openshift.io/schedule-"
persistentvolumeclaim/clone-pvc annotated
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io
No resources found in openshift-storage namespace.
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io
No resources found in openshift-storage namespace.
```
### Annotation
```bash
2023-06-08T11:30:28.491Z INFO Successfully created reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"clone-pvc","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc", "reconcileID": "c887e64e-ab3c-4e60-bf2f-7df8649a8731", "Schedule": "*/2 * * * *", "ReclaimSpaceCronJobName": "clone-pvc-1686223828"}
```
### ReclaimSpaceCronJob
```bash
rakshith:kubernetes-csi-addons$ k get reclaimspacecronjobs.csiaddons.openshift.io clone-pvc-1686223828 -o yaml
apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceCronJob
metadata:
creationTimestamp: "2023-06-08T11:30:28Z"
generation: 1
name: clone-pvc-1686223828
namespace: openshift-storage
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
controller: true
kind: PersistentVolumeClaim
name: clone-pvc
uid: 75c0fbe3-271b-4486-a22c-0110b13234da
resourceVersion: "94922"
uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
spec:
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 1
jobTemplate:
metadata: {}
spec:
backOffLimit: 6
retryDeadlineSeconds: 600
target:
persistentVolumeClaim: clone-pvc
schedule: '*/2 * * * *'
successfulJobsHistoryLimit: 3
status:
lastScheduleTime: "2023-06-08T11:34:00Z"
lastSuccessfulTime: "2023-06-08T11:34:00Z"
```
```bash
2023-06-08T11:34:00.008Z INFO Successfully created reclaimSpaceJob for reclaimSpaceCronJob run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"clone-pvc-1686223828","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828", "reconcileID": "bfe9b421-c797-45b5-b82e-301ddb007f36", "now": "2023-06-08T11:34:00.004Z", "nextRun": "2023-06-08T11:36:00.000Z", "currentRun": "2023-06-08T11:34:00.000Z", "reclaimSpacejob": "clone-pvc-1686223828-1686224040"}
```
### ReclaimSpaceJob
```bash
rakshith:kubernetes-csi-addons$ k get reclaimspacejobs.csiaddons.openshift.io -o yaml
apiVersion: v1
items:
- apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceJob
metadata:
annotations:
csiaddons.openshift.io/scheduled-at: "2023-06-08T11:32:00Z"
creationTimestamp: "2023-06-08T11:32:00Z"
generation: 1
name: clone-pvc-1686223828-1686223920
namespace: openshift-storage
ownerReferences:
- apiVersion: csiaddons.openshift.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: ReclaimSpaceCronJob
name: clone-pvc-1686223828
uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
resourceVersion: "93904"
uid: 5f770abf-af1d-499d-9569-98ff6414fbf9
spec:
backOffLimit: 6
retryDeadlineSeconds: 600
target:
persistentVolumeClaim: clone-pvc
status:
completionTime: "2023-06-08T11:32:00Z"
message: Reclaim Space operation successfully completed.
result: Succeeded
startTime: "2023-06-08T11:32:00Z"
- apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceJob
metadata:
annotations:
csiaddons.openshift.io/scheduled-at: "2023-06-08T11:34:00Z"
creationTimestamp: "2023-06-08T11:34:00Z"
generation: 1
name: clone-pvc-1686223828-1686224040
namespace: openshift-storage
ownerReferences:
- apiVersion: csiaddons.openshift.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: ReclaimSpaceCronJob
name: clone-pvc-1686223828
uid: fd5d2e8d-41bd-436e-bd17-e76e109cc000
resourceVersion: "94921"
uid: a34b147c-fe12-4415-88a1-0adc389fcb76
spec:
backOffLimit: 6
retryDeadlineSeconds: 600
target:
persistentVolumeClaim: clone-pvc
status:
completionTime: "2023-06-08T11:34:00Z"
message: Reclaim Space operation successfully completed.
result: Succeeded
startTime: "2023-06-08T11:34:00Z"
kind: List
metadata:
resourceVersion: ""
```
```bash
2023-06-08T11:32:00.472Z INFO No upcoming scheduled times, requeue with delay till next run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"clone-pvc-1686223828","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828", "reconcileID": "ed5e6231-b487-489e-99a4-b9f4b7e7f278", "now": "2023-06-08T11:32:00.472Z", "nextRun": "2023-06-08T11:34:00.000Z"}
```
#### ReclaimSpace from CSI-Addons controller
```bash
2023-06-08T11:32:00.212Z INFO Making node reclaim space request {"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh"}
2023-06-08T11:32:00.254Z INFO Making controller reclaim space request {"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh", "controllerClient": "openshift-storage/csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd"}
2023-06-08T11:32:00.458Z INFO Successfully completed reclaim space operation {"controller": "reclaimspacejob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceJob", "ReclaimSpaceJob": {"name":"clone-pvc-1686223828-1686223920","namespace":"openshift-storage"}, "namespace": "openshift-storage", "name": "clone-pvc-1686223828-1686223920", "reconcileID": "c71e7add-652a-48f4-828f-05c5b3c039c4", "PVCName": "clone-pvc", "PVCNamespace": "openshift-storage", "PVName": "pvc-75c0fbe3-271b-4486-a22c-0110b13234da", "NodeID": "ip-10-0-221-171.us-west-2.compute.internal", "nodeClient": "openshift-storage/csi-rbdplugin-rpjzh", "controllerClient": "openshift-storage/csi-rbdplugin-provisioner-5d8b9d46f9-lgcpd"}
```
#### ControllerReclaimSpace
```bash
I0608 11:36:00.033207 1 utils.go:195] ID: 21 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0608 11:36:00.033301 1 utils.go:206] ID: 21 GRPC request: {"parameters":{"clusterID":"openshift-storage","imageFeatures":"layering,deep-flatten,exclusive-lock,object-map,fast-diff","imageFormat":"2","imageName":"csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b","journalPool":"ocs-storagecluster-cephblockpool","pool":"ocs-storagecluster-cephblockpool","storage.kubernetes.io/csiProvisionerIdentity":"1686220647580-8081-openshift-storage.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:36:00.036733 1 omap.go:88] ID: 21 got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"): map[csi.imageid:374935432858 csi.imagename:csi-vol-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b csi.volname:pvc-75c0fbe3-271b-4486-a22c-0110b13234da csi.volume.owner:openshift-storage]
I0608 11:36:00.180510 1 utils.go:212] ID: 21 GRPC response: {}
```
#### NodeReclaimSpace
```bash
I0608 11:36:00.022006 116821 utils.go:195] ID: 53 GRPC call: /reclaimspace.ReclaimSpaceNode/NodeReclaimSpace
I0608 11:36:00.022109 116821 utils.go:206] ID: 53 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b"}
I0608 11:36:00.023829 116821 cephcmds.go:105] ID: 53 command succeeded: fstrim [/var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.rbd.csi.ceph.com/5ba9f3dc4e76a3ea6dda002302fa1652e2ec728f8c9c93a2611170a6c770561b/globalmount/0001-0011-openshift-storage-0000000000000001-cae7f13f-3e63-46ef-a4b6-8809b3b5a87b]
I0608 11:36:00.023859 116821 utils.go:212] ID: 53 GRPC response: {}
```
Sign in with Wallet
Connect another wallet