# Design Doc for Auto ReclaimSpace
## Current Solution
Currently, users can annotate the PVCs or the Namespace of the PVCs for which ReclaimSpaceCronJob has to be created.
## Problem
- Annotating hundreds of PVCs is not a good solution. Having a centralized control would be a better option.
- Annotating Namespace is not a good practice as Namespace are used for logical organization of resources, not configuration.
- deleting/modifying annotation on Namespace will require admin to do same operation on PVCs within that Namespace.
## Approach
### 1. CSIDriver annotation
This approach involves annotating the CSIDriver object and PVCs provisioned by these drivers will be annotated for auto ReclaimSpace.
:+1: simple configuration, centralized control and consistent reclaim space schedules for all PVCs provisioned by same provisioner.
:-1: this does not provide fine-grained control over a group of PVCs provisioned by a particular SC.
### 2. StorageClass annotation
This approach involves annotating the StorageClass, whereby PVCs associated with this StorageClass will be annotated for ReclaimSpace, and a respective ReclaimSpace CronJob will be created.
It would still require admins to modify/delete annotations for each PVCs. This can be solved by watching the SC objects and reconciling the PVCs if annotations are modified/deleted on SC.
:::danger
In current implementation of PVC controller, SC annotations are superseded by PVC annotations. So, add/modify/delete annotations on SC will have no impact on the PVCs RS Jobs.
:::
One way of modifying the annotations on PVC would be,
1. add the `/remove` annotation on the SC, which will remove all `reclaimspace.` annotations from associated PVCs
2. remove the `/remove` annotation from the SC and add the updated `/schedule` annotation, which will then add the updated annotation to PVCs of this SC.
:+1: consistent RS for PVCs of same SC, control schedule at SC level instead at provisioner level.
:-1: admins still need to manager annotations for each SC
### 3. ReclaimSpace ConfigMap
This approach involves having a new CM for ReclaimSpace configuration.
For example,
```
apiVersion: v1
kind: ConfigMap
metadata:
name: csi-addons-reclaim-config
namespace: csi-addons-system
data:
config.json: |
{
"timeout": "10m",
"auto-reclaim-storageclass": {
"rbd-sc": "daily",
"rbd1-sc": "weekly",
...
},
"reclaim-space-enable": "true"
}
```
- `auto-reclaim-storageclass` option to provide maps of `<sc-name>:<schedule>` entries for auto reclaimspace
- `reclaim-space-enable` can help admins to enable/disable the RS incase needed instead of adding/deleting the annotating from multiple resource objects.
:+1: easy to manager and update configuration from single place.
:-1: controller need to watch for CM and handle the changes.
<hr>
# Handle existing PVCs
## Problem
## Solution
## Testing
```yaml
$ k get sc rook-ceph-block -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
creationTimestamp: "2024-07-08T06:48:23Z"
name: rook-ceph-block
resourceVersion: "1879671"
uid: 8e8e36e4-f0fb-460c-b352-3efd8799479e
parameters:
clusterID: rook-ceph
csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
encrypted: "true"
encryptionKMSID: azure-test
imageFeatures: layering
imageFormat: "2"
pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
```
```yaml
$ k get pvc rbd-pvc -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
creationTimestamp: "2024-07-11T07:09:18Z"
finalizers:
- kubernetes.io/pvc-protection
name: rbd-pvc
namespace: test
resourceVersion: "2311226"
uid: 7388ae15-3ee4-4051-90d5-0661181b36d6
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: rook-ceph-block
volumeMode: Block
volumeName: pvc-7388ae15-3ee4-4051-90d5-0661181b36d6
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
```
```yaml
$ k annotate sc rook-ceph-block "reclaimspace.csiaddons.openshift.io/schedule=@daily"
storageclass.storage.k8s.io/rook-ceph-block annotated
$ k get pvc rbd-pvc -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
reclaimspace.csiaddons.openshift.io/cronjob: rbd-pvc-1720682724
reclaimspace.csiaddons.openshift.io/schedule: '@daily'
volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
creationTimestamp: "2024-07-11T07:09:18Z"
finalizers:
- kubernetes.io/pvc-protection
name: rbd-pvc
namespace: test
resourceVersion: "2312547"
uid: 7388ae15-3ee4-4051-90d5-0661181b36d6
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: rook-ceph-block
volumeMode: Block
volumeName: pvc-7388ae15-3ee4-4051-90d5-0661181b36d6
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
```
```
2024-07-11T07:11:04.520Z INFO Annotation not set, exiting reconcile {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "8521eaa0-3fc1-4541-8214-c66d34003ecf"}
2024-07-11T07:25:24.966Z INFO Adding annotation {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "477395f4-160d-4d0b-b67d-033d69aa20a9", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Annotation": "{\"metadata\":{\"annotations\":{\"reclaimspace.csiaddons.openshift.io/cronjob\":\"rbd-pvc-1720682724\",\"reclaimspace.csiaddons.openshift.io/schedule\":\"@daily\"}}}"}
2024-07-11T07:25:24.996Z INFO KubeAPIWarningLogger unknown field "spec.jobTemplate.metadata.creationTimestamp"
2024-07-11T07:25:24.997Z INFO Successfully created reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "477395f4-160d-4d0b-b67d-033d69aa20a9", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724"}
2024-07-11T07:25:25.017Z INFO No upcoming scheduled times, requeue with delay till next run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-1720682724","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-1720682724", "reconcileID": "7761b968-0fda-4e2b-8abd-4bde8f3c3854", "now": "2024-07-11T07:25:25.017Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:25:25.019Z ERROR Failed to update reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "4ca1a3ee-7162-434e-9694-7295c5e3b939", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Schedule": "@daily", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-1720682724\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons.(*PersistentVolumeClaimReconciler).Reconcile
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons/persistentvolumeclaim_controller.go:173
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222
2024-07-11T07:25:25.021Z ERROR Reconciler error {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "4ca1a3ee-7162-434e-9694-7295c5e3b939", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-1720682724\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222
2024-07-11T07:25:25.030Z INFO No upcoming scheduled times, requeue with delay till next run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-1720682724","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-1720682724", "reconcileID": "dd49f11d-a0d5-4349-9417-f0185933a84b", "now": "2024-07-11T07:25:25.030Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:25:25.037Z INFO Successfully updated reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "654882a0-7e69-4aa3-84b2-d5ab3668466e", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Schedule": "@daily"}
```
```yaml
$ k create -f pvc.yaml
persistentvolumeclaim/rbd-pvc-new created
$ k get pvc rbd-pvc-new -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
reclaimspace.csiaddons.openshift.io/cronjob: rbd-pvc-new-1720683030
reclaimspace.csiaddons.openshift.io/schedule: '@daily'
volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
creationTimestamp: "2024-07-11T07:30:24Z"
finalizers:
- kubernetes.io/pvc-protection
name: rbd-pvc-new
namespace: test
resourceVersion: "2313046"
uid: decb16c6-44be-45d2-ba75-7b4ab029be9d
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: rook-ceph-block
volumeMode: Block
volumeName: pvc-decb16c6-44be-45d2-ba75-7b4ab029be9d
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
phase: Bound
```
```
2024-07-11T07:30:27.566Z INFO PVC is not in bound state {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "1d67ad16-003a-434a-a297-82877fa44f8d", "PVCPhase": "Pending"}
2024-07-11T07:30:30.128Z INFO Adding annotation {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "43bae20b-8561-4e55-8769-9f8708fead39", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Annotation": "{\"metadata\":{\"annotations\":{\"reclaimspace.csiaddons.openshift.io/cronjob\":\"rbd-pvc-new-1720683030\",\"reclaimspace.csiaddons.openshift.io/schedule\":\"@daily\"}}}"}
2024-07-11T07:30:30.150Z INFO Successfully created reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "43bae20b-8561-4e55-8769-9f8708fead39", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030"}
2024-07-11T07:30:30.159Z INFO Successfully updated reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "3fd51186-854a-445e-8a8c-88667e33e479", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily"}
2024-07-11T07:30:30.165Z INFO No upcoming scheduled times, requeue with delay till next run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-new-1720683030","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new-1720683030", "reconcileID": "e9c95d30-2f78-4625-a3b6-bf4c3c861c76", "now": "2024-07-11T07:30:30.165Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:30:30.170Z ERROR Failed to update reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "0b54e3b8-1c01-49dd-b571-ee7935c8c3b6", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-new-1720683030\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons.(*PersistentVolumeClaimReconciler).Reconcile
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/controllers/csiaddons/persistentvolumeclaim_controller.go:173
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222
2024-07-11T07:30:30.170Z ERROR Reconciler error {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "0b54e3b8-1c01-49dd-b571-ee7935c8c3b6", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-new-1720683030\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/workspace/go/src/github.com/csi-addons/kubernetes-csi-addons/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222
2024-07-11T07:30:30.176Z INFO No upcoming scheduled times, requeue with delay till next run {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-new-1720683030","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new-1720683030", "reconcileID": "d269ebe5-1b02-4aba-b9c8-27d6655f3fc3", "now": "2024-07-11T07:30:30.175Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:30:30.184Z INFO Successfully updated reclaimSpaceCronJob {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "480a067e-56d5-4d9f-94db-7e8d6ea94706", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily"}
```
# Handle annotation modification on StorageClass.
## Problem
deleting/modifying annotation on StorageClas will require admin to do same operation on PVCs of that StorageClass.
## Solution
Watch for the ReclaimSpace annotations change on StorageClass and Enqueue all PVCs associated with this StorageClass for Reconcile.
### challenges
- We don't know which PVCs to be considered for annotation update. (since few PVCs will already be having annotations set and those might not be from its StorageClass, they might be added by user or namespace annotation.)
:::success
We could add one more annotation something like,
- reclaimspace.csiaddons.csi.com/owner:controller or
- reclaimspace.csiaddons.csi.com/added-by-controller: true
differentiating who added the annotations?
:::
### Scenarios
1. User adds the annotations to
- PVC - reconcile and create CronJob
- StorageClass - reconcile, add annotations to PVCs (schedule, added-by-controller) and create CronJob
2. User updates the annotations on
- PVC -
- If `added-by-controller` annotation is not set, update annotations
- else, remove `added-by-controller` annotation and update `schedule` annotation with user provided
- StorageClass - Reconcile all PVCs with annotations `added-by-controller` and without schedule annotation, update annotations on PVCs
3. User removes the annotations from
- PVC -
- If `added-by-controller` annotation not set, remove and exit
- else, remove `added-by-controller` and `schedule` annotations
- StorageClass - Reconcile all PVCs with annotations `added-by-controller` and remove annotations from PVCs
---
Sign in with Wallet
Connect another wallet