## check for https://issues.redhat.com/browse/OADP-3306

Question: is the vsc `deletionPolicy: Retain` required as specified in the doc or does Velero correctly label volumesnapshots for backup and restore?

* pretty sure sseago patched this to work regardless of deletionPolicy in vsc.
* can't find PR, github issue or related jira issue.

My conclusion atm:
* Regardless of the deletionPolicy: [retain,delete] in the vsc. Velero is doing the right thing at backup and restore.
* Documentation stating that the volumeSnapShotClass deletionPolicy must be retain is NOT required.

# OADP 1.2.x

testing w/ OADP-1.2.3

## Attempt #1 also showing the VolumeSnapshotContents
* dpa

apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  creationTimestamp: "2024-02-08T15:57:52Z"
  generation: 1
  name: dpa-sample
  namespace: openshift-adp
  resourceVersion: "14268662"
  uid: 7ced43f5-b728-457e-8c5d-ba870f69200e
spec:
  backupLocations:
  - velero:
      config:
        profile: default
        region: us-west-2
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: cvpbucketuswest2
        prefix: velero
      provider: aws
  configuration:
    restic:
      enable: false
    velero:
      defaultPlugins:
      - openshift
      - aws
      - csi
  features:
    dataMover:
      credentialName: restic-secret
      enable: false
  snapshotLocations:
  - velero:
      config:
        profile: default
        region: us-west-2
      provider: aws
status:
  conditions:
  - lastTransitionTime: "2024-02-08T15:57:52Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

* VolumeSnapshotClass

oc get volumesnapshotclass csi-aws-vsc-velero -o yaml
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: ebs.csi.aws.com
kind: VolumeSnapshotClass
metadata:
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-02-08T18:37:18Z"
  generation: 3
  labels:
    velero.io/csi-volumesnapshot-class: "true"
  name: csi-aws-vsc-velero
  resourceVersion: "14348947"
  uid: 446d6aa0-f711-4da5-9efc-aa9b281a480a

## clean system

whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get volumesnapshot -A
No resources found
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get volumesnapshotcontents -A
No resources found

## volumesnapshotclass

whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator/tests/e2e/sample-applications/mysql-persistent$ oc get volumesnapshotclass csi-aws-vsc-velero -o yaml
apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: ebs.csi.aws.com
kind: VolumeSnapshotClass
metadata:
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2024-02-08T18:37:18Z"
  generation: 3
  labels:
    velero.io/csi-volumesnapshot-class: "true"
  name: csi-aws-vsc-velero
  resourceVersion: "14348947"
  uid: 446d6aa0-f711-4da5-9efc-aa9b281a480a

## backup cr

apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-persistent-csi-vsc-retain-policy-delete-2
  namespace: openshift-adp
spec:
  includedNamespaces:
  - mysql-persistent
  storageLocation: dpa-sample-1
  ttl: 720h0m0s

## execution

oc create -f mysql-backup.yaml 
backup.velero.io/mysql-persistent-csi-vsc-retain-policy-delete-2 created

## backup details

Name:         mysql-persistent-csi-vsc-retain-policy-delete-2
Namespace:    openshift-adp
Labels:       velero.io/storage-location=dpa-sample-1
Annotations:  velero.io/source-cluster-k8s-gitversion=v1.27.8+4fab27b
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=27

Phase:  Completed


Namespaces:
  Included:  mysql-persistent
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  dpa-sample-1

Velero-Native Snapshot PVs:  auto

TTL:  720h0m0s

CSISnapshotTimeout:    10m0s
ItemOperationTimeout:  1h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2024-02-08 19:48:23 +0000 UTC
Completed:  2024-02-08 19:48:33 +0000 UTC

Expiration:  2024-03-09 19:48:23 +0000 UTC

Total items to be backed up:  96
Items backed up:              96

Resource List:
  apiextensions.k8s.io/v1/CustomResourceDefinition:
    - clusterserviceversions.operators.coreos.com
  apps.openshift.io/v1/DeploymentConfig:
    - mysql-persistent/todolist
  apps/v1/Deployment:
    - mysql-persistent/mysql
  apps/v1/ReplicaSet:
    - mysql-persistent/mysql-5d4bf749b
  authorization.openshift.io/v1/RoleBinding:
    - mysql-persistent/system:deployers
    - mysql-persistent/system:image-builders
    - mysql-persistent/system:image-pullers
  discovery.k8s.io/v1/EndpointSlice:
    - mysql-persistent/mysql-cgr7v
    - mysql-persistent/todolist-pbg79
  operators.coreos.com/v1alpha1/ClusterServiceVersion:
    - mysql-persistent/volsync-product.v0.7.4-0.1698026108.p
  rbac.authorization.k8s.io/v1/RoleBinding:
    - mysql-persistent/system:deployers
    - mysql-persistent/system:image-builders
    - mysql-persistent/system:image-pullers
  route.openshift.io/v1/Route:
    - mysql-persistent/todolist-route
  security.openshift.io/v1/SecurityContextConstraints:
    - mysql-persistent-scc
  snapshot.storage.k8s.io/v1/VolumeSnapshot:
    - mysql-persistent/velero-mysql-9wfk8
  snapshot.storage.k8s.io/v1/VolumeSnapshotClass:
    - csi-aws-vsc-velero
  snapshot.storage.k8s.io/v1/VolumeSnapshotContent:
    - snapcontent-031323f7-1cce-4a10-afa1-483b2d558ca0
  v1/ConfigMap:
    - mysql-persistent/kube-root-ca.crt
    - mysql-persistent/openshift-service-ca.crt
  v1/Endpoints:
    - mysql-persistent/mysql
    - mysql-persistent/todolist
  v1/Event:
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f7924dabb9eb
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f79352bbff26
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f793da3c29f8
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f793dc32e90b
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f79402451c77
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f7940ac8fc16
    - mysql-persistent/mysql-5d4bf749b-trlx6.17b1f7940c5016b3
    - mysql-persistent/mysql-5d4bf749b.17b1f7915d32ff6f
    - mysql-persistent/mysql.17b1f790fc800fc1
    - mysql-persistent/mysql.17b1f79158eecc13
    - mysql-persistent/mysql.17b1f7915ed59511
    - mysql-persistent/mysql.17b1f7915eef22ed
    - mysql-persistent/mysql.17b1f79226e874f5
    - mysql-persistent/todolist-1-7r2kl.17b1f791e227d86e
    - mysql-persistent/todolist-1-7r2kl.17b1f792458f6eb4
    - mysql-persistent/todolist-1-7r2kl.17b1f79247e271c2
    - mysql-persistent/todolist-1-7r2kl.17b1f792dc326fbc
    - mysql-persistent/todolist-1-7r2kl.17b1f792e3b357be
    - mysql-persistent/todolist-1-7r2kl.17b1f792e4fa8ae9
    - mysql-persistent/todolist-1-7r2kl.17b1f79b001a1ffe
    - mysql-persistent/todolist-1-7r2kl.17b1f79bed691b81
    - mysql-persistent/todolist-1-7r2kl.17b1f79bf4f50ccd
    - mysql-persistent/todolist-1-7r2kl.17b1f79bf64a825f
    - mysql-persistent/todolist-1-deploy.17b1f7915aebb518
    - mysql-persistent/todolist-1-deploy.17b1f791ca7b25c4
    - mysql-persistent/todolist-1-deploy.17b1f791cc5c49ba
    - mysql-persistent/todolist-1-deploy.17b1f791d53eeb94
    - mysql-persistent/todolist-1-deploy.17b1f791d73bd878
    - mysql-persistent/todolist-1.17b1f791e13b4941
    - mysql-persistent/todolist.17b1f79155976696
    - mysql-persistent/velero-mysql-8w4nl.17b1fa082321e3a9
    - mysql-persistent/velero-mysql-8w4nl.17b1fa0835fdd4ed
    - mysql-persistent/velero-mysql-8w4nl.17b1fa08b07736ee
    - mysql-persistent/velero-mysql-f84bq.17b1f790fe949989
    - mysql-persistent/velero-mysql-f84bq.17b1f790fe94fdba
    - mysql-persistent/velero-mysql-ps5bz.17b1f9ed4173794b
    - mysql-persistent/velero-mysql-ps5bz.17b1f9ed5aa0a6ac
  v1/Namespace:
    - mysql-persistent
  v1/PersistentVolume:
    - pvc-4dc157d2-b60b-4b3e-86f5-68b291e5dac2
  v1/PersistentVolumeClaim:
    - mysql-persistent/mysql
  v1/Pod:
    - mysql-persistent/mysql-5d4bf749b-trlx6
    - mysql-persistent/todolist-1-7r2kl
    - mysql-persistent/todolist-1-deploy
  v1/ReplicationController:
    - mysql-persistent/todolist-1
  v1/Secret:
    - mysql-persistent/builder-dockercfg-9rqz6
    - mysql-persistent/builder-dockercfg-j9chj
    - mysql-persistent/builder-dockercfg-lnlrl
    - mysql-persistent/builder-dockercfg-t5dg2
    - mysql-persistent/builder-dockercfg-zh7rf
    - mysql-persistent/builder-token-zvhwx
    - mysql-persistent/default-dockercfg-4flvr
    - mysql-persistent/default-dockercfg-7rzq4
    - mysql-persistent/default-dockercfg-bk4nj
    - mysql-persistent/default-dockercfg-pw7z5
    - mysql-persistent/default-dockercfg-xkk89
    - mysql-persistent/default-token-8d24l
    - mysql-persistent/deployer-dockercfg-7fx68
    - mysql-persistent/deployer-dockercfg-dtsm9
    - mysql-persistent/deployer-dockercfg-hmb7x
    - mysql-persistent/deployer-dockercfg-hszgh
    - mysql-persistent/deployer-dockercfg-rgprg
    - mysql-persistent/deployer-token-vc58r
    - mysql-persistent/mysql-persistent-sa-dockercfg-27g2b
    - mysql-persistent/mysql-persistent-sa-dockercfg-2dfhl
    - mysql-persistent/mysql-persistent-sa-dockercfg-5shc7
    - mysql-persistent/mysql-persistent-sa-dockercfg-htgdp
    - mysql-persistent/mysql-persistent-sa-dockercfg-ps68b
    - mysql-persistent/mysql-persistent-sa-token-gnd7s
  v1/Service:
    - mysql-persistent/mysql
    - mysql-persistent/todolist
  v1/ServiceAccount:
    - mysql-persistent/builder
    - mysql-persistent/default
    - mysql-persistent/deployer
    - mysql-persistent/mysql-persistent-sa

Velero-Native Snapshots: <none included>
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/VMWARE-TANZU/velero$ 

* volumesnapshot and volumesnapshotcontents created.
* the volumesnapshot is deleted

whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get volumesnapshot -A
No resources found
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get volumesnapshotcontents -A
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT                              VOLUMESNAPSHOTNAMESPACE                   AGE
snapcontent-031323f7-1cce-4a10-afa1-483b2d558ca0   true         1073741824    Retain           ebs.csi.aws.com   csi-aws-vsc-velero    name-316814b2-6611-4e8e-9040-08a80891c2f5   ns-316814b2-6611-4e8e-9040-08a80891c2f5   4m16s

* volumesnapshotcontents details
* NOTE: this is set to retain automatically with velero, even though the volumesnapshotclass is set to delete
* spec:
* deletionPolicy: Retain

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  creationTimestamp: "2024-02-08T19:48:35Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
  generation: 1
  labels:
    velero.io/backup-name: mysql-persistent-csi-vsc-retain-policy-delete-2
  name: snapcontent-031323f7-1cce-4a10-afa1-483b2d558ca0
  resourceVersion: "14351786"
  uid: 173d0316-4ffc-4507-a537-333f0cea347d
spec:
  deletionPolicy: Retain
  driver: ebs.csi.aws.com
  source:
    snapshotHandle: snap-091cab53f897a9d08
  volumeSnapshotClassName: csi-aws-vsc-velero
  volumeSnapshotRef:
    apiVersion: snapshot.storage.k8s.io/v1
    kind: VolumeSnapshot
    name: name-316814b2-6611-4e8e-9040-08a80891c2f5
    namespace: ns-316814b2-6611-4e8e-9040-08a80891c2f5
status:
  creationTime: 1707421708291000000
  readyToUse: true
  restoreSize: 1073741824
  snapshotHandle: snap-091cab53f897a9d08

## delete the app namespace

oc delete namespace mysql-persistent
namespace "mysql-persistent" deleted

## execute a restore

 oc create -f mysql-restore.yaml 
restore.velero.io/mysql-persistent created

## check status

oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
No resources found in mysql-persistent namespace.
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                        READY   STATUS     RESTARTS   AGE
pod/mysql-5d4bf749b-kgznr   0/1     Running    0          29s
pod/todolist-1-6d4ms        0/1     Init:0/1   0          27s
pod/todolist-1-deploy       1/1     Running    0          29s

NAME                               DESIRED   CURRENT   READY   AGE
replicationcontroller/todolist-1   1         1         0       29s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql      ClusterIP   172.30.176.255   <none>        3306/TCP   29s
service/todolist   ClusterIP   172.30.40.64     <none>        8000/TCP   29s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   0/1     1            0           29s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-5d4bf749b   1         1         0       29s

NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/todolist   1          1         1         config

NAME                                      HOST/PORT                                                                                PATH   SERVICES   PORT    TERMINATION   WILDCARD
route.route.openshift.io/todolist-route   todolist-route-mysql-persistent.apps.cluster-wdh01102024g.wdh01102024g.mg.dog8code.com   /      todolist   <all>                 None
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                        READY   STATUS     RESTARTS   AGE
pod/mysql-5d4bf749b-kgznr   0/1     Running    0          36s
pod/todolist-1-6d4ms        0/1     Init:0/1   0          34s
pod/todolist-1-deploy       1/1     Running    0          36s

NAME                               DESIRED   CURRENT   READY   AGE
replicationcontroller/todolist-1   1         1         0       36s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql      ClusterIP   172.30.176.255   <none>        3306/TCP   36s
service/todolist   ClusterIP   172.30.40.64     <none>        8000/TCP   36s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   0/1     1            0           36s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-5d4bf749b   1         1         0       36s

NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/todolist   1          1         1         config

NAME                                      HOST/PORT                                                                                PATH   SERVICES   PORT    TERMINATION   WILDCARD
route.route.openshift.io/todolist-route   todolist-route-mysql-persistent.apps.cluster-wdh01102024g.wdh01102024g.mg.dog8code.com   /      todolist   <all>                 None
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                        READY   STATUS            RESTARTS   AGE
pod/mysql-5d4bf749b-kgznr   1/1     Running           0          45s
pod/todolist-1-6d4ms        0/1     PodInitializing   0          43s
pod/todolist-1-deploy       1/1     Running           0          45s

NAME                               DESIRED   CURRENT   READY   AGE
replicationcontroller/todolist-1   1         1         0       45s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql      ClusterIP   172.30.176.255   <none>        3306/TCP   45s
service/todolist   ClusterIP   172.30.40.64     <none>        8000/TCP   45s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           45s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-5d4bf749b   1         1         1       45s

NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/todolist   1          1         1         config

NAME                                      HOST/PORT                                                                                PATH   SERVICES   PORT    TERMINATION   WILDCARD
route.route.openshift.io/todolist-route   todolist-route-mysql-persistent.apps.cluster-wdh01102024g.wdh01102024g.mg.dog8code.com   /      todolist   <all>                 None
whayutin@thinkdoe:~/OADP/SETUP/datamover$ oc get all -n mysql-persistent
Warning: apps.openshift.io/v1 DeploymentConfig is deprecated in v4.14+, unavailable in v4.10000+
NAME                        READY   STATUS      RESTARTS   AGE
pod/mysql-5d4bf749b-kgznr   1/1     Running     0          49s
pod/todolist-1-6d4ms        1/1     Running     0          47s
pod/todolist-1-deploy       0/1     Completed   0          49s

NAME                               DESIRED   CURRENT   READY   AGE
replicationcontroller/todolist-1   1         1         1       49s

NAME               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/mysql      ClusterIP   172.30.176.255   <none>        3306/TCP   49s
service/todolist   ClusterIP   172.30.40.64     <none>        8000/TCP   49s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/mysql   1/1     1            1           49s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/mysql-5d4bf749b   1         1         1       49s

NAME                                          REVISION   DESIRED   CURRENT   TRIGGERED BY
deploymentconfig.apps.openshift.io/todolist   1          1         1         config

NAME                                      HOST/PORT                                                                                PATH   SERVICES   PORT    TERMINATION   WILDCARD
route.route.openshift.io/todolist-route   todolist-route-mysql-persistent.apps.cluster-wdh01102024g.wdh01102024g.mg.dog8code.com   /      todolist   <all>                 None

## check volumesnapshots and volumesnapshotcontents

* volumesnapshots

oc get volumesnapshot -A
NAMESPACE          NAME                 READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT             RESTORESIZE   SNAPSHOTCLASS        SNAPSHOTCONTENT                   CREATIONTIME   AGE
mysql-persistent   velero-mysql-9wfk8   true                     velero-velero-mysql-9wfk8-wk6fk   1Gi           csi-aws-vsc-velero   velero-velero-mysql-9wfk8-wk6fk   9m49s          2m36s

* volumesnapshotcontents

oc get volumesnapshotcontents -A
NAME                                               READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER            VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT                              VOLUMESNAPSHOTNAMESPACE                   AGE
snapcontent-031323f7-1cce-4a10-afa1-483b2d558ca0   true         1073741824    Retain           ebs.csi.aws.com   csi-aws-vsc-velero    name-316814b2-6611-4e8e-9040-08a80891c2f5   ns-316814b2-6611-4e8e-9040-08a80891c2f5   10m
velero-velero-mysql-9wfk8-wk6fk                    true         1073741824    Retain           ebs.csi.aws.com   csi-aws-vsc-velero    velero-mysql-9wfk8                          mysql-persistent    

## manually verified the data was correct between backup and restore.

## s3 backup

s3cmd ls s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/
2024-02-08 19:48          418  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-csi-volumesnapshotclasses.json.gz
2024-02-08 19:48          707  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-csi-volumesnapshotcontents.json.gz
2024-02-08 19:48          586  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-csi-volumesnapshots.json.gz
2024-02-08 19:48           27  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-itemoperations.json.gz
2024-02-08 19:48        11841  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-logs.gz
2024-02-08 19:48           29  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-podvolumebackups.json.gz
2024-02-08 19:48         1188  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-resource-list.json.gz
2024-02-08 19:48           49  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-results.gz
2024-02-08 19:48           29  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2-volumesnapshots.json.gz
2024-02-08 19:48       252526  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/mysql-persistent-csi-vsc-retain-policy-delete-2.tar.gz
2024-02-08 19:48         3053  s3://cvpbucketuswest2/velero/backups/mysql-persistent-csi-vsc-retain-policy-delete-2/velero-backup.json

## related code
https://github.com/search?q=repo%3Avmware-tanzu%2Fvelero%20language%3AGo%20deletionPolicy&type=code

## related velero doc:
https://github.com/vmware-tanzu/velero/blob/release-1.11/site/content/posts/2020-05-27-CSI-integration.md