# storage class OADP-1821 notes #2

## context

The dpa volumeOptionsForStorageClasses is for volsync operations only.
The volumes are originally create w/ gp2-csi-copy-1 in the application namespace.

When and ONLY when volumes are used in the protected namespace ( openshift-adp ) volsync will use the storageclass gp2-csi-copy-2

volumeOptionsForStorageClasses parameters are volsync ONLY parameters and used in the context of volsync only.

volumeOptionsForStorageClasses:
  gp2-csi-copy-1:
    destinationVolumeOptions:
      storageClassName: gp2-csi-copy-2
    sourceVolumeOptions:
      storageClassName: gp2-csi-copy-2

[whayutin@thinkdoe minimal-8csivol]$ oc get sc
NAME             PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2-csi          ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   13d
gp2-csi-copy-1   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   4h44m
gp2-csi-copy-2   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   4h43m
gp3-csi          ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   13d

[whayutin@thinkdoe minimal-8csivol]$ oc get dpa -n openshift-adp dpa-sample -o yaml
apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  creationTimestamp: "2023-06-05T17:22:53Z"
  generation: 8
  name: dpa-sample
  namespace: openshift-adp
  resourceVersion: "15356595"
  uid: 0acbe7e3-6d14-429e-975d-786187f6b905
spec:
  backupLocations:
  - velero:
      config:
        profile: default
        region: us-east-1
      credential:
        key: cloud
        name: cloud-credentials
      default: true
      objectStorage:
        bucket: cvpbucket
        prefix: velero
      provider: aws
  configuration:
    restic:
      enable: false
    velero:
      defaultPlugins:
      - openshift
      - aws
      - csi
      - vsm
      featureFlags:
      - EnableCSI
  features:
    dataMover:
      credentialName: restic-secret
      enable: true
      volumeOptionsForStorageClasses:
        gp2-csi-copy-1:
          destinationVolumeOptions:
            storageClassName: gp2-csi-copy-2
          sourceVolumeOptions:
            storageClassName: gp2-csi-copy-2
  snapshotLocations:
  - velero:
      config:
        profile: default
        region: us-east-1
      provider: aws
status:
  conditions:
  - lastTransitionTime: "2023-06-05T17:22:54Z"
    message: Reconcile complete
    reason: Complete
    status: "True"
    type: Reconciled

## setup

screen capture of test: https://file.rdu.redhat.com/~whayutin/OADP/OADP-1821.webm

window #1: `watch -n 5 oc get pvc -n minimal-8csivol`
window #2: `watch -n 5 oc get pvc -n openshift-adp`
window #3: `watch -n 5 datamover_resources.sh -[b,r] -d`
window #4: cli
* https://github.com/openshift/oadp-operator/blob/master/docs/examples/datamover_resources.sh

## machine state

clean

[whayutin@thinkdoe minimal-8csivol]$ oc get backup -n openshift-adp
No resources found in openshift-adp namespace.
[whayutin@thinkdoe minimal-8csivol]$ oc get restore -n openshift-adp
No resources found in openshift-adp namespace.
[whayutin@thinkdoe minimal-8csivol]$ s3cmd ls s3://cvpbucket/
[whayutin@thinkdoe minimal-8csivol]$ 

[whayutin@thinkdoe minimal-8csivol]$ oc delete vsb -A --all; oc delete vsr -A --all; oc delete vsc -A --all; oc delete vs -A --all; oc delete replicationsources.volsync.backube -A --all; oc delete replicationdestination.volsync.backube -A --all
No resources found
No resources found
No resources found
No resources found
No resources found
No resources found
[whayutin@thinkdoe minimal-8csivol]$ 

hen deleting "minimal-8csivol.yaml": namespaces "minimal-8csivol" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": deployments.apps "minimal-8csivol" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume1" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume2" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume3" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume4" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume5" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume6" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume7" not found
Error from server (NotFound): error when deleting "minimal-8csivol.yaml": persistentvolumeclaims "volume8" not found

## create minimal-8csivol.yaml

[whayutin@thinkdoe minimal-8csivol]$ oc create -f minimal-8csivol.yaml 
namespace/minimal-8csivol created
deployment.apps/minimal-8csivol created
persistentvolumeclaim/volume1 created
persistentvolumeclaim/volume2 created
persistentvolumeclaim/volume3 created
persistentvolumeclaim/volume4 created
persistentvolumeclaim/volume5 created
persistentvolumeclaim/volume6 created
persistentvolumeclaim/volume7 created
persistentvolumeclaim/volume8 created
[whayutin@thinkdoe minimal-8csivol]$ oc get pvc -n minimal-8csivol
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
volume1   Bound    pvc-2af3bb5b-d3d9-4b05-8038-c8a5351e0fd8   1Gi        RWO            gp2-csi-copy-1   14s
volume2   Bound    pvc-5c622a35-f48e-45c1-9be6-ebcae40629a2   1Gi        RWO            gp2-csi-copy-1   14s
volume3   Bound    pvc-6a4e9a8a-a73e-41ce-944a-71c63b16b2bc   1Gi        RWO            gp2-csi-copy-1   14s
volume4   Bound    pvc-a11c6e4d-1e82-4e81-9fd5-84c5ba62c1a7   1Gi        RWO            gp2-csi-copy-1   14s
volume5   Bound    pvc-9e01f16e-ceee-47cc-945f-a764be6cd0a2   1Gi        RWO            gp2-csi-copy-1   14s
volume6   Bound    pvc-9039b911-3ab1-40e1-bd41-062e5956ecd5   1Gi        RWO            gp2-csi-copy-1   14s
volume7   Bound    pvc-264d2018-cabf-46ab-be21-28a432096cc4   1Gi        RWO            gp2-csi-copy-1   14s
volume8   Bound    pvc-b1672d20-9185-43c7-af96-865e4c24a921   1Gi        RWO            gp2-csi-copy-1   14s
[whayutin@thinkdoe minimal-8csivol]$ 
[whayutin@thinkdoe minimal-8csivol]$ 
[whayutin@thinkdoe minimal-8csivol]$ ls
minimal-3csivol-backup.yaml   minimal-3csivol.yaml         minimal-8csivol-restore.yaml
minimal-3csivol-restore.yaml  minimal-8csivol-backup.yaml  minimal-8csivol.yaml
[whayutin@thinkdoe minimal-8csivol]$ oc create -f minimal-8csivol-backup.yaml 
backup.velero.io/minimal-8csivol-1 created



[whayutin@thinkdoe minimal-8csivol]$ oc get backup.velero.io/minimal-8csivol-1 -n openshift-adp -o yaml
apiVersion: velero.io/v1
kind: Backup
metadata:
  annotations:
    velero.io/source-cluster-k8s-gitversion: v1.25.8+27e744f
    velero.io/source-cluster-k8s-major-version: "1"
    velero.io/source-cluster-k8s-minor-version: "25"
  creationTimestamp: "2023-06-07T18:26:19Z"
  generation: 4
  labels:
    velero.io/storage-location: dpa-sample-1
  name: minimal-8csivol-1
  namespace: openshift-adp
  resourceVersion: "15450289"
  uid: 13e3e32a-c86b-49c3-9c8f-2415981ebf88
spec:
  csiSnapshotTimeout: 10m0s
  defaultVolumesToFsBackup: false
  includedNamespaces:
  - minimal-8csivol
  itemOperationTimeout: 1h0m0s
  storageLocation: dpa-sample-1
  ttl: 720h0m0s
  volumeSnapshotLocations:
  - dpa-sample-1
status:
  backupItemOperationsAttempted: 8
  backupItemOperationsCompleted: 8
  csiVolumeSnapshotsAttempted: 8
  csiVolumeSnapshotsCompleted: 8
  expiration: "2023-07-07T18:26:19Z"
  formatVersion: 1.1.0
  phase: Finalizing
  startTimestamp: "2023-06-07T18:26:19Z"
  version: 1

Every 5.0s: oc get pvc -n openshift-adp                                                                                                 thinkdoe: Wed Jun  7 12:35:21 2023

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
volsync-vsb-7ngck-rep-src-cache   Bound    pvc-4fdd9ef2-fd3b-4711-a80c-7f7bc18c8da3   1Gi        RWO            gp2-csi-copy-2   2m38s
volsync-vsb-c4rbr-rep-src-cache   Bound    pvc-78042f80-1246-4bea-8f66-97f5ec5428ee   1Gi        RWO            gp2-csi-copy-2   5m23s
volsync-vsb-d5x2d-rep-src-cache   Bound    pvc-077b26b3-356a-467b-9475-220e98bed220   1Gi        RWO            gp2-csi-copy-2   7m30s
volsync-vsb-gqlqf-rep-src-cache   Bound    pvc-73670130-ff69-4414-bcb9-60a656971763   1Gi        RWO            gp2-csi-copy-2   91s
volsync-vsb-hk4sn-rep-src-cache   Bound    pvc-b47ee505-bae5-4364-bd10-a7fcb061552e   1Gi        RWO            gp2-csi-copy-2   4m1s
volsync-vsb-l8nzt-rep-src-cache   Bound    pvc-7d1092c8-426f-4ff4-8644-560463b31531   1Gi        RWO            gp2-csi-copy-2   2m12s
volsync-vsb-swzlq-rep-src-cache   Bound    pvc-75e29343-54a9-4db5-8ca2-00e966b2edc9   1Gi        RWO            gp2-csi-copy-2   4m17s
volsync-vsb-vp588-rep-src-cache   Bound    pvc-96f6fe75-8f2e-44cc-a866-6f89ec17ccc5   1Gi        RWO            gp2-csi-copy-2   3m

Every 5.0s: oc get pvc -n minimal-8csivol                                                               thinkdoe: Wed Jun  7 12:35:35 2023

NAME	  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
volume1   Bound    pvc-2af3bb5b-d3d9-4b05-8038-c8a5351e0fd8   1Gi        RWO            gp2-csi-copy-1   10m
volume2   Bound    pvc-5c622a35-f48e-45c1-9be6-ebcae40629a2   1Gi        RWO            gp2-csi-copy-1   10m
volume3   Bound    pvc-6a4e9a8a-a73e-41ce-944a-71c63b16b2bc   1Gi        RWO            gp2-csi-copy-1   10m
volume4   Bound    pvc-a11c6e4d-1e82-4e81-9fd5-84c5ba62c1a7   1Gi        RWO            gp2-csi-copy-1   10m
volume5   Bound    pvc-9e01f16e-ceee-47cc-945f-a764be6cd0a2   1Gi        RWO            gp2-csi-copy-1   10m
volume6   Bound    pvc-9039b911-3ab1-40e1-bd41-062e5956ecd5   1Gi        RWO            gp2-csi-copy-1   10m
volume7   Bound    pvc-264d2018-cabf-46ab-be21-28a432096cc4   1Gi        RWO            gp2-csi-copy-1   10m
volume8   Bound    pvc-b1672d20-9185-43c7-af96-865e4c24a921   1Gi        RWO            gp2-csi-copy-1   10m

## delete app and restore

[whayutin@thinkdoe minimal-8csivol]$ oc delete -f minimal-8csivol.yaml 
namespace "minimal-8csivol" deleted
deployment.apps "minimal-8csivol" deleted
persistentvolumeclaim "volume1" deleted
persistentvolumeclaim "volume2" deleted
persistentvolumeclaim "volume3" deleted
persistentvolumeclaim "volume4" deleted
persistentvolumeclaim "volume5" deleted
persistentvolumeclaim "volume6" deleted
persistentvolumeclaim "volume7" deleted
persistentvolumeclaim "volume8" deleted
[whayutin@thinkdoe minimal-8csivol]$ oc delete vsb -A --all; oc delete vsr -A --all; oc delete vsc -A --all; oc delete vs -A --all; oc delete replicationsources.volsync.backube -A --all; oc delete replicationdestination.volsync.backube -A --all
No resources found
No resources found
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-33ddc8f6-1ddf-4b53-b189-62c4428b83df" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-383c5dfa-3dfe-497b-aabd-e8566178bc48" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-4c531dfc-f8d2-41c7-8dce-d61c590ba6e3" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-575eb81d-fc71-47c0-a120-076dd10acf90" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-9095e586-ac44-491b-8cfd-b86cd2016f47" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-9e4f1bac-d88d-453f-b09b-712528463da9" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-b7ac405c-8632-453a-9cb5-3037d889e995" deleted
volumesnapshotcontent.snapshot.storage.k8s.io "snapcontent-f0a1bce2-5549-4c03-9935-70efd26bc581" deleted
No resources found
replicationsource.volsync.backube "vsb-7ngck-rep-src" deleted
replicationsource.volsync.backube "vsb-c4rbr-rep-src" deleted
replicationsource.volsync.backube "vsb-d5x2d-rep-src" deleted
replicationsource.volsync.backube "vsb-gqlqf-rep-src" deleted
replicationsource.volsync.backube "vsb-hk4sn-rep-src" deleted
replicationsource.volsync.backube "vsb-l8nzt-rep-src" deleted
replicationsource.volsync.backube "vsb-swzlq-rep-src" deleted
replicationsource.volsync.backube "vsb-vp588-rep-src" deleted
No resources found
[whayutin@thinkdoe minimal-8csivol]$ 
[whayutin@thinkdoe minimal-8csivol]$ 
[whayutin@thinkdoe minimal-8csivol]$ oc create -f minimal-8csivol-restore.yaml 
restore.velero.io/minimal-8csivol-restore-1 created
[whayutin@thinkdoe minimal-8csivol]$ oc get restore.velero.io/minimal-8csivol-restore-1 -n openshift-adp -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2023-06-07T18:37:13Z"
  generation: 4
  name: minimal-8csivol-restore-1
  namespace: openshift-adp
  resourceVersion: "15452597"
  uid: b371309e-36d7-42f5-88f1-95ae6283311b
spec:
  backupName: minimal-8csivol-1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  restorePVs: true
status:
  phase: InProgress
  progress:
    itemsRestored: 1
    totalItems: 76
  startTimestamp: "2023-06-07T18:37:13Z"
[whayutin@thinkdoe minimal-8csivol]$ 

## final state

Every 5.0s: oc get pvc -n openshift-adp                                                         thinkdoe: Wed Jun  7 12:40:18 2023

No resources found in openshift-adp namespace.

Every 5.0s: oc get pvc -n minimal-8csivol                                 thinkdoe: Wed Jun  7 12:40:33 2023

NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
volume1   Bound    pvc-4c84a276-c727-493a-bc6f-35d0f9a2c2af   1Gi        RWO            gp2-csi-copy-1   41s
volume2   Bound    pvc-02bcc29a-c563-46ff-b587-4b06df0f45c1   1Gi        RWO            gp2-csi-copy-1   41s
volume3   Bound    pvc-848ebeef-4d29-417d-9506-b3c2875101b2   1Gi        RWO            gp2-csi-copy-1   41s
volume4   Bound    pvc-ef2c0ecc-55fc-4d16-ab2b-054ee17ff8c3   1Gi        RWO            gp2-csi-copy-1   40s
volume5   Bound    pvc-27d5439f-dd07-4aa3-9af5-8e30e6a3360b   1Gi        RWO            gp2-csi-copy-1   40s
volume6   Bound    pvc-9754ac65-bc2c-4e4f-8e73-4f5174cbdc3a   1Gi        RWO            gp2-csi-copy-1   40s
volume7   Bound    pvc-7178e6fa-fe20-4e2f-9ecb-7b182291efb3   1Gi        RWO            gp2-csi-copy-1   40s
volume8   Bound    pvc-519e7169-8c58-4486-bed7-eb9c26104b52   1Gi        RWO            gp2-csi-copy-1   39s

Every 5.0s: datamover_resources.sh -r -d                                             thinkdoe: Wed Jun  7 12:40:38 2023

Get Restores:

NAME                        BACKUP              STATUS      STARTED                         COMPLETED
     ERRORS   WARNINGS   CREATED                         SELECTOR
minimal-8csivol-restore-1   minimal-8csivol-1   Completed   2023-06-07 18:37:13 +0000 UTC   2023-06-07 18:40:27 +0000 U
TC   0        14         2023-06-07 18:37:13 +0000 UTC   <none>
No resources found

Total VSR:  0
No resources found
Completed:  0
No resources found
InProgress:  0
No resources found
SnapshotRestoreDone:  0

VSR STATUS
Completed:  0
Processing:  0
Queued:  0
No resources found

Total ReplicationDestinations:  0

***** VOLUME SNAPSHOT CONTENTS ******

[whayutin@thinkdoe minimal-8csivol]$ oc get restore.velero.io/minimal-8csivol-restore-1 -n openshift-adp -o yaml
apiVersion: velero.io/v1
kind: Restore
metadata:
  creationTimestamp: "2023-06-07T18:37:13Z"
  generation: 24
  name: minimal-8csivol-restore-1
  namespace: openshift-adp
  resourceVersion: "15456229"
  uid: b371309e-36d7-42f5-88f1-95ae6283311b
spec:
  backupName: minimal-8csivol-1
  excludedResources:
  - nodes
  - events
  - events.events.k8s.io
  - backups.velero.io
  - restores.velero.io
  - resticrepositories.velero.io
  - csinodes.storage.k8s.io
  - volumeattachments.storage.k8s.io
  - backuprepositories.velero.io
  itemOperationTimeout: 1h0m0s
  restorePVs: true
status:
  completionTimestamp: "2023-06-07T18:40:27Z"
  phase: Completed
  progress:
    itemsRestored: 77
    totalItems: 77
  restoreItemOperationsAttempted: 8
  restoreItemOperationsCompleted: 8
  startTimestamp: "2023-06-07T18:37:13Z"
  warnings: 14

[whayutin@thinkdoe minimal-8csivol]$ oc get all -n minimal-8csivol
NAME                                   READY   STATUS    RESTARTS   AGE
pod/minimal-8csivol-6d848c77f8-4jvt7   1/1     Running   0          116s

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/minimal-8csivol   1/1     1            1           116s

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/minimal-8csivol-6d848c77f8   1         1         1       116s
[whayutin@thinkdoe minimal-8csivol]$