my notes

following https://github.com/openshift/oadp-operator/blob/54aad556507aee70f3c7647afe24021d458d1b23/docs/developer/unsupported_server_args.md

Create configmaps

my-config-velero-server.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config-velero-server
  namespace: openshift-adp
data:
  # Configuration values
  features: "EnableCSI"
  uploader-type: "kopia"
  fs-backup-timeout: "5h"
  disable-informer-cache: "false"
  restore-resource-priorities: "securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io"

my-config-node-agent-server.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config-node-agent-server
  namespace: openshift-adp
data:
  # Configuration values
  data-mover-prepare-timeout: 45m
  resource-timeout: 15m

add labels to DPA w/o creating configmaps

5 apiVersion: oadp.openshift.io/v1alpha1
  6 kind: DataProtectionApplication
  7 metadata:
  8   annotations:
  9     oadp.openshift.io/unsupported-node-agent-server-args: my-config-node-agent-server
 10     oadp.openshift.io/unsupported-velero-server-args: my-config-velero-server
 11   creationTimestamp: "2024-07-17T14:55:32Z"

this is expected and velero server continues to run :)


	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234
1.7212318149180899e+09	ERROR	Reconciler error	{"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "f0391420-e941-49e7-9162-dc2ef8b145a4", "error": "ConfigMap \"my-config-velero-server\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234
1.72123182525822e+09	ERROR	Reconciler error	{"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "8651ea86-bb94-4180-9fc2-e9215e96a79d", "error": "ConfigMap \"my-config-velero-server\" not found"}

create the config maps

oc create configmaps
velero restarts as expected


1.7212326436534598e+09	DEBUG	events	Normal	{"object": {"kind":"Deployment","namespace":"openshift-adp","name":"velero","uid":"0e7056c1-82d7-4cbc-b5a8-b33a00e1836a","apiVersion":"apps/v1","resourceVersion":"7568574"}, "reason": "VeleroDeploymentReconciled", "message": "performed updated on velero deployment openshift-adp/velero"}
1.7212326436925373e+09	DEBUG	events	Normal	{"object": {"kind":"DaemonSet","namespace":"openshift-adp","name":"node-agent","uid":"75b3e75b-8474-4bf3-9dd1-bb927fbd3d3d","apiVersion":"apps/v1","resourceVersion":"7568576"}, "reason": "NodeAgentDaemonsetReconciled", "message": "performed updated on NodeAgent deployment openshift-adp/node-agent"}

velero deployment diff

--- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/orig/velero.yaml
+++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/velero.yaml
@@ -2,9 +2,9 @@
 kind: Deployment
 metadata:
   annotations:
-    deployment.kubernetes.io/revision: "1"
+    deployment.kubernetes.io/revision: "2"
   creationTimestamp: "2024-07-17T14:57:03Z"
-  generation: 1
+  generation: 2
   labels:
     app.kubernetes.io/component: server
     app.kubernetes.io/instance: dpa-sample
@@ -21,7 +21,7 @@
     kind: DataProtectionApplication
     name: dpa-sample
     uid: b355a54f-dda3-448d-94aa-faf4775f3891
-  resourceVersion: "7492736"
+  resourceVersion: "7569152"
   uid: 0e7056c1-82d7-4cbc-b5a8-b33a00e1836a
 spec:
   progressDeadlineSeconds: 600
@@ -60,11 +60,11 @@
       containers:
       - args:
         - server
+        - --disable-informer-cache=false
         - --features=EnableCSI
+        - --fs-backup-timeout=5h
+        - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io
         - --uploader-type=kopia
-        - --fs-backup-timeout=4h
-        - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io
-        - --disable-informer-cache=false
         command:
         - /velero
         env:
@@ -168,12 +168,12 @@
     status: "True"
     type: Available
   - lastTransitionTime: "2024-07-17T14:57:03Z"
-    lastUpdateTime: "2024-07-17T14:57:24Z"
-    message: ReplicaSet "velero-6b9fcc6f48" has successfully progressed.
+    lastUpdateTime: "2024-07-17T16:11:03Z"
+    message: ReplicaSet "velero-77d6b5fcf5" has successfully progressed.
     reason: NewReplicaSetAvailable
     status: "True"
     type: Progressing
-  observedGeneration: 1
+  observedGeneration: 2
   readyReplicas: 1
   replicas: 1
   updatedReplicas: 1

deamonset node-agent diff


--- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/orig/node-agent.yaml
+++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/node-agent.yaml
@@ -2,9 +2,9 @@
 kind: DaemonSet
 metadata:
   annotations:
-    deprecated.daemonset.template.generation: "1"
+    deprecated.daemonset.template.generation: "2"
   creationTimestamp: "2024-07-17T14:57:03Z"
-  generation: 1
+  generation: 2
   labels:
     component: velero
     name: node-agent
@@ -17,7 +17,7 @@
     kind: DataProtectionApplication
     name: dpa-sample
     uid: b355a54f-dda3-448d-94aa-faf4775f3891
-  resourceVersion: "7492533"
+  resourceVersion: "7569460"
   uid: 75b3e75b-8474-4bf3-9dd1-bb927fbd3d3d
 spec:
   revisionHistoryLimit: 10
@@ -36,6 +36,8 @@
       - args:
         - node-agent
         - server
+        - --data-mover-prepare-timeout=45m
+        - --resource-timeout=15m
         command:
         - /velero
         env:
@@ -117,5 +119,5 @@
   numberAvailable: 9
   numberMisscheduled: 0
   numberReady: 9
-  observedGeneration: 1
+  observedGeneration: 2
   updatedNumberScheduled: 9

small edit on velero configmap

change the order of restore resources priorities, save configmap.

1.721238622789183e+09	DEBUG	events	Normal	{"object": {"kind":"Deployment","namespace":"openshift-adp","name":"velero","uid":"0e7056c1-82d7-4cbc-b5a8-b33a00e1836a","apiVersion":"apps/v1","resourceVersion":"7672191"}, "reason": "VeleroDeploymentReconciled", "message": "performed updated on velero deployment openshift-adp/velero"}

velero pod restarts, and change reflected in deployment

--- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/velero.yaml
+++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/1/velero.yml
@@ -2,9 +2,9 @@
 kind: Deployment
 metadata:
   annotations:
-    deployment.kubernetes.io/revision: "2"
+    deployment.kubernetes.io/revision: "3"
   creationTimestamp: "2024-07-17T14:57:03Z"
-  generation: 2
+  generation: 3
   labels:
     app.kubernetes.io/component: server
     app.kubernetes.io/instance: dpa-sample
@@ -21,7 +21,7 @@
     kind: DataProtectionApplication
     name: dpa-sample
     uid: b355a54f-dda3-448d-94aa-faf4775f3891
-  resourceVersion: "7569152"
+  resourceVersion: "7672352"
   uid: 0e7056c1-82d7-4cbc-b5a8-b33a00e1836a
 spec:
   progressDeadlineSeconds: 600
@@ -63,7 +63,7 @@
         - --disable-informer-cache=false
         - --features=EnableCSI
         - --fs-backup-timeout=5h
-        - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io
+        - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusterresourcesets.addons.cluster.x-k8s.io,clusters.cluster.x-k8s.io
         - --uploader-type=kopia
         command:
         - /velero
@@ -168,12 +168,12 @@
     status: "True"
     type: Available
   - lastTransitionTime: "2024-07-17T14:57:03Z"
-    lastUpdateTime: "2024-07-17T16:11:03Z"
-    message: ReplicaSet "velero-77d6b5fcf5" has successfully progressed.
+    lastUpdateTime: "2024-07-17T17:50:29Z"
+    message: ReplicaSet "velero-5c88755855" has successfully progressed.
     reason: NewReplicaSetAvailable
     status: "True"
     type: Progressing
-  observedGeneration: 2
+  observedGeneration: 3
   readyReplicas: 1
   replicas: 1
   updatedReplicas: 1

Out of Sync DPA w/ unsupported args config map.

DPA uploader type: kopia
Set the config map to uploader type: restic
- reconciles, updates velero deployment to - --uploader-type=restic
- DPA remains set to:
```
configuration:
nodeAgent:
  enable: true
  uploaderType: kopia
```
This is expected but needs to be CALLED out in developer docs
- https://github.com/openshift/oadp-operator/blob/54aad556507aee70f3c7647afe24021d458d1b23/docs/developer/unsupported_server_args.md

oadp-must-gather

In the current must-gather for OADP-1.3+ testing if the custom config maps are there…
- YES
  - In dir abd05fc5/namespaces/openshift-adp/core/configmaps.yaml
Probably doc this for developer debug if needed.

Change the name of the configmap in the dpa annotation

Velero is not restarted
Correctly get reconcile errors:

1.7212403003205552e+09	ERROR	Reconciler error	{"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "3930ac76-ccd8-489f-89c0-f9a02182fe82", "error": "ConfigMap \"foo-config-velero-server\" not found"}
sigs.k8s.io/controller-runtime/pkg/inter

Changing the configmap name back to the proper name, Does NOT work


1.7212404696502051e+09	ERROR	Reconciler error	{"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "9b7e1032-6aba-45e5-b41f-26444faca474", "error": "Operation cannot be fulfilled on dataprotectionapplications.oadp.openshift.io \"dpa-sample\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234

Add invalid key/value pairs to the configmap

Add invalid key pair


apiVersion: v1
  6 data:
  7   disable-informer-cache: "false"
  8   features: EnableCSI
  9   fs-backup-timeout: 5h
 10   joe: Shmoe

configmap is updated, reconciled

Velero deployment correctly updated

      containers:
  - args:
    - server
    - --disable-informer-cache=false
    - --features=EnableCSI
    - --fs-backup-timeout=5h
    - --joe=Shmoe

Velero correctly goes into crashloopbackoff

Remove the invalid key/value pair from the configmap..
- rereconciles
- velero restarts correctly :)

my notes

Create configmaps

add labels to DPA w/o creating configmaps

create the config maps

velero deployment diff

deamonset node-agent diff

small edit on velero configmap

Out of Sync DPA w/ unsupported args config map.

oadp-must-gather

Change the name of the configmap in the dpa annotation

Add invalid key/value pairs to the configmap

Read more

OADP-3866

virt-todo

draft Create a minio server using local storage

Cloud Casa / block devices Notes