# my notes * following https://github.com/openshift/oadp-operator/blob/54aad556507aee70f3c7647afe24021d458d1b23/docs/developer/unsupported_server_args.md ## Create configmaps * my-config-velero-server.yml ``` apiVersion: v1 kind: ConfigMap metadata: name: my-config-velero-server namespace: openshift-adp data: # Configuration values features: "EnableCSI" uploader-type: "kopia" fs-backup-timeout: "5h" disable-informer-cache: "false" restore-resource-priorities: "securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io" ``` * my-config-node-agent-server.yml ``` apiVersion: v1 kind: ConfigMap metadata: name: my-config-node-agent-server namespace: openshift-adp data: # Configuration values data-mover-prepare-timeout: 45m resource-timeout: 15m ``` ## add labels to DPA w/o creating configmaps ``` 5 apiVersion: oadp.openshift.io/v1alpha1 6 kind: DataProtectionApplication 7 metadata: 8 annotations: 9 oadp.openshift.io/unsupported-node-agent-server-args: my-config-node-agent-server 10 oadp.openshift.io/unsupported-velero-server-args: my-config-velero-server 11 creationTimestamp: "2024-07-17T14:55:32Z" ``` * this is expected and velero server continues to run :) ``` /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234 1.7212318149180899e+09 ERROR Reconciler error {"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "f0391420-e941-49e7-9162-dc2ef8b145a4", "error": "ConfigMap \"my-config-velero-server\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234 1.72123182525822e+09 ERROR Reconciler error {"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "8651ea86-bb94-4180-9fc2-e9215e96a79d", "error": "ConfigMap \"my-config-velero-server\" not found"} ``` ## create the config maps * oc create configmaps * velero restarts as expected ``` 1.7212326436534598e+09 DEBUG events Normal {"object": {"kind":"Deployment","namespace":"openshift-adp","name":"velero","uid":"0e7056c1-82d7-4cbc-b5a8-b33a00e1836a","apiVersion":"apps/v1","resourceVersion":"7568574"}, "reason": "VeleroDeploymentReconciled", "message": "performed updated on velero deployment openshift-adp/velero"} 1.7212326436925373e+09 DEBUG events Normal {"object": {"kind":"DaemonSet","namespace":"openshift-adp","name":"node-agent","uid":"75b3e75b-8474-4bf3-9dd1-bb927fbd3d3d","apiVersion":"apps/v1","resourceVersion":"7568576"}, "reason": "NodeAgentDaemonsetReconciled", "message": "performed updated on NodeAgent deployment openshift-adp/node-agent"} ``` #### velero deployment diff ``` --- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/orig/velero.yaml +++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/velero.yaml @@ -2,9 +2,9 @@ kind: Deployment metadata: annotations: - deployment.kubernetes.io/revision: "1" + deployment.kubernetes.io/revision: "2" creationTimestamp: "2024-07-17T14:57:03Z" - generation: 1 + generation: 2 labels: app.kubernetes.io/component: server app.kubernetes.io/instance: dpa-sample @@ -21,7 +21,7 @@ kind: DataProtectionApplication name: dpa-sample uid: b355a54f-dda3-448d-94aa-faf4775f3891 - resourceVersion: "7492736" + resourceVersion: "7569152" uid: 0e7056c1-82d7-4cbc-b5a8-b33a00e1836a spec: progressDeadlineSeconds: 600 @@ -60,11 +60,11 @@ containers: - args: - server + - --disable-informer-cache=false - --features=EnableCSI + - --fs-backup-timeout=5h + - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io - --uploader-type=kopia - - --fs-backup-timeout=4h - - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io - - --disable-informer-cache=false command: - /velero env: @@ -168,12 +168,12 @@ status: "True" type: Available - lastTransitionTime: "2024-07-17T14:57:03Z" - lastUpdateTime: "2024-07-17T14:57:24Z" - message: ReplicaSet "velero-6b9fcc6f48" has successfully progressed. + lastUpdateTime: "2024-07-17T16:11:03Z" + message: ReplicaSet "velero-77d6b5fcf5" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - observedGeneration: 1 + observedGeneration: 2 readyReplicas: 1 replicas: 1 updatedReplicas: 1 ``` #### deamonset node-agent diff ``` --- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/orig/node-agent.yaml +++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/node-agent.yaml @@ -2,9 +2,9 @@ kind: DaemonSet metadata: annotations: - deprecated.daemonset.template.generation: "1" + deprecated.daemonset.template.generation: "2" creationTimestamp: "2024-07-17T14:57:03Z" - generation: 1 + generation: 2 labels: component: velero name: node-agent @@ -17,7 +17,7 @@ kind: DataProtectionApplication name: dpa-sample uid: b355a54f-dda3-448d-94aa-faf4775f3891 - resourceVersion: "7492533" + resourceVersion: "7569460" uid: 75b3e75b-8474-4bf3-9dd1-bb927fbd3d3d spec: revisionHistoryLimit: 10 @@ -36,6 +36,8 @@ - args: - node-agent - server + - --data-mover-prepare-timeout=45m + - --resource-timeout=15m command: - /velero env: @@ -117,5 +119,5 @@ numberAvailable: 9 numberMisscheduled: 0 numberReady: 9 - observedGeneration: 1 + observedGeneration: 2 updatedNumberScheduled: 9 ``` #### small edit on velero configmap * change the order of restore resources priorities, save configmap. ``` 1.721238622789183e+09 DEBUG events Normal {"object": {"kind":"Deployment","namespace":"openshift-adp","name":"velero","uid":"0e7056c1-82d7-4cbc-b5a8-b33a00e1836a","apiVersion":"apps/v1","resourceVersion":"7672191"}, "reason": "VeleroDeploymentReconciled", "message": "performed updated on velero deployment openshift-adp/velero"} ``` * velero pod restarts, and change reflected in deployment ``` --- /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/velero.yaml +++ /home/whayutin/OADP/TEST/UNSUPPORTED_ARGS/after_configmap/1/velero.yml @@ -2,9 +2,9 @@ kind: Deployment metadata: annotations: - deployment.kubernetes.io/revision: "2" + deployment.kubernetes.io/revision: "3" creationTimestamp: "2024-07-17T14:57:03Z" - generation: 2 + generation: 3 labels: app.kubernetes.io/component: server app.kubernetes.io/instance: dpa-sample @@ -21,7 +21,7 @@ kind: DataProtectionApplication name: dpa-sample uid: b355a54f-dda3-448d-94aa-faf4775f3891 - resourceVersion: "7569152" + resourceVersion: "7672352" uid: 0e7056c1-82d7-4cbc-b5a8-b33a00e1836a spec: progressDeadlineSeconds: 600 @@ -63,7 +63,7 @@ - --disable-informer-cache=false - --features=EnableCSI - --fs-backup-timeout=5h - - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusters.cluster.x-k8s.io,clusterresourcesets.addons.cluster.x-k8s.io + - --restore-resource-priorities=securitycontextconstraints,customresourcedefinitions,klusterletconfigs.config.open-cluster-management.io,managedcluster.cluster.open-cluster-management.io,namespaces,roles,rolebindings,clusterrolebindings,klusterletaddonconfig.agent.open-cluster-management.io,managedclusteraddon.addon.open-cluster-management.io,storageclasses,volumesnapshotclass.snapshot.storage.k8s.io,volumesnapshotcontents.snapshot.storage.k8s.io,volumesnapshots.snapshot.storage.k8s.io,datauploads.velero.io,persistentvolumes,persistentvolumeclaims,serviceaccounts,secrets,configmaps,limitranges,pods,replicasets.apps,clusterclasses.cluster.x-k8s.io,endpoints,services,-,clusterbootstraps.run.tanzu.vmware.com,clusterresourcesets.addons.cluster.x-k8s.io,clusters.cluster.x-k8s.io - --uploader-type=kopia command: - /velero @@ -168,12 +168,12 @@ status: "True" type: Available - lastTransitionTime: "2024-07-17T14:57:03Z" - lastUpdateTime: "2024-07-17T16:11:03Z" - message: ReplicaSet "velero-77d6b5fcf5" has successfully progressed. + lastUpdateTime: "2024-07-17T17:50:29Z" + message: ReplicaSet "velero-5c88755855" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - observedGeneration: 2 + observedGeneration: 3 readyReplicas: 1 replicas: 1 updatedReplicas: 1 ``` ### Out of Sync DPA w/ unsupported args config map. * DPA uploader type: kopia * Set the config map to uploader type: restic * reconciles, updates velero deployment to `- --uploader-type=restic` * DPA remains set to: ``` configuration: nodeAgent: enable: true uploaderType: kopia ``` * This is expected but needs to be CALLED out in developer docs * https://github.com/openshift/oadp-operator/blob/54aad556507aee70f3c7647afe24021d458d1b23/docs/developer/unsupported_server_args.md ### oadp-must-gather * In the current must-gather for OADP-1.3+ testing if the custom config maps are there... * YES * In dir `abd05fc5/namespaces/openshift-adp/core/configmaps.yaml` * Probably doc this for developer debug if needed. ## Change the name of the configmap in the dpa annotation * Velero is not restarted * Correctly get reconcile errors: ``` 1.7212403003205552e+09 ERROR Reconciler error {"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "3930ac76-ccd8-489f-89c0-f9a02182fe82", "error": "ConfigMap \"foo-config-velero-server\" not found"} sigs.k8s.io/controller-runtime/pkg/inter ``` * Changing the configmap name back to the proper name, Does NOT work ``` 1.7212404696502051e+09 ERROR Reconciler error {"controller": "dataprotectionapplication", "controllerGroup": "oadp.openshift.io", "controllerKind": "DataProtectionApplication", "dataProtectionApplication": {"name":"dpa-sample","namespace":"openshift-adp"}, "namespace": "openshift-adp", "name": "dpa-sample", "reconcileID": "9b7e1032-6aba-45e5-b41f-26444faca474", "error": "Operation cannot be fulfilled on dataprotectionapplications.oadp.openshift.io \"dpa-sample\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:273 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.12.2/pkg/internal/controller/controller.go:234 ``` ## Add invalid key/value pairs to the configmap * Add invalid key pair ``` apiVersion: v1 6 data: 7 disable-informer-cache: "false" 8 features: EnableCSI 9 fs-backup-timeout: 5h 10 joe: Shmoe ``` * configmap is updated, reconciled * Velero deployment correctly updated ``` containers: - args: - server - --disable-informer-cache=false - --features=EnableCSI - --fs-backup-timeout=5h - --joe=Shmoe ``` * Velero correctly goes into crashloopbackoff * Remove the invalid key/value pair from the configmap.. * rereconciles * velero restarts correctly :) *