Kopia Pull Request and Testing

Kopia repo configurations ## official documentation kopia client: https://kopia.io/docs/reference/command-line/ kopia common commands: https://kopia.io/docs/reference/command-line/common/ kopia advanced commands: https://kopia.io/docs/reference/command-line/advanced/ ### Executing kopia from Openshift Create a pod that pulls the oadp-1.3+ must-gather image ``` apiVersion: v1 kind: Pod metadata: name: oadp-mustgather-pod labels: purpose: user-interaction spec: containers: - name: oadp-mustgather-container image: registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.3 command: ["sleep"] args: ["infinity"] ``` * NOTE, for kopia to to connect with defaults the SCC must be 'anyuid' ``` oc describe pod/oadp-mustgather-pod | grep scc openshift.io/scc: anyuid ``` * Alternatively one may set the cache directory to /tmp. See: https://kopia.io/docs/advanced/caching/ Now rsh into the running container :) ``` oc -n openshift-adp rsh pod/oadp-mustgather-pod sh-5.1# which kopia /usr/bin/kopia sh-5.1# kopia --help usage: kopia [<flags>] <command> [<args> ...] Kopia - Fast And Secure Open-Source Backup Flags: --[no-]help Show context-sensitive help (also try --help-long and --help-man). --[no-]version Show application version. --log-file=LOG-FILE Override log file. --log-dir="/root/.cache/kopia" Directory where log files should be written. ($KOPIA_LOG_DIR) ``` ### Connecting to kopia repository using `kopia` CLI Once the repository is created by Velero it's possible to connect to it and run number of benchmarks ```shell= export S3_BUCKET=<your bucket name> export S3_REPOSITORY_PATH=<path without S3_BUCKET> #rquires trailing slash / export S3_ACCESS_KEY=<s3 access key> export S3_SECRET_ACCESS_KEY=<s3 secret access key> # Use static-passw0rd as it is hardcoded kopia repository connect s3 \ --bucket="$S3_BUCKET" \ --prefix="$S3_REPOSITORY_PATH" \ --access-key="$S3_ACCESS_KEY" \ --secret-access-key="$S3_SECRET_ACCESS_KEY" \ --password=static-passw0rd ``` * Example: ``` kopia repository connect s3 --bucket="$S3_BUCKET" --prefix="$S3_REPOSITORY_PATH" --access-key="$S3_ACCESS_KEY" --secret-access-key="$S3_SECRET_ACCESS_KEY" --password=static-passw0rd ``` ``` kopia repository connect s3 --bucket cvpbucket2 --prefix "velero/kopia/mysql-persistent/" --access-key=AKIAV --secret-access-key=dfPi --password="static-passw0rd" ``` Once connected get information about kopia repository ```shell= kopia repository status ``` And run number of benchmarks for hashing, encryption and splitter algorithm ```shell= kopia benchmark hashing kopia benchmark encryption kopia benchmark splitter ``` ## Examples Example that presents repository algorithms ```shell= $ kopia repository status [...] Hash: BLAKE2B-256-128 Encryption: AES256-GCM-HMAC-SHA256 Splitter: DYNAMIC-4M-BUZHASH [...] ``` Example for hashing algorithm, you can see that for this machine fastest is ```--block-hash=BLAKE3-256```, which should be 367.5% faster then the one used for repo configuration. ```shell= $ kopia benchmark hashing Benchmarking hash 'BLAKE2B-256' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'BLAKE2B-256-128' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'BLAKE2S-128' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'BLAKE2S-256' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'BLAKE3-256' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'BLAKE3-256-128' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'HMAC-SHA224' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'HMAC-SHA256' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'HMAC-SHA256-128' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'HMAC-SHA3-224' (100 x 1048576 bytes, parallelism 1) Benchmarking hash 'HMAC-SHA3-256' (100 x 1048576 bytes, parallelism 1) Hash Throughput ----------------------------------------------------------------- 0. BLAKE3-256 14.7 GB / second 1. BLAKE3-256-128 14.3 GB / second 2. HMAC-SHA256-128 6.2 GB / second 3. HMAC-SHA224 6.2 GB / second 4. HMAC-SHA256 6 GB / second 5. BLAKE2B-256-128 4 GB / second 6. BLAKE2B-256 3.9 GB / second 7. BLAKE2S-256 2.8 GB / second 8. BLAKE2S-128 2.7 GB / second 9. HMAC-SHA3-224 1.5 GB / second 10. HMAC-SHA3-256 1.4 GB / second ----------------------------------------------------------------- Fastest option for this machine is: --block-hash=BLAKE3-256 ``` Example for splitting algorithm, fastest for this machine is ```--object-splitter=FIXED-4M``` ***NOTE***: Using fixed splitter will require entire file to be re-uploaded if one byte is changed, which may be not ideal for VM. ```shell= $ kopia benchmark splitter [...] 0. FIXED-4M 393.6 TB/s count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304 1. FIXED-2M 146.4 TB/s count:256 min:2097152 10th:2097152 25th:2097152 50th:2097152 75th:2097152 90th:2097152 max:2097152 2. FIXED-1M 125.6 TB/s count:512 min:1048576 10th:1048576 25th:1048576 50th:1048576 75th:1048576 90th:1048576 max:1048576 3. FIXED-8M 88.3 TB/s count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608 4. FIXED-512K 44.4 TB/s count:1024 min:524288 10th:524288 25th:524288 50th:524288 75th:524288 90th:524288 max:524288 5. FIXED 25.7 TB/s count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304 6. FIXED-256K 20 TB/s count:2048 min:262144 10th:262144 25th:262144 50th:262144 75th:262144 90th:262144 max:262144 7. FIXED-128K 11.2 TB/s count:4096 min:131072 10th:131072 25th:131072 50th:131072 75th:131072 90th:131072 max:131072 8. DYNAMIC-1M-BUZHASH 722.8 MB/s count:428 min:9467 10th:612999 25th:766808 50th:1158068 75th:1744194 90th:2097152 max:2097152 9. DYNAMIC 703.3 MB/s count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608 10. DYNAMIC-128K-BUZHASH 692.1 MB/s count:3183 min:3076 10th:80896 25th:104312 50th:157621 75th:249115 90th:262144 max:262144 11. DYNAMIC-256K-BUZHASH 670.1 MB/s count:1612 min:9467 10th:155695 25th:206410 50th:315497 75th:493851 90th:524288 max:524288 12. DYNAMIC-512K-BUZHASH 668.4 MB/s count:812 min:9467 10th:313948 25th:400696 50th:619560 75th:978932 90th:1048576 max:1048576 13. DYNAMIC-4M-BUZHASH 657.6 MB/s count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608 14. DYNAMIC-2M-BUZHASH 656 MB/s count:204 min:64697 10th:1210184 25th:1638276 50th:2585985 75th:3944217 90th:4194304 max:4194304 15. DYNAMIC-8M-BUZHASH 532.2 MB/s count:49 min:677680 10th:5260579 25th:6528562 50th:11102775 75th:16777216 90th:16777216 max:16777216 16. DYNAMIC-128K-RABINKARP 136.8 MB/s count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144 17. DYNAMIC-8M-RABINKARP 136.1 MB/s count:60 min:1446246 10th:4337385 25th:5293196 50th:8419217 75th:12334953 90th:16777216 max:16777216 18. DYNAMIC-4M-RABINKARP 135 MB/s count:110 min:535925 10th:2242307 25th:2767610 50th:4400962 75th:6813401 90th:8388608 max:8388608 19. DYNAMIC-2M-RABINKARP 134 MB/s count:204 min:535925 10th:1235674 25th:1675441 50th:2525341 75th:3658905 90th:4194304 max:4194304 20. DYNAMIC-256K-RABINKARP 132.8 MB/s count:1575 min:49480 10th:159059 25th:212920 50th:316420 75th:508831 90th:524288 max:524288 21. DYNAMIC-1M-RABINKARP 132.4 MB/s count:391 min:213638 10th:623062 25th:813953 50th:1328673 75th:2097152 90th:2097152 max:2097152 22. DYNAMIC-512K-RABINKARP 129.4 MB/s count:777 min:124727 10th:322811 25th:431615 50th:643713 75th:1048576 90th:1048576 max:1048576 ``` Example for encyption algorithm, you can see that for this machine fastest is ```--encryption=AES256-GCM-HMAC-SHA256```, however difference is marginal ```shell= $ kopia benchmark encryption Benchmarking encryption 'AES256-GCM-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1) Benchmarking encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1) Encryption Throughput ----------------------------------------------------------------- 0. AES256-GCM-HMAC-SHA256 1.3 GB / second 1. CHACHA20-POLY1305-HMAC-SHA256 1.1 GB / second ----------------------------------------------------------------- Fastest option for this machine is: --encryption=AES256-GCM-HMAC-SHA256 ``` # Pull Request and Testing ## Pull Request Downstream only pull request that will be carried over to allow easilly switch between different encryption/hashing/splitting algorithms is available at https://github.com/openshift/velero/pull/334 ## Container image The Velero image built with that pull request is avalable at: quay.io/migi/velero:testing_args_env ## Testing ### NOTE For different algorithms it's mandatory to clean up s3 storage or use different prefix in DPA ### DPA Configuration - Example using `mprycoadp` S3 bucket from AWS within `eu-central-1` region. CLI `aws` is installed on the system and `~/.aws/credentials` files contains: ```shell= [default] aws_access_key_id = aws_secret_access_key = ``` 1. Ensure credentials used in the DPA `hashing-testing-credentials` for s3 are within `openshift-adp` namespace: ```shell= oc create secret generic hashing-testing-credentials --namespace openshift-adp --from-file cloud=~/.aws/credentials ``` DPA requires two additions to change the algorithm(s): - `veleroImageFqin` option in the unsupportedOverrides - `env` variables in the `podConfig` under `velero` config ```yaml= apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: dpa-name namespace: openshift-adp spec: unsupportedOverrides: veleroImageFqin: 'quay.io/migi/velero:testing_args_env' configuration: configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultSnapshotMoveData: true podConfig: env: - name: KOPIA_HASHING_ALGORITHM value: BLAKE3-256 [...] ``` ### Case 1: Create backup with different Hashing algorithm 1. DPA: ```yaml= apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: velero-sample namespace: openshift-adp spec: unsupportedOverrides: veleroImageFqin: 'quay.io/migi/velero:testing_args_env' configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultSnapshotMoveData: true podConfig: env: - name: KOPIA_HASHING_ALGORITHM value: BLAKE3-256 defaultPlugins: - openshift - aws - csi backupLocations: - velero: config: profile: default region: eu-central-1 credential: key: cloud name: hashing-testing-credentials objectStorage: bucket: mprycoadp prefix: hashingalgorithm default: true provider: aws ``` 2. Created Sample Deployment ```shell= # oadp-operator/tests/e2e/sample-applications/mysql-persistent/ oc apply -f mysql-persistent-csi.yaml -f pvc/default_sc.yaml ``` 3. Ensure s3 backup does not contain any files ```shell= aws s3 rm --recursive s3://mprycoadp/ ``` 4. Create backup `backup.yaml`: ```shell= apiVersion: velero.io/v1 kind: Backup metadata: name: mysql-blake-backup namespace: openshift-adp spec: includedNamespaces: - mysql-persistent storageLocation: velero-sample-1 ``` ```shell= oc apply -f backup.yaml ``` 5. After backup is completed check if kopia repository is on the s3 storage: ```shell= $ aws s3 ls s3://mprycoadp/hashingalgorithm/kopia/mysql-persistent/ 2024-08-07 15:32:26 823 _log_20240807133225_e00c_1723037545_1723037545_1_2f48473a4298fe7dea7850ffedc108124f34419b1b8ccec5965aa40edff6b0f5 2024-08-07 15:32:29 6964 _log_20240807133227_86fa_1723037547_1723037548_1_a12c6af84af9292a5d6a02a02464527a2fac379d5895007e178b486ae0e2d53e 2024-08-07 15:32:25 30 kopia.blobcfg 2024-08-07 15:32:25 1067 kopia.repository 2024-08-07 15:32:29 15880546 p6377a65f1655edfdd80b60caacc3019e-s15455f66b99032a912b 2024-08-07 15:32:28 4330 q396b0a69c58babb34d619a499a2fe69a-s8cc2323062d9fce512b 2024-08-07 15:32:28 13029 q525f125201906e889a62e2a9fa82fc3b-s15455f66b99032a912b 2024-08-07 15:32:26 4330 q995947ba192cc7ea19f9f19d1a9ffa4a-s07ce71aca3045a9312b 2024-08-07 15:32:28 159 xn0_2328a9b7a5319dad95591260f1167755372c2d55cdd02d193a9b1a41f1678b91-s8cc2323062d9fce512b-c1 2024-08-07 15:32:26 159 xn0_6e68373823547b16d1884aafb6a0de131c76a414d8f888a5bdb05ee4b6ef32d2-s07ce71aca3045a9312b-c1 2024-08-07 15:32:29 5075 xn0_a67b12c21f4ce344aebecfdd60e78acfd39e2e12bced167386dda0d378094961-s15455f66b99032a912b-c1 ``` 6. Connect your temp directory to kopia repository, for that you need to install `kopia` CLI on your system and confirm that Hashing algorithm is one from the DPA ```shell= $ export AWS_SECRET_ACCESS_KEY= $ export AWS_ACCES_KEY= $ kopia repository connect s3 \ --bucket=mprycoadp \ --prefix=hashingalgorithm/kopia/mysql-persistent/ \ --password=static-passw0rd \ --access-key="${AWS_ACCES_KEY}" \ --secret-access-key="${SECRET_AWS_ACCES_KEY}" $ kopia repo status [...] Hash: BLAKE3-256 Encryption: AES256-GCM-HMAC-SHA256 Splitter: DYNAMIC-4M-BUZHASH [...] ``` 7. Restore, delete `mysql-persistent` and create `Restore` object on the cluster. ```shell= $ oc delete project mysql-persistent project.project.openshift.io "mysql-persistent" deleted ``` File `restore.yaml`: ```yaml= apiVersion: velero.io/v1 kind: Restore metadata: name: mysql-persistent namespace: openshift-adp spec: backupName: mysql-blake-backup restorePVs: true ``` ```shell= oc apply -f restore.yaml ``` Check if the application was restored ```shell= oc describe restore mysql-persistent -n openshift-adp oc get all -n mysql-persistent ``` >> SUCCESS with restore ### Case 2: Check if it's possible to restore from different hashing 1. DPA: ```yaml= apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: velero-sample namespace: openshift-adp spec: unsupportedOverrides: veleroImageFqin: 'quay.io/migi/velero:testing_args_env' configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultSnapshotMoveData: true podConfig: env: - name: KOPIA_HASHING_ALGORITHM value: BLAKE3-256 defaultPlugins: - openshift - aws - csi backupLocations: - velero: config: profile: default region: eu-central-1 credential: key: cloud name: hashing-testing-credentials objectStorage: bucket: mprycoadp prefix: hashingalgorithm default: true provider: aws ``` 2. Delete previous restore ```shell= oc delete restore mysql-persistent -n openshift-adp ``` 3. Re-use previously created `restore.yaml` restore file ```shell= oc apply -f restore.yaml ``` 4. Confirm restore is success ```shell= oc describe restore mysql-persistent -n openshift-adp oc get all -n mysql-persistent ``` >> SUCCESS with restore ### Case 3: Similar to Case 1, but with different splitting, hashing and encyprion algorithms 1. DPA, we will use different s3 prefix: ```yaml= apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: velero-sample namespace: openshift-adp spec: unsupportedOverrides: veleroImageFqin: 'quay.io/migi/velero:testing_args_env' configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultSnapshotMoveData: true podConfig: env: - name: KOPIA_HASHING_ALGORITHM value: BLAKE3-256 - name: KOPIA_ENCRYPTION_ALGORITHM value: CHACHA20-POLY1305-HMAC-SHA256 - name: KOPIA_SPLITTER_ALGORITHM value: DYNAMIC-8M-RABINKARP defaultPlugins: - openshift - aws - csi backupLocations: - velero: config: profile: default region: eu-central-1 credential: key: cloud name: hashing-testing-credentials objectStorage: bucket: mprycoadp prefix: allalgorithms default: true provider: aws ``` 2. Remove previous dpa and apply above one 3. Create new `backup-new.yaml` and apply it ``` apiVersion: velero.io/v1 kind: Backup metadata: name: mysql-allalgorithms-backup namespace: openshift-adp spec: includedNamespaces: - mysql-persistent storageLocation: velero-sample-1 ``` ```shell= oc apply -f backup-new.yaml oc describe backup mysql-allalgorithms-backup -n openshift-adp ``` 4. Connect to the repository ```shell= $ kopia repository connect s3 \ --bucket=mprycoadp \ --prefix=allalgorithms/kopia/mysql-persistent/ \ --password=static-passw0rd \ --access-key="${AWS_ACCES_KEY}" \ --secret-access-key="${SECRET_AWS_ACCES_KEY}" $ kopia status [...] Hash: BLAKE3-256 Encryption: CHACHA20-POLY1305-HMAC-SHA256 Splitter: DYNAMIC-8M-RABINKARP [...] ``` 5. Restore, delete `mysql-persistent` and create `Restore` object on the cluster. ```shell= $ oc delete project mysql-persistent project.project.openshift.io "mysql-persistent" deleted ``` File `restore.yaml`: ```yaml= apiVersion: velero.io/v1 kind: Restore metadata: name: mysql-persistent namespace: openshift-adp spec: backupName: mysql-all-backup restorePVs: true ``` ```shell= oc apply -f restore.yaml ``` >> SUCCESS with restore ### Case 4: Similar to Case 2, but restore from backup that used w different splitting, hashing and encyprion algorithms 1. Use the dpa: ```yaml= apiVersion: oadp.openshift.io/v1alpha1 kind: DataProtectionApplication metadata: name: velero-sample namespace: openshift-adp spec: unsupportedOverrides: veleroImageFqin: 'quay.io/migi/velero:testing_args_env' configuration: nodeAgent: enable: true uploaderType: kopia velero: defaultSnapshotMoveData: true defaultPlugins: - openshift - aws - csi backupLocations: - velero: config: profile: default region: eu-central-1 credential: key: cloud name: hashing-testing-credentials objectStorage: bucket: mprycoadp prefix: allalgorithms default: true provider: aws ``` >> SUCCESS with restore ### Case 5: Create second backup with different algorithms in the DPA 1. After restore from Case 4, which was using default algorithms in velero pod: ``` apiVersion: velero.io/v1 kind: Backup metadata: name: mysql-all-backup-second namespace: openshift-adp spec: includedNamespaces: - mysql-persistent storageLocation: velero-sample-1 ``` ```shell= $ oc describe backup mysql-all-backup-second ``` 2. We can see that initial repo encryption/split/hash is preserved ```shell= $ kopia repo status [...] Hash: BLAKE3-256 Encryption: CHACHA20-POLY1305-HMAC-SHA256 Splitter: DYNAMIC-8M-RABINKARP [...] ``` >> Backup succeeded, previous algorhtms are preserved