Kopia repo configurations

official documentation

kopia client: https://kopia.io/docs/reference/command-line/
kopia common commands: https://kopia.io/docs/reference/command-line/common/
kopia advanced commands: https://kopia.io/docs/reference/command-line/advanced/

Executing kopia from Openshift

Create a pod that pulls the oadp-1.3+ must-gather image

apiVersion: v1
kind: Pod
metadata:
  name: oadp-mustgather-pod
  labels:
    purpose: user-interaction
spec:
  containers:
  - name: oadp-mustgather-container
    image: registry.redhat.io/oadp/oadp-mustgather-rhel9:v1.3
    command: ["sleep"]
    args: ["infinity"]
  • NOTE, for kopia to to connect with defaults the SCC must be 'anyuid'
oc describe pod/oadp-mustgather-pod | grep scc
                  openshift.io/scc: anyuid

Now rsh into the running container :)

oc -n openshift-adp rsh pod/oadp-mustgather-pod

sh-5.1# which kopia
/usr/bin/kopia
sh-5.1# kopia --help

usage: kopia [<flags>] <command> [<args> ...]
Kopia - Fast And Secure Open-Source Backup

Flags:
      --[no-]help             Show context-sensitive help (also try --help-long and --help-man).
      --[no-]version          Show application version.
      --log-file=LOG-FILE     Override log file.
      --log-dir="/root/.cache/kopia"  
                              Directory where log files should be written. ($KOPIA_LOG_DIR)

Connecting to kopia repository using kopia CLI

Once the repository is created by Velero it's possible to connect to it and run number of benchmarks














export S3_BUCKET=<your bucket name>
export S3_REPOSITORY_PATH=<path without S3_BUCKET> #rquires trailing slash /
export S3_ACCESS_KEY=<s3 access key>
export S3_SECRET_ACCESS_KEY=<s3 secret access key>

# Use static-passw0rd as it is hardcoded

kopia repository connect s3 \
    --bucket="$S3_BUCKET" \
    --prefix="$S3_REPOSITORY_PATH" \
    --access-key="$S3_ACCESS_KEY" \
    --secret-access-key="$S3_SECRET_ACCESS_KEY" \
    --password=static-passw0rd
  • Example:
kopia repository connect s3 --bucket="$S3_BUCKET" --prefix="$S3_REPOSITORY_PATH" --access-key="$S3_ACCESS_KEY" --secret-access-key="$S3_SECRET_ACCESS_KEY" --password=static-passw0rd

kopia repository connect s3 --bucket cvpbucket2 --prefix "velero/kopia/mysql-persistent/" --access-key=AKIAV --secret-access-key=dfPi --password="static-passw0rd"

Once connected get information about kopia repository

kopia repository status

And run number of benchmarks for hashing, encryption and splitter algorithm




kopia benchmark hashing
kopia benchmark encryption
kopia benchmark splitter

Examples

Example that presents repository algorithms







$ kopia repository status
[...]
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
[...]

Example for hashing algorithm, you can see that for this machine fastest is --block-hash=BLAKE3-256, which should be 367.5% faster then the one used for repo configuration.




























$ kopia benchmark hashing
Benchmarking hash 'BLAKE2B-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2B-256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE2S-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'BLAKE3-256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA224' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA256-128' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-224' (100 x 1048576 bytes, parallelism 1)
Benchmarking hash 'HMAC-SHA3-256' (100 x 1048576 bytes, parallelism 1)
     Hash                 Throughput
-----------------------------------------------------------------
  0. BLAKE3-256           14.7 GB / second
  1. BLAKE3-256-128       14.3 GB / second
  2. HMAC-SHA256-128      6.2 GB / second
  3. HMAC-SHA224          6.2 GB / second
  4. HMAC-SHA256          6 GB / second
  5. BLAKE2B-256-128      4 GB / second
  6. BLAKE2B-256          3.9 GB / second
  7. BLAKE2S-256          2.8 GB / second
  8. BLAKE2S-128          2.7 GB / second
  9. HMAC-SHA3-224        1.5 GB / second
 10. HMAC-SHA3-256        1.4 GB / second
-----------------------------------------------------------------
Fastest option for this machine is: --block-hash=BLAKE3-256

Example for splitting algorithm, fastest for this machine is --object-splitter=FIXED-4M
NOTE: Using fixed splitter will require entire file to be re-uploaded if one byte is changed, which may be not ideal for VM.


























$ kopia benchmark splitter
[...]
  0. FIXED-4M                  393.6 TB/s   count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304
  1. FIXED-2M                  146.4 TB/s   count:256 min:2097152 10th:2097152 25th:2097152 50th:2097152 75th:2097152 90th:2097152 max:2097152
  2. FIXED-1M                  125.6 TB/s   count:512 min:1048576 10th:1048576 25th:1048576 50th:1048576 75th:1048576 90th:1048576 max:1048576
  3. FIXED-8M                  88.3 TB/s    count:64 min:8388608 10th:8388608 25th:8388608 50th:8388608 75th:8388608 90th:8388608 max:8388608
  4. FIXED-512K                44.4 TB/s    count:1024 min:524288 10th:524288 25th:524288 50th:524288 75th:524288 90th:524288 max:524288
  5. FIXED                     25.7 TB/s    count:128 min:4194304 10th:4194304 25th:4194304 50th:4194304 75th:4194304 90th:4194304 max:4194304
  6. FIXED-256K                20 TB/s      count:2048 min:262144 10th:262144 25th:262144 50th:262144 75th:262144 90th:262144 max:262144
  7. FIXED-128K                11.2 TB/s    count:4096 min:131072 10th:131072 25th:131072 50th:131072 75th:131072 90th:131072 max:131072
  8. DYNAMIC-1M-BUZHASH        722.8 MB/s   count:428 min:9467 10th:612999 25th:766808 50th:1158068 75th:1744194 90th:2097152 max:2097152
  9. DYNAMIC                   703.3 MB/s   count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608
 10. DYNAMIC-128K-BUZHASH      692.1 MB/s   count:3183 min:3076 10th:80896 25th:104312 50th:157621 75th:249115 90th:262144 max:262144
 11. DYNAMIC-256K-BUZHASH      670.1 MB/s   count:1612 min:9467 10th:155695 25th:206410 50th:315497 75th:493851 90th:524288 max:524288
 12. DYNAMIC-512K-BUZHASH      668.4 MB/s   count:812 min:9467 10th:313948 25th:400696 50th:619560 75th:978932 90th:1048576 max:1048576
 13. DYNAMIC-4M-BUZHASH        657.6 MB/s   count:107 min:9467 10th:2277562 25th:2971794 50th:4747177 75th:7603998 90th:8388608 max:8388608
 14. DYNAMIC-2M-BUZHASH        656 MB/s     count:204 min:64697 10th:1210184 25th:1638276 50th:2585985 75th:3944217 90th:4194304 max:4194304
 15. DYNAMIC-8M-BUZHASH        532.2 MB/s   count:49 min:677680 10th:5260579 25th:6528562 50th:11102775 75th:16777216 90th:16777216 max:16777216
 16. DYNAMIC-128K-RABINKARP    136.8 MB/s   count:3160 min:9667 10th:80098 25th:106626 50th:162269 75th:250655 90th:262144 max:262144
 17. DYNAMIC-8M-RABINKARP      136.1 MB/s   count:60 min:1446246 10th:4337385 25th:5293196 50th:8419217 75th:12334953 90th:16777216 max:16777216
 18. DYNAMIC-4M-RABINKARP      135 MB/s     count:110 min:535925 10th:2242307 25th:2767610 50th:4400962 75th:6813401 90th:8388608 max:8388608
 19. DYNAMIC-2M-RABINKARP      134 MB/s     count:204 min:535925 10th:1235674 25th:1675441 50th:2525341 75th:3658905 90th:4194304 max:4194304
 20. DYNAMIC-256K-RABINKARP    132.8 MB/s   count:1575 min:49480 10th:159059 25th:212920 50th:316420 75th:508831 90th:524288 max:524288
 21. DYNAMIC-1M-RABINKARP      132.4 MB/s   count:391 min:213638 10th:623062 25th:813953 50th:1328673 75th:2097152 90th:2097152 max:2097152
 22. DYNAMIC-512K-RABINKARP    129.4 MB/s   count:777 min:124727 10th:322811 25th:431615 50th:643713 75th:1048576 90th:1048576 max:1048576

Example for encyption algorithm, you can see that for this machine fastest is --encryption=AES256-GCM-HMAC-SHA256, however difference is marginal











$ kopia benchmark encryption
Benchmarking encryption 'AES256-GCM-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
Benchmarking encryption 'CHACHA20-POLY1305-HMAC-SHA256'... (1000 x 1048576 bytes, parallelism 1)
     Encryption                     Throughput
-----------------------------------------------------------------
  0. AES256-GCM-HMAC-SHA256         1.3 GB / second
  1. CHACHA20-POLY1305-HMAC-SHA256  1.1 GB / second
-----------------------------------------------------------------
Fastest option for this machine is: --encryption=AES256-GCM-HMAC-SHA256

Pull Request and Testing

Pull Request

Downstream only pull request that will be carried over to allow easilly switch between different encryption/hashing/splitting algorithms is available at https://github.com/openshift/velero/pull/334

Container image

The Velero image built with that pull request is avalable at:
quay.io/migi/velero:testing_args_env

Testing

NOTE

For different algorithms it's mandatory to clean up s3 storage or use different prefix in DPA

DPA Configuration

  • Example using mprycoadp S3 bucket from AWS within eu-central-1 region. CLI aws is installed on the system and ~/.aws/credentials files contains:



[default]
aws_access_key_id =
aws_secret_access_key =
  1. Ensure credentials used in the DPA hashing-testing-credentials for s3 are within openshift-adp namespace:
oc create secret generic hashing-testing-credentials --namespace openshift-adp --from-file cloud=~/.aws/credentials

DPA requires two additions to change the algorithm(s):

  • veleroImageFqin option in the unsupportedOverrides
  • env variables in the podConfig under velero config




















apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: dpa-name
  namespace: openshift-adp
spec:
  unsupportedOverrides:
    veleroImageFqin: 'quay.io/migi/velero:testing_args_env'
  configuration:
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultSnapshotMoveData: true
      podConfig:
        env:
          - name: KOPIA_HASHING_ALGORITHM
            value: BLAKE3-256
[...]

Case 1: Create backup with different Hashing algorithm

  1. DPA:



































apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  unsupportedOverrides:
    veleroImageFqin: 'quay.io/migi/velero:testing_args_env'
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultSnapshotMoveData: true
      podConfig:
        env:
          - name: KOPIA_HASHING_ALGORITHM
            value: BLAKE3-256
      defaultPlugins:
        - openshift
        - aws
        - csi
  backupLocations:
    - velero:
        config:
          profile: default
          region: eu-central-1
        credential:
          key: cloud
          name: hashing-testing-credentials
        objectStorage:
          bucket: mprycoadp
          prefix: hashingalgorithm
        default: true
        provider: aws
  1. Created Sample Deployment


# oadp-operator/tests/e2e/sample-applications/mysql-persistent/
oc apply -f mysql-persistent-csi.yaml -f pvc/default_sc.yaml
  1. Ensure s3 backup does not contain any files
aws s3 rm --recursive s3://mprycoadp/
  1. Create backup backup.yaml:









apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-blake-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - mysql-persistent
  storageLocation: velero-sample-1

oc apply -f backup.yaml
  1. After backup is completed check if kopia repository is on the s3 storage:












$ aws s3 ls s3://mprycoadp/hashingalgorithm/kopia/mysql-persistent/
2024-08-07 15:32:26        823 _log_20240807133225_e00c_1723037545_1723037545_1_2f48473a4298fe7dea7850ffedc108124f34419b1b8ccec5965aa40edff6b0f5
2024-08-07 15:32:29       6964 _log_20240807133227_86fa_1723037547_1723037548_1_a12c6af84af9292a5d6a02a02464527a2fac379d5895007e178b486ae0e2d53e
2024-08-07 15:32:25         30 kopia.blobcfg
2024-08-07 15:32:25       1067 kopia.repository
2024-08-07 15:32:29   15880546 p6377a65f1655edfdd80b60caacc3019e-s15455f66b99032a912b
2024-08-07 15:32:28       4330 q396b0a69c58babb34d619a499a2fe69a-s8cc2323062d9fce512b
2024-08-07 15:32:28      13029 q525f125201906e889a62e2a9fa82fc3b-s15455f66b99032a912b
2024-08-07 15:32:26       4330 q995947ba192cc7ea19f9f19d1a9ffa4a-s07ce71aca3045a9312b
2024-08-07 15:32:28        159 xn0_2328a9b7a5319dad95591260f1167755372c2d55cdd02d193a9b1a41f1678b91-s8cc2323062d9fce512b-c1
2024-08-07 15:32:26        159 xn0_6e68373823547b16d1884aafb6a0de131c76a414d8f888a5bdb05ee4b6ef32d2-s07ce71aca3045a9312b-c1
2024-08-07 15:32:29       5075 xn0_a67b12c21f4ce344aebecfdd60e78acfd39e2e12bced167386dda0d378094961-s15455f66b99032a912b-c1
  1. Connect your temp directory to kopia repository, for that you need to install kopia CLI on your system and confirm that Hashing algorithm is one from the DPA















$ export AWS_SECRET_ACCESS_KEY=
$ export AWS_ACCES_KEY=
$ kopia repository connect s3 \
   --bucket=mprycoadp \
   --prefix=hashingalgorithm/kopia/mysql-persistent/ \
   --password=static-passw0rd \
   --access-key="${AWS_ACCES_KEY}" \
   --secret-access-key="${SECRET_AWS_ACCES_KEY}"

$ kopia repo status
[...]
Hash:                BLAKE3-256
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
[...]
  1. Restore, delete mysql-persistent and create Restore object on the cluster.


$ oc delete project mysql-persistent
project.project.openshift.io "mysql-persistent" deleted

File restore.yaml:









apiVersion: velero.io/v1
kind: Restore
metadata:
  name: mysql-persistent
  namespace: openshift-adp
spec:
  backupName: mysql-blake-backup
  restorePVs: true

oc apply -f restore.yaml

Check if the application was restored



oc describe restore mysql-persistent -n openshift-adp
oc get all -n mysql-persistent

SUCCESS with restore

Case 2: Check if it's possible to restore from different hashing

  1. DPA:



































apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  unsupportedOverrides:
    veleroImageFqin: 'quay.io/migi/velero:testing_args_env'
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultSnapshotMoveData: true
      podConfig:
        env:
          - name: KOPIA_HASHING_ALGORITHM
            value: BLAKE3-256
      defaultPlugins:
        - openshift
        - aws
        - csi
  backupLocations:
    - velero:
        config:
          profile: default
          region: eu-central-1
        credential:
          key: cloud
          name: hashing-testing-credentials
        objectStorage:
          bucket: mprycoadp
          prefix: hashingalgorithm
        default: true
        provider: aws
  1. Delete previous restore
oc delete restore mysql-persistent -n openshift-adp
  1. Re-use previously created restore.yaml restore file
oc apply -f restore.yaml
  1. Confirm restore is success


oc describe restore mysql-persistent -n openshift-adp
oc get all -n mysql-persistent

SUCCESS with restore

Case 3: Similar to Case 1, but with different splitting, hashing and encyprion algorithms

  1. DPA, we will use different s3 prefix:







































apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  unsupportedOverrides:
    veleroImageFqin: 'quay.io/migi/velero:testing_args_env'
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultSnapshotMoveData: true
      podConfig:
        env:
          - name: KOPIA_HASHING_ALGORITHM
            value: BLAKE3-256
          - name: KOPIA_ENCRYPTION_ALGORITHM
            value: CHACHA20-POLY1305-HMAC-SHA256
          - name: KOPIA_SPLITTER_ALGORITHM
            value: DYNAMIC-8M-RABINKARP
      defaultPlugins:
        - openshift
        - aws
        - csi
  backupLocations:
    - velero:
        config:
          profile: default
          region: eu-central-1
        credential:
          key: cloud
          name: hashing-testing-credentials
        objectStorage:
          bucket: mprycoadp
          prefix: allalgorithms
        default: true
        provider: aws
  1. Remove previous dpa and apply above one
  2. Create new backup-new.yaml and apply it
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-allalgorithms-backup
  namespace: openshift-adp
spec:
  includedNamespaces:
    - mysql-persistent
  storageLocation: velero-sample-1



oc apply -f backup-new.yaml
oc describe backup mysql-allalgorithms-backup -n openshift-adp
  1. Connect to the repository













$ kopia repository connect s3 \
   --bucket=mprycoadp \
   --prefix=allalgorithms/kopia/mysql-persistent/ \
   --password=static-passw0rd \
   --access-key="${AWS_ACCES_KEY}" \
   --secret-access-key="${SECRET_AWS_ACCES_KEY}"

$ kopia status
[...]
Hash:                BLAKE3-256
Encryption:          CHACHA20-POLY1305-HMAC-SHA256
Splitter:            DYNAMIC-8M-RABINKARP
[...]
  1. Restore, delete mysql-persistent and create Restore object on the cluster.


$ oc delete project mysql-persistent
project.project.openshift.io "mysql-persistent" deleted

File restore.yaml:









apiVersion: velero.io/v1
kind: Restore
metadata:
  name: mysql-persistent
  namespace: openshift-adp
spec:
  backupName: mysql-all-backup
  restorePVs: true

oc apply -f restore.yaml

SUCCESS with restore

Case 4: Similar to Case 2, but restore from backup that used w different splitting, hashing and encyprion algorithms

  1. Use the dpa:































apiVersion: oadp.openshift.io/v1alpha1
kind: DataProtectionApplication
metadata:
  name: velero-sample
  namespace: openshift-adp
spec:
  unsupportedOverrides:
    veleroImageFqin: 'quay.io/migi/velero:testing_args_env'
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      defaultSnapshotMoveData: true
      defaultPlugins:
        - openshift
        - aws
        - csi
  backupLocations:
    - velero:
        config:
          profile: default
          region: eu-central-1
        credential:
          key: cloud
          name: hashing-testing-credentials
        objectStorage:
          bucket: mprycoadp
          prefix: allalgorithms
        default: true
        provider: aws

SUCCESS with restore

Case 5: Create second backup with different algorithms in the DPA

  1. After restore from Case 4, which was using default algorithms in velero pod:
apiVersion: velero.io/v1
kind: Backup
metadata:
  name: mysql-all-backup-second
  namespace: openshift-adp
spec:
  includedNamespaces:
    - mysql-persistent
  storageLocation: velero-sample-1

$ oc describe backup mysql-all-backup-second
  1. We can see that initial repo encryption/split/hash is preserved






$ kopia repo status
[...]
Hash:                BLAKE3-256
Encryption:          CHACHA20-POLY1305-HMAC-SHA256
Splitter:            DYNAMIC-8M-RABINKARP
[...]

Backup succeeded, previous algorhtms are preserved

Select a repo