---
# System prepended metadata

title: Kopia Repo Maint
tags: [kopia]

---

# kopia repo maint notes

First pass at notes:

## Upstream doc:
* https://velero.io/docs/v1.14/repository-maintenance/
* kopia blob_gc_test.go https://github.com/kopia/kopia/blob/master/repo/maintenance/blob_gc_test.go

## customer info
https://docs.google.com/document/d/1zpyL_sux7H2p50rH0MBdo_JxppBuACVAnEvPvLBEvsA/edit?tab=t.0



## OADP dpa.spec

```
  default-repo-maintain-frequency	<integer>
    How often (in nanoseconds) 'maintain' is run for backup repositories by
    default.

```

```
oc explain dpa.spec.configuration.velero.args.default-repo-maintain-frequency
GROUP:      oadp.openshift.io
KIND:       DataProtectionApplication
VERSION:    v1alpha1

FIELD: default-repo-maintain-frequency <integer>

DESCRIPTION:
    How often (in nanoseconds) 'maintain' is run for backup repositories by
    default.
    

```

## DPA
```
spec:
  backupLocations:
    - velero:
        config:
          profile: default
          region: us-west-2
        credential:
          key: cloud
          name: cloud-credentials
        default: true
        objectStorage:
          bucket: cvpbucketuswest2
          prefix: velero
        provider: aws
  configuration:
    nodeAgent:
      enable: true
      uploaderType: kopia
    velero:
      args:
        default-repo-maintain-frequency: 360
      defaultPlugins:
        - kubevirt
        - csi
        - openshift
        - aws
  snapshotLocations:
    - velero:
        config:
          profile: default
          region: us-west-2
        provider: aws
status:
  conditions:
    - lastTransitionTime: '2024-10-22T19:27:00Z'
      message: Reconcile complete
      reason: Complete
      status: 'True'
      type: Reconciled
```







### Wes's test notes
* start w/ a clean bucket
```
aws s3 ls --summarize  --recursive --human-readable  s3://cvpbucketuswest2/

Total Objects: 0
   Total Size: 0 Bytes
```

* first volley
```
velero backup create vol24-1 --include-namespaces minimal-24csivol --snapshot-move-data=true
```

* repo maint jobs started.. seem to be running FULL maint
  * oc logs pod/repo-maintain-job-1729626671302-zbdsq
  * https://termbin.com/079j

```
time="2024-10-22T19:51:12Z" level=warning msg="active indexes [xn0_41d7e4c001635d817cc31ffc4a15f0b7-sca1a20b0845c562c12e-c1 xn0_9e4b7b2bedf4e266102b87e865ec9800-sca19c4675875bd2b12e-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" logger name="[index-blob-manager]" sublevel=error
time="2024-10-22T19:51:12Z" level=info msg="Start to open repo for maintenance, allow index write on load" logSource="pkg/repository/udmrepo/kopialib/lib_repo.go:165"
time="2024-10-22T19:51:12Z" level=warning msg="active indexes [xn0_41d7e4c001635d817cc31ffc4a15f0b7-sca1a20b0845c562c12e-c1 xn0_9e4b7b2bedf4e266102b87e865ec9800-sca19c4675875bd2b12e-c1 xn0_e8b4c3d621246975962f11cea3dc1eb1-sd12627bda768460f12e-c1] deletion watermark 0001-01-01 00:00:00 +0000 UTC" logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:101" logger name="[index-blob-manager]" sublevel=error
time="2024-10-22T19:51:12Z" level=info msg="Succeeded to open repo for maintenance" logSource="pkg/repository/udmrepo/kopialib/lib_repo.go:172"
time="2024-10-22T19:51:12Z" level=info msg="Running full maintenance..." logModule=kopia/maintenance logSource="pkg/kopia/kopia_log.go:94" logger name="[shared-manager]"
time="2024-10-22T19:51:12Z" level=info msg="Running full maintenance..." logModule=kopia/kopia/format logSource="pkg/kopia/kopia_log.go:94" logger name="[shared-manager]"
```
* repomaint logs after backup deleted w/ velero cli
    * https://termbin.com/1oku  - after several repo maint runs.


### create dm backup w/ ttl 1hr 
```
remote_velero backup create vol24-ttl-short --include-namespaces minimal-24csivol --snapshot-move-data=true --ttl 1h0m0s
Backup request "vol24-ttl-short" submitted successfully.

```
```
NAME              STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
vol24-ttl-short   Completed   0        0          2024-10-22 20:19:29 +0000 UTC   35m       dpa-sample-1       <none>

```

```
Total Objects: 331
   Total Size: 4.0 GiB
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$ date
Tue Oct 22 02:40:40 PM MDT 2024

```
* backup was deleted after ttl expired
```
Total Objects: 352
   Total Size: 4.0 GiB
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$ date
Tue Oct 22 05:07:41 PM MDT 2024
```

### connected to repo manually
```
kopia maintenance info
```

```
Owner: default@default
Quick Cycle:
  scheduled: true
  interval: 1h0m0s
  next run: now
Full Cycle:
  scheduled: true
  interval: 24h0m0s
  next run: 2024-10-23 20:21:05 UTC (in 20h57m1s)
Log Retention:
  max count:       10000
  max age of logs: 720h0m0s
  max total size:  1.1 GB
Object Lock Extension: disabled
Recent Maintenance Runs:
  compact-single-epoch:
    2024-10-22 22:22:10 UTC (0s) SUCCESS
    2024-10-22 21:22:10 UTC (0s) SUCCESS
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  delete-superseded-epoch-indexes:
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  full-rewrite-contents:
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  generate-epoch-range-index:
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  snapshot-gc:
    2024-10-22 20:21:05 UTC (0s) SUCCESS
  advance-epoch:
    2024-10-22 22:22:10 UTC (0s) SUCCESS
    2024-10-22 21:22:10 UTC (0s) SUCCESS
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  cleanup-epoch-markers:
    2024-10-22 20:21:06 UTC (0s) SUCCESS
  cleanup-logs:
    2024-10-22 20:21:06 UTC (0s) SUCCESS

```


```
kopia repository status
```
```
Config file:         /root/.config/kopia/repository.config

Description:         Repository in S3: s3.amazonaws.com cvpbucketuswest2
Hostname:            oadp-mustgather-pod
Username:            root
Read-only:           false
Format blob cache:   15m0s

Storage type:        s3
Storage capacity:    unbounded
Storage config:      {
                       "bucket": "cvpbucketuswest2",
                       "prefix": "velero/kopia/minimal-24csivol/",
                       "endpoint": "s3.amazonaws.com",
                       "accessKeyID": "AKIAVBQYB2FD4NQGDQOB",
                       "secretAccessKey": "****************************************",
                       "sessionToken": ""
                     }

Unique ID:           7d847c816937c6b1bc282e0649cf0ae51ca1ec11d3042cb2d5308cbee6b9cfeb
Hash:                BLAKE2B-256-128
Encryption:          AES256-GCM-HMAC-SHA256
Splitter:            DYNAMIC-4M-BUZHASH
Format version:      3
Content compression: true
Password changes:    true
Max pack length:     21 MB
Index Format:        v2

Epoch Manager:       enabled
Current Epoch: 0

Epoch refresh frequency: 20m0s
Epoch advance on:        20 blobs or 10.5 MB, minimum 24h0m0s
Epoch cleanup margin:    4h0m0s
Epoch checkpoint every:  7 epochs
```
### update the maintenance owner and run maint
```
kopia maintenance set --owner=root@oadp-mustgather-pod
```

```
kopia maintenance run --full
```
```
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 0 unused contents (0 B)
GC found 875 unused contents that are too recent to delete (4.3 GB)
GC found 0 in-use contents (0 B)
GC found 54 in-use system-contents (39.2 KB)
Previous content rewrite has not been finalized yet, waiting until the next blob deletion.
Not enough time has passed since previous successful Snapshot GC. Will try again next time.
Looking for unreferenced blobs...
Deleted total 0 unreferenced blobs (0 B)
Cleaned up 0 logs.
Cleaning up old index blobs which have already been compacted...
Finished full maintenance.

```

### turn off the child safety 

```
kopia maintenance run --full --safety=none
```
```
kopia maintenance run --full --safety=none
Running full maintenance...
Looking for active contents...
Looking for unreferenced contents...
GC found 875 unused contents (4.3 GB)
GC found 0 unused contents that are too recent to delete (0 B)
GC found 0 in-use contents (0 B)
GC found 54 in-use system-contents (39.2 KB)
Rewriting contents from short packs...
Total bytes rewritten 13.8 MB
Found safe time to drop indexes: 2024-10-22 23:33:34.799396449 +0000 UTC
Dropping contents deleted before 2024-10-22 23:33:34.799396449 +0000 UTC
Looking for unreferenced blobs...
  deleted 100 unreferenced blobs (2.3 GB)
  deleted 200 unreferenced blobs (4.2 GB)
Deleted total 236 unreferenced blobs (4.3 GB)
Cleaned up 0 logs.afety
Cleaning up old index blobs which have already been compacted...
Finished full maintenance.
```

### AND THAT DID IT FOLKS

```
2024-10-22 17:33:35   18 Bytes velero/kopia/minimal-24csivol/xw1729640014

Total Objects: 133
   Total Size: 13.6 MiB
whayutin@thinkdoe:~/OPENSHIFT/git/OADP/oadp-operator$ 

```

## Some notes from digging into Kopia src

First, `safety=none` is only something we can run if we can guarantee that nothing will access the repository during that time. This means that before running a manual full maintenance with safety off, users must first shut down all Velero instances which reference the repository. This is not something we want to recommend that users do under normal circumstances -- it should be limited to exceptional circumstances -- something like "We accidentally backed up a 10TB volume we didn't intend to back up, so we deleted the Velero backup and need to immediately get rid of this in our bucket." Or also "We have found a bug in Velero and full maintenance is not working properly. The data should have been removed days ago but has not."

In terms of normal expectations for GC, these are the values Kopia uses to govern GC behavior during full maintenance:
```
	// SafetyFull has default safety parameters which allow safe GC concurrent with snapshotting
	// by other Kopia clients.
	SafetyFull = SafetyParameters{
		BlobDeleteMinAge:                24 * time.Hour, //nolint:mnd
		DropContentFromIndexExtraMargin: time.Hour,
		MarginBetweenSnapshotGC:         4 * time.Hour,  //nolint:mnd
		MinContentAgeSubjectToGC:        24 * time.Hour, //nolint:mnd
		RewriteMinAge:                   2 * time.Hour,  //nolint:mnd
		SessionExpirationAge:            96 * time.Hour, //nolint:mnd
		RequireTwoGCCycles:              true,
		MinRewriteToOrphanDeletionDelay: time.Hour,
	}
```
https://github.com/kopia/kopia/blob/master/repo/maintenance/maintenance_safety.go#L56C1-L68C1

The above values are used in all cases except when `safety=none`

It appears that`BlobDeleteMinAge` and `MinContentAgeSubjectToGC` reference creation time, meaning that content created fewer than 24 hour ago is not yet subject to garbage collection. Everything below assumes that we're talking about content from snapshots that were created at least 24 hours ago.

`RequireTwoGCCycles`, `MarginBetweenSnapshotGC`, and `DropContentFromIndexExtraMargin` are used as follows:

When `RequireTwoGCCycles` is true, then any content first marked for deletion during one full maintenance cycle won't be actually deleted until a second cycle confirms that it's still marked for deletion. That second cycle must be at least 4 hours later (`MarginBetweenSnapshotGC`). Once this condition has been met, as long as the content was marked for deletion no later than one hour after the start of that GC cycle (`DropContentFromIndexExtraMargin`) then it will be removed.

The reason for the above is to guard against the first full maintenance starting at about the same time as another snapshot is created which references the old content.