# Virt incremental qcow2 datamover

In a kubevirt environment, one possibility for improving volume backup performance to allow more frequent backups would be to take incremental qcow2 backups using libvirt tools. Integrating this into OADP/Velero backups will require some new components to OADP (plugins, controller), as well as some modifications to the velero codebase.

## Background

Taking a VolumeSnapshot and then using kopia to process the entire volume and copy the required incremental changes into the Backup Storage Location (BSL) is a heavyweight process. Creating an incremental qcow2 backup for the same volume is generally a much more lightweight action. We want to make use of the existing Velero backup/restore process, with actual libvirt backup/restore happening via a new controller. For the moment, this will be referred to as the Virt Backup Controller. This action will be coordinated with Velero via existing infrastructure -- BackupItemActions (BIAs), RestoreItemActions (RIAs) and the DataUpload/DataDownload CRs. Initial implementation should require minimal changes to Velero, since Velero currently ignores DataUploads with `Spec.DataMover` set to something other than `velero`.

## Goals

- Back up and restore VM volumes using libvirt incremental backup instead of Velero's built-in CSI datamover
- Use existing Velero infrastructure to integrate this feature into regular velero backup and restore
- Implementation based on kubevirt enhancement defined at https://github.com/kubevirt/enhancements/blob/main/veps/sig-storage/incremental-backup.md

## Non goals
- Deep integration with velero data mover pluggability (this could be considered in the long-term though, which would minimize duplication of effort and enhance mantainability)

## Use cases
- As a user I want to use OADP to trigger backups that will back up volume data using libvirt tooling rather than CSI snapshots
- Volume backups will be incremental when possible (first backup for a given volume will be full, subsequent backups for that same volume will be incremental)
- Based on the assumption that libvirt incremental volume backups should be much faster than CSI snapshots followed by incremental kopia snapshot copy, the expectation is that users might run libvirt-based OADP backups more frequently than they would for CSI-based backups.
- If users are backing up more frequently with this method, they should ensure that they are not using CSI snapshots or fs-backup via the resource/volume policy configuration in the backup.

## High-Level design

### Upstream velero changes
- Update velero volume policy model to allow unrecognized volume policy actions. Velero would treat unrecognized actions as "skip" (i.e. no snapshot or fs-backup), but the libvirt datamover could only act if the policy action is "virt".
- In `pkg/restore/actions/dataupload_retrieve_action.go` and in `DataDownload` we need to add SnapshotType.

### BackupItemAction/RestoreItemAction plugins
- VitualMachineInstance BIA plugin
- This plugin will check to see whether QEMU backup is enabled in any of the disks -- whether `Spec.Domain.Devices.Disk[i].CBT` is `true` for at least one disk. If none are enabled, then the plugin exits without action (question: is this still valid, or is it replaced by the next?)
- The plugin will check whether the VirtualMachineInstance's `status.ChangedBlockTracking` is `Enabled`
- The plugin must also determine whether the VM is running, since offline backup is not supported in the initial release.
- If QEMU backup is enabled, the next question is whether the user wants to use the virt datamover for this VM's volumes. We will use volume policies for this, although it's a bit more complicated since a VM could have multiple PVCs. If at least one PVC for the VM has the "virt" policy action specified, and no PVCs in this VM have other non-skip policies (i.e. "snapshot", etc.) then we'll use the virt datamover
- Iterate over all PVCs for the VM
- If any PVC has an action other than "virt" or "skip", exit without action
- If at least one PVC has an action of "virt", then use the virt datamover
- A temporary PVC is created which will be used to store the qcow2 images until they are copied into the BSL.
- An annotation will be added to the PVC identifying the VMInstance we're backing up.
- This PVC probably doesn't need to be included in the backup -- if it turns out that it will be needed for restore purposes, it should be backed up *without* snapshotting or other PVC content backup. Whether we need this on restore will determine whether the PVC is added to the `additionalItems` return.
- Note: we may need to move PVC creation into the controller (see open questions below)
- This plugin will create a DataUpload with `Spec.SnapshotType` set to "qcow2" and `Spec.DataMover` set to "virt"
- Add `velerov1api.DataUploadNameAnnotation` to VMInstance
- OperationID will be created and returned similar to what's done with the CSI PVC plugin, and the async operation Progress method will report on progress based on the DU status (similar to CSI PVC plugin)
- VirtualMachineInstance RIA plugin
- Similar in functionality to csi PVC restore action
- Create temporary PVC
- Create DD based on DU annotation and DU ConfigMap

### Virt Backup Controller
- Responsible for reconciling DataUploads/DataDownloads where `spec.DataMover` is "virt"
- DataUpload reconciler (backup):
- identify the (temporary) PVC from the DU metadata
- identify the VMInstance from the PVC metadata.
- identify the already-existing qcow2 files that this incremental backup will be based on
- create the VirtualMachineBackup CR with `source` set to the VMInstance, `pvcName` set to the temporary PVC, and `backupTracker` set to the name of a `VirtualMachineBackupTracker` referencing the most recent backup checkpoint, and (optionally)`forceFullBackup`, set to `true` to force a full backup
- Wait for VMBackup to complete (monitoring status)
- Launch virt datamover pod mounting the temporary PVC with the qcow2 file(s) from the backup.
- This pod needs to be running a command that will do the datamover operation from pvc to object storage
- Will there be multiple qcow2 files if there are multiple disks with CBT enabled? Will these all use the same checkpoint name?
- Copy the new file to object storage (see [Where to store qcow2 files](#Wherehow-to-store-qcow2-files-and-metadata) below)
- Save any required metadata to identify the stored data (collection of qcow2 pathnames/checkpoints, etc.), along with identifying the backup and VMInstance they're associated with. Save this metadata file as well (see [Where to store qcow2 files](#Wherehow-to-store-qcow2-files-and-metadata) below)
- The VirtualMachineBackupTracker CR will be helpful here.
- We need to properly handle cases where we attempt an incremental backup but a full backup is taken instead (checkpoint lost, CSI snapshot restore since last checkpoint, VM restart, etc.)
- Aborted backups also need to be handled (resulting in a failed PVC backup on the Velero side)
- DataDownload reconciler (restore)
- (this area is less well defined so far, since the kubevirt enhancement doesn't go into as much detail on restore)
- We will need a temporary PVC for pulling qcow2 images from object store (if we're restoring the empty temp PVC from backup, that might work here)
- We also need PVCs created for each VM disk we're restoring from qcow2 images.
- We'll need to create another datamover pod here which will do the following:
- pod will have temp PVC mounted, as well as PVCs mounted for each vm disk we're creating.
- pod running command/image will first get the list of qcow2 files to pull from object storage
- once we have the qcow2 full backup and incremental files from object store, repeatedly call `qemu-img rebase -b fullbackup.qcow2 -f incrementalN.qcow2` for each incremental backup, in order
- Then, convert qcow2 image to raw disk image: `qemu-img convert -f qcow2 -O raw fullbackup.qcow2 restored-raw.img`
- Finally, write this image to the PVC which contains the VM disk
- (repeat process for each disk if VM has multiple disks to restore)
- Note that the various `qemu-img` actions might eventually be combined into a single kubevirt API call, but for the moment this would need to be done manually.
- Once datamover pod has restored the VM disks, it will exit and the VMInstance can launch with these disks (following the velero datamover model where the temporary PVC is deleted, retaining the PV, which then binds to the VM's PVCs may work here). The temporary PVC (containing the qcow2 images, not the restored VM disk image) should be completely removed at this point, including PV content.

### Where/how to store qcow2 files and metadata
- Current approach:
- Use the Velero object store plugin API but not the velero-specific logic in `persistence/object-store.go`
- Create a top level dir in the BSL (under the BSL prefix, parallel to backups/restores/kopia) for libvirt datamover.
- Actually, this may have to be outside the prefix (i.e. if prefix is "data" then we may need to create a parallel dir "data-qcow2" or something similar, since I think Velero allows only its own files under the prefix)
- Copy individual qcow2 files and metadata files identifying the required qcow2 checkpoints. We may want a subdir per VMInstance for qcow2 files. For metadata files, these should probably be organized by velero backup.
- We need to manage storage usage on backup deletion -- when deleting a backup, we should delete any qcow2 files no longer referenced by any still-existing backups.
- Other approaches:
- On volume in cluster
- Likely the simplest approach
- Volume must be mounted by the controller pod
- Will require its own periodic velero backups (less frequently than the VM incremental backups) for disaster recovery purposes
- Using kopia
- We could use kopia on top of the object storage API, but I'm not sure we get any real benefits, since we're already working with files that represent just the data diff we need. We can just manage them as invidivual objects
- This will also require additional overhead around kopia maintenance, etc. and on top of that, we still may need to manage qcow2 file deletion manually.

### Open questions
- How to determine PVC size?
- user-configurable? configmap or annotation?
- From the kubevirt enhancement: "Before the process begins, an estimation of the required backup size will be performed. If the provided PVC size is insufficient, an error will be returned"
- If the PVC is too small, we need a clear error on the backup indicating that it failed due to insufficient PVC space.
- If controller is responsible for PVC creation rather than plugin, then the controller may be able to respond to PVC too small errors by retrying with a larger PVC.
- The virt datamover controller will be responsible for deleting the `VirtualMachineBackup` resource once it's no longer needed. When should this happen? Upon velero backup deletion? This would enable debugging in the case of failed operations. If we delete it immediately, that will make troubleshooting more difficult. If on backup deletion, we'll need to write a `DeleteItemAction` plugin.
- When a backup is deleted, its snapshots may be needed by more recent incremental backups. What ever means we're using to map snapshots to backups will need to update this mapping to associate the deleted backup's snapshots with the next most recent backup that relies on these files. This means that for a given backup and volume we could have more than one snapshot file. We need to handle this mapping, including the order of application.
- Do we need an option to force full backups? If we're always doing incremental, eventually the incremental backup list becomes really long, requiring applying possibly hundreds of incremental files for a single restore.

### General notes
- SnapshotMoveData must be true on the backup or DU/DD processing won't work properly
- Longer-term, we can probably eliminate some of the custom code in the new controller by refactoring the velero datamover pluggability features, allowing the node agent to orchestrate this (with a custom image and configuration for datamover pods, etc.)
- The kubevirt enhancement references both "push mode" and "pull mode" -- initial implementation on the kubevirt side will be push mode only. This OADP proposal is also push mode only for the initial implementation