# ideation for backing up a file(s) off a VM

Downstream only solution as Velero will NOT have any interest in this.

## other known implementations
* https://helpcenter.veeam.com/docs/backup/vsphere/file_share_recovery_restore_files_folders.html?ver=120
* https://www.novabackup.com/blog/how-to-restore-a-single-file-from-vmware-vm-backups

## Use cases
* Simple use case: Customer does not want to back up the VM daily, perhaps just once a week. However the customer does want to backup a few files on a daily basis. Restore would be to restore 1 week old VM backup and then layer on the most recent files.

## Notes from OCP-V
* https://docs.google.com/document/d/1tAzeAiOHFXHxmTMOXYN3gJy9q2JUcXB72kIJU2rKijc/edit

## qemu guest agent
* https://wiki.qemu.org/Features/GuestAgent#Asynchronous_Commands

## New Backup CR
* launch a pod / PV
* tooling to talk to the guest-agent of a particular VM

## backup workflow
* backup-vm-file cr created
* launch pod/pv ( in vm namespace?)
* call vm guest agent of vm
* pull files from VM to pod to dir /BACKUP
* ensure user and user permissions are maintained

* Once completed
* Call a opinionated Velero Backup CR on pod
* no object data
* kopia filesystem backup of directory /BACKUP
* done?

## restore workflow

## Some impl/POC notes

This should probably leverage standard Velero workflow for some of the actions.

### Minimal POC with no new code changes

On backup:

1. Manually create a pod/pvc in VM namespace with creds to access VM guest agent API, a label unique to this operation, and hook annotations that specify the following hooks:
- pre-backup hook to pull files from VM to pod's /BACKUP dir using guest agent
- post-restore hook to push files from pod's /BACKUP dir to vm using guest agent
2. Create a regular Velero backup using a label selector to specify the operation label used for the pod
- Prior to pod backup, the pre-backup hook will run, pulling the data into /BACKUP before the volume is backed up.
4. Manually delete pod/pvc using label selector

On restore:
1. Restore the backup created above
- After pod restore, the post-restore hook will run, pushing the data from /BACKUP into the VM.
2. Manually delete pod/pvc using label selector

### Slightly less minimal POC

Same as above, but with scripts to run the steps on backup/restore

### Longer-term supported solution

Create a new VMFilesBackup CR that will specify the following:
1. The VM to access via guest agent
2. Credentials for accessing the VM
3. A list of paths/files to back up

Create a new VMFilesRestore CR which will specify the following
1. The VMFilesBackup CR to restore

Create a new controller (in OADP operator pod?) which will do the following:
- processing VMFilesBackup CR:
1. Create a pod and PVC with the following
- a label unique to this CR
- credentials for the VM guest agent available (mounted secret?)
- a list of files to back up from the VM (in annotation? same mounted secret as above? mounted ConfigMap?)
- pre-backup hook annotations to pull files from VM onto mounted volume using guest agent API
- post-restore hook annotations to push files from mounted volume to VM using guest agent API
2. (wait for pod to run) Create a velero backup CR using the label created above for the selection criteria
3. (wait for successful backup completion) Delete the pod and PVC (and secret/configmap if relevant) using the label
- processing VMFilesRestore CR:
1. Create a velero restore CR referencing the listed VMFilesBackup
2. (wait for successful restore completion ) Delete the pod and PVC (and secret/configmap if relevant) using the label

### Another longer-term possibility

Once the pre/post backup/restore hooks/plugins have been implemented in Velero (https://github.com/vmware-tanzu/velero/issues/4067 ), it may be possible to implement this using those rather than needing a new controller. Depending on how that is implemented, it may be possible to issue a backup, again specifying a label custom to this operation, which has a pre operation to create the pod+PVC(+secret+configmap) and a post operation to delete them, and then on restore, create a restore with the appropriate label and a post operation to delete the pod/pvc/etc.

## Matthew's Notes

Probably the guest agent file copy commands are the nicest base for this feature, vs. installing freeze hooks into the VM beforehand as in the OCP-V document linked above. In order for OADP to use this, the commands need to be added to the Kubevirt REST API: https://github.com/kubevirt/kubevirt/blob/9ef063d99d46e7ad291540fe58cc18cefb681b86/cmd/virt-handler/virt-handler.go#L551

There is existing code that uses the guest agent file commands for adding SSH keys, but it is not exposed for general use: https://github.com/kubevirt/kubevirt/blob/9ef063d99d46e7ad291540fe58cc18cefb681b86/pkg/virt-launcher/virtwrap/access-credentials/access_credentials.go#L122

These functions would need to be moved to a common location and plumbed up through the internal libvirt interface to the REST API, for example: https://github.com/kubevirt/kubevirt/blob/9ef063d99d46e7ad291540fe58cc18cefb681b86/pkg/virt-handler/cmd-client/client.go#L83

Once this is availble from Kubevirt, the Kubevirt velero plugin will need to change to call these commands to copy files to or from the VM. It will also need some interface with OADP to know what files to copy/restore.

Questions:
- Are users going to want to specify every file path exactly? Or do we need some kind of glob pattern or regular expression?
- On restore, does it matter that the VM must be started before files can be restored? Some applications may start using these files before the restored version can be copied in.
- OADP/Velero generally does not wait for the VM to start after restoring it. Do we need some kind of extra waiting for the VM to go running before restoring single files? Or is it better to use the freeze hooks after all? I am not sure how the hooks can get files into the backup archive, maybe it is only actually useful for purging to reduce backup size.

## Ales Nosek ( notes )

## 10/02/2024
Chat w/ Adam Litke
OpenShift-Virt ships containers w/ libguestfs-tools, guestfish etc..etc.

Perhaps a different approach should also be considered as the libvirt guest-agent may not fully support xfer of large files etc at this moment.

* Clone VM's PV's
* use libguestfs to container to mount the volume so that block or filesystem devices can be read.
* backup required files via Velero

### how would restore work in this scenerio?

With the VM powered off, the current VM PV would need to be attached to a velero pod. Then the files can be copied in with libguestfs tooling. This assumes no issues writing to NTFS for Windows VMs, which may not be a great assumption in all cases.

With the VM powered on, a PV containing the files to restore can be attached to the running guest as removable storage. Then the guest agent's "guest-exec" command can be used to run a script that copies the files into place from inside the VM. The commands to run will need to be different for different operating systems, but a bash script for Linux and a batch file for Windows should cover most cases. For compatibility, the PV can be formatted as exFAT if licensing allows.