### Problem Statement

Today, Velero does not support asynchronous processes triggered from within a plugin. This means that all plugins must block on external processes/workflows within a plugin, which blocks and prevents the main backup process from completing. The introduction of asynchronous plugin types is required to enable the implementation of a data mover without blocking the velero backup/restore process.

### Why do we need this for data mover?

Data mover implementation for Vsphere and CSI today requires that the velero backup_item_action plugin blocks waiting for the mover to complete. This prevents other plugins in the backup/restore operation from running as well as prevents the progress reporting of the movement to be bubbled back up to the velero process. Because of this, we want to be able to send data asynchronously back to the velero backup/restore process to allow the other plugins to run if a user determines the functionality is independent of other plugins and can operate asynchronously.

### Abstract

This proposal is to introduce a new plugin types called `AsyncBackupItemAction` & `AsyncRestoreItemAction` which enables the ability for velero plugins to execute without blocking backup/restore operations, allowing velero to periodically check in for status via gRPC.

### Glossary
- <b>ABIA</b>: AsyncBackupItemAction
- <b>ARIA</b>: AsyncRestoreItemAction

### Scenarios To Support

- Data Movers
- Data movers are asynchronous processes that are triggered via backup/restore of kubernetes resources. A common use case for this is the backup/restore of PVCs whose data we want to move to some form of backup storage.
- Workflow
- User takes velero backup of PVC A
- ABIA plugin applies to PVCs with compatible storage driver
- ABIA plugin triggers data mover (for example via creation of a custom resource)
- ABIA plugin returns
- Velero backup process continues
- Main velero backup process monitors running ABIA threads via gRPC to determine if process is done and healthy
- ABIA/ARIA Triggering creation of a custom resource
- Plugin creates CR
- Separate controller does the async work
- Plugin reports status back to velero
- ABIA/ARIA doing the asynchronous process inside the plugin
- Plugin does the async work
- Plugin reports status back to velero

### Goals

- Implement new plugin types that allows for asynchronous backup/restore operations
- Asynchronous plugins should allow operations on specific resource types and criteria using the same ResourceSelector logic used by current Backup/RestoreItemAction plugins.
- Enable vendors to plug their own data movers into velero using the asynchronous backup/restore plugins

### Non-Goals
- Today, Velero is unable to recover from an in progress backup when the velero server crashes (pod is deleted). This has an impact on running asynchronous processes, but it’s not something we intend to solve in this design.

### High Level Design

We will introduce 2 new plugin types `AsyncBackupItemAction` and `AsyncRestoreItemAction` which will contain methods that allow for control over what resources the plugin applies to, the action itself, and how the status of the operation is reported back to the velero process.

The asynchronous plugins allow plugin authors to define how Velero can launch and monitor external processes while allowing Velero to continue running through the backup/restore operations. This will include changes to the backup and restore controllers to Run and Wait for these processes to complete while not blocking the operation.

### Detailed Design

#### Plugin Methods and Signatures

We will introduce a new plugin type that plugin authors can use which includes new methods to backup/restore item actions enabling more complex behavior. The new plugin types will have the same methods with slightly different signatures. Every async plugin must implement the following methods:
- <b>AppliesTo</b>
- Defines which type of resources this plugin applies to.
- This remains identical to existing BIA/RIA signature
- Optionally may include other criterias that a resource must satisfy for the plugin to be applicable. Explore extending resource selector for what we need
- <b>Start</b>
- Defines what action is taken when this plugin criteria has been satisfied. The expectation is that this action is short-lived, triggering a long-standing operation outside of this methods lifecycle
- Returns a uid of the operation we have started for velero to hold a reference to the plugin operation that was launched
- If no operation is started, because the resource didn’t satisfy some set of criteria then the UID returned should be empty
- Example: ABIA that only applies to CSI compatible PVCs, if a PVC that satisfied `AppliesTo` is not CSI compatible, resource is skipped
- <b>Cancel</b>
- Accepts UID of operation, and plugin can optionally implement some ability to stop or cancel the operation. This allows for configurable timeouts of async processes.
- <b>Progress</b>
- Takes in a uid to get current state of the operation
- Returns a struct which defines whether the async process:
- Has completed?
- Errored?
- State/Progression
- Velero will use the completion/alive booleans to determine the health of the process, and will simply report the progression as a string back to the user as a condition.
- Configurable setting to determine how often velero polls this method

Potentially, there are multiple ways to implement an async plugin. For an example where we are creating a CR, and a separate controller is doing the “execute” work, then `progress()` is just reading a CR and returning it’s status. However if a goroutine launched from `run()` is itself the process doing the work, then it may want to update the plugin’s internal struct with the current state, and `progress()` is reading off the plugin struct. The possibilities are endless, and we do not intend to restrict the types of asynchronous processes that Velero will execute.

At backup/restore time, Velero will call `AppliesTo` for a resource included in the backup/restore operation and determine if the ABIA or ARIA should be run. If the ABIA/ARIA should be run, call `Run()`. Take the result of run (which returns a reference/uid to the running operation) and launch a monitor in the backup/restore process which is checking the `Progress()` method to determine if the asynchronous process has completed. Once all asynchronous processes (in addition to the other plugin processes) have completed, the backup/restore can complete.

<b>Proto:</b>

As this will be a new plugin type, we will have to define the `.proto` file, the `AsyncBackupItemAction.proto` would look like:

syntax = "proto3";
package generated;

import "Shared.proto";


message AsyncBackupItemActionAppliesToRequest {
    string plugin = 1;
}

message AsyncBackupItemActionAppliesToResponse {
    ResourceSelector ResourceSelector = 1;
}

message AsyncBackupItemActionStartRequest {
    string plugin = 1;
    bytes item = 2;
    bytes backup = 3;
}

message AsyncBackupItemActionStartResponse {
    bytes item = 1;
    int32 uid = 1;
}

message AsyncBackupItemActionCancelRequest {
  int32 uid = 1;
  int32 timeout = 2;
}

message AsyncBackupItemActionCancelResponse {
  bool ack = 1;
}

message AsyncBackupItemActionProgressRequest {
  int32 uid = 1;
}

message AsyncBackupItemActionProgressResponse {
  bytes state = 1;
}


service AsyncBackupItemAction {
    rpc AppliesTo(AsyncBackupItemActionAppliesToRequest) returns (AsyncBackupItemActionAppliesToResponse);
    rpc Start(AsyncBackupItemActionStartRequest) returns (AsyncBackupItemActionStartResponse);
    rpc Cancel(AsyncBackupItemActionCancelRequest) return (AsyncBackupItemActionCancelResponse)
    rpc Progress(AsyncBackupItemActionProgressRequest) returns ( AsyncBackupItemActionProgressResponse);
}

Similarly. the `AsyncRestoreItemAction.proto` would look like as follows:

syntax = "proto3";
package generated;

import "Shared.proto";

message AsyncRestoreItemActionAppliesToRequest {
    string plugin = 1;
}

message AsyncRestoreItemActionAppliesToResponse {
    ResourceSelector ResourceSelector = 1;

message AsyncRestoreItemActionStartRequest {
    string plugin = 1;
    bytes item = 2;
    bytes restore = 3;
    bytes itemFromBackup = 4;
}

message AsyncRestoreItemActionStartResponse {
    bytes item = 1;
    int32 uid = 2;
    bool skipRestore = 3;
}

message AsyncRestoreItemActionCancelRequest {
  int32 uid = 1;
  int32 timeout = 2;
}

message AsyncRestoreItemActionCancelResponse {
  bool ack = 1;
}

message AsyncRestoreItemActionProgressRequest {
  int32 uid = 1;
}

message AsyncRestoreItemActionProgressResponse {
  bytes state = 1;
}

service AsyncRestoreItemAction {
    rpc AppliesTo(AsyncRestoreItemActionAppliesToRequest) returns (AsyncRestoreItemActionAppliesToResponse);
    rpc Start(AsyncRestoreItemActionStartRequest) returns (AsyncRestoreItemActionStartResponse);\
    rpc Cancel(AsyncRestoreItemActionCancelRequest) return (AsyncRestoreItemActionCancelResponse)
    rpc Status(AsyncRestoreItemActionProgressItemActionRequest) returns (stream AsyncRestoreItemActionProgressResponse); 
}

}

Backup/Restore CR Status Changes
<Insert new struct to hold state of async operations>

Describe Output Changes
<Running velero describe should give details about running operations>

Changes to Backup/Restore Controller
- New asyncBackupItemActionResolver that will gather the required asynchronous actions that need to be run

Add new function to `executeAsyncActions` here: https://github.com/vmware-tanzu/velero/blob/main/pkg/backup/item_backupper.go#L211

`executeAsyncActions` launches a thread to kick off `Run()` and call `Progress()` until processes are complete. In today's current workflow, these processes would run after waiting for all restic actions to complete,

Run synchronous plugins before async plugins
<Describe places in the code where these operations would be Run and Waited for>

### Alternatives Considered

### Security Considerations

We do not foresee any extra security considerations that aren’t currently considered with synchronous backup/restore item actions.

### Compatibility

These plugins are optional to implement, but open up a number of other use cases. In particular, this enables third party data movers to be integrated into Velero as well as enhancing Velero’s existing Restic Data Mover solution. Here are examples of how this plugin can benefit both:

<b>vSphere Data Mover:</b>
Let’s take the behavior of the current vSphere plugin as an example of how a data mover could benefit from an asynchronous plugin. Today, the vSphere plugin registers a PVC backupItemAction with Velero that can optionally trigger an external data mover. The PVC backupItemAction triggers the creation of a `Snapshot` Custom Resource, which causes a separate controller to perform the snapshot/movement operations. Because this process is synchronous, the vSphere plugin blocks until the mover is successful, and the backup operation cannot continue until the mover is complete (Wait code). This is problematic, because if a backup contains 5 PVCs, the plugin is blocking 5 times in serial rather than launching the processes and waiting at the end. To change this behavior in the context of an ABIA, the user would move the logic of creating the `Snapshot` CR into `Run()` and immediately return. Then Velero will monitor the state of the mover with `Progress()` once all the other backup plugins have executed. This will allow the vSphere data mover to launch multiple `Snapshot` operations without Velero blocking and allowing the Snapshot controller to operate on these resources in parallel.

<b>Kopia/Restic:</b>
For the current implementation of Restic (and by extension Kopia), Restic PVBs/PVRs are launched manually from the core velero server code, and the waiting for these processes to complete is done as a special implementation in the backup/restore code. With an async plugin, the restic/kopia code can be implemented as a plugin which gets installed with Velero, that appliesTo Pods and creates PVB/PVR custom resources. Then, the `Progress()` function would monitor the state of the custom resources rather than the core velero code waiting for these CRs to complete directly.

### Implementation

In order to reach milestones for Data Mover, this feature should be implemented in Velero 1.10. Because this is an optional plugin type, existing plugin authors are not required to use it and it will be backwards compatible with plugins authored in previous Velero versions.

### Open Questions

- Do we enforce ABIA/ARIA registration to include restrictions on what resources trigger the plugin at the API level or do we leave this to the implementation of the plugin to discard resources we don’t care about?
- For timeout of an async process, should it be something that is configurable and controllable by velero server? Or is this something that’s configured by the plugin? Do we need timeouts in both places?
- Velero-wide setting for ALL async plugins and individual plugins have their own timeouts?
- What does the debug experience look like? When things go wrong how do we get enough information back to the user? Should we include a `Logs` method to dump logs from the external process?
- Additionally, is the `Progress()` method expressive enough to get enough information back to the user about the current state of the operation?
- Is it valid for multiple ABIA/ARIA plugins to operate on the same resource?
- Do asynchronous plugins run before or after the normal backup/restore_item_actions?
- Hunch is we want to run these after the synchronous item actions
- For restore, async plugin needs to understand there may be some delay in created resource in the cluster before it’s ready. Document this