# KEP-NNNN: Kubernetes CSI Differential Snapshot API

<!-- toc -->
- [Summary](#summary)
- [Motivation](#motivation)
- [Goals](#goals)
- [Non-Goals](#non-goals)
- [Proposal](#proposal)
- [Design Details](#design-details)
- [Alternative Designs](#Alternative-Designs)
<!-- /toc -->

## Summary
Kubernetes CSI Differential Snapshots provides a common API to query for the list of changes between any arbitrary pair of Kubernetes CSI snapshots of the same volumes to facilitate an efficient backup and restore for Kubernetes CSI volumes. This enhancement to CSI will only cover volumes backed by Block Volumes in the backend storage.

## Motivation
Efficient backup of data is an important feature for a backup system. Since all of the data in a volume does not change between backups, only backing up the data that has been changed is desirable. Many storage systems track the changes that were made since a previous point-in-time and are able to expose this to the backup application. Kubernetes CSI Snapshots provide a standard API to snapshot data but do not provide a standard way to find out which data has changed.

### Goals
* Provide changes between any arbitrary pair of snapshots of the same volume so that changed data can be identified quickly and easily for backup.
* Handle changes in volumes backed by block storage.
* Optional: This interface is optional. If this interface is not implemented by the storage vendor of the volumes being backed up, the backup software may use propiertary differential snapshot service or running full backup of the volumes.
* Changes should be able to be requested against snapshots that have been deleted. If the storage system supports it (vSphere for example) it should return the change tracking information otherwise it should return that all data has been changed.
* The minimum change granularity is a single device block.
* Handle multiple block sizes.
* Be supportable by a majority of hardware/software/cloud storage vendors.

### Non-Goals
* Handle changes for file share volumes
* Provide file system level differences
* Directory vs file level
* Blocks within files
* Subdirectory vs entire volume
## Proposal

The proposal involves enhancing the CSI with DIFFERENTIAL_SNAPSHOT_SERVICE. Any storage vendor supports DIFFERENTIAL_SNAPSHOT_SERVICE will implement gRPC call Get DifferentialSnapshot. The system also includes DifferentialSnapshot CRD for the user/client to start the differential snapshot process.


First, the user/client creates DifferentialSnapshot object in the same namespace as the target VolumeSnapshots. The user/client then watches for update of DifferentialSnapshot CR until Status is Success or Failure.

The DiffSnap Controller is a new sidecar of the CSI external-snapshotter. It will listen to the creation of DifferentialSnapshot CR and validate the VolumeSnapshots and check if the CSI Driver supports DIFFERENTIAL_SNAPSHOT_SERVICE. DiffSnap Controller then creates a DiffSnapResource on the DiffSnapshot Server and generates the URL corresponding to that DiffSnapResource. The URL is then put on the Status of the DifferentialSnapshot CR and set Status as "Success".

When the user/client sees the DifferentialSnapshot Status changed to "Success", the user/client will use the URL in the Status to issue a GET command request to the DiffSnap Server to fetch list of ChangedBlocks and NextOffset. After received the response, as long as the NextOffset is not nil, the user/client will continue to issue another GET command request with NextOffset as a parameter.

The DiffSnap Server is another thread of the DiffSnap Controller. It serves all REST GET request by issue gRPC call GetDifferentialSnapshot to the CSI Driver specified in the DiffSnapResource. The CSI Driver will create differential snapshot between 2 snapshot specified in the gPRC call. The CSI Driver may have to convert the vendor-specific metadata to DifferentialSnapshot metadata and sending the DifferentialSnapshotResponse back to the DiffSnap Server. The DiffSnap Server then put the result in the body of GET command response and send it back to the user/client.

### User Stories
#### User Story 1

#### User Story 2

### Notes/Constraints/Caveats

### Risks and Mitigations

## Design Details

### CSI DifferentialSnapshot Interface

The CSI DifferentialSnapshot will be added to the existing CSI that Storage vendor will provide as part of their CSI Volume Driver. If this service is implemented, GetPluginCapabilities response will contain DIFFERENTIAL_SNAPSHOT_SERVICE along with other CSI services. Only Block mode is supported in this KEP. Future KEP will add fields for File mode.

service DifferentialSnapshot {
  rpc GetDifferentialSnapshot(DifferentialSnapshotRequest)
    returns (DifferentialSnapshotResponse) {}
}

type DifferentialSnapshotRequest struct {
    // If SnapshotBase is not specified, return all used blocks.
    SnapshotBase       string         // Snapshot handle, optional.
    SnapshotTarget     string         // Snapshot handle, required.  
    Mode               string         // Volume mode, default "Block"
    VolumeId           string         // optional   
    StartOffset        string         // Logical offset from beginning of disk/volume.
                                      // Use string instead of uint64 to give vendor
                                      // the flexibility of implementing it either 
                                      // string "token" or a number.
    MaxEntries         uint64         // Maximum number of entries in the response
    Parameters         map[string]string    // Vendor specific parameters passed in as opaque key-value pairs.  Optional. 
}

type DifferentialSnapshotResponse struct {
    ChangeBlockList   []ChangedBlock  // array of ChangedBlock (for "Block" mode)
    NextOffset        string          // StartOffset of the next “page”.
    VolumeSize        uint64          // size of volume in bytes
    Timeout           uint64          // Time when the result of this differential snapshot is
                                      // available in the backend storage.  Seconds since epoch.
 }

type ChangedBlock struct {
    Offset            uint64          // logical offset
    Size              uint64          // size of the block data
    Context           []byte          // additional vendor specific info.  Optional.
    ZeroOut           bool            // If ZeroOut is true, this block in SnapshotTarget is zero out.
                                      // This is for optimization to avoid data mover to transfer zero blocks.
                                      // Not all vendors support this zeroout.
}

### Differential Snapshot CRDs

If BaseVolumeSnapshotName is invalid or the snapshot has been deleted, the controller will respond with respond with appropriate error. If TargetVolumeSnapshotName is specified but BaseVolumeSnapshotName is nil, the controller will respond with all used blocks in the volume.

// DifferentialSnapshot is a specification for a DifferentialSnapshot resource
type DifferentialSnapshot struct {
	metav1.TypeMeta   `json:",inline"`
	// +optional
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec   DifferentialSnapshotSpec   `json:"spec"`
	// +optional
	Status DifferentialSnapshotStatus `json:"status,omitempty"`
}

// DifferentialSnapshotSpec is the spec for a DifferentialSnapshot resource
type DifferentialSnapshotSpec struct {
	BaseVolumeSnapshotName string  `json:"baseVolumeSnapshotName,omitempty"` // Name of the base VolumeSnapshot, optional.
	TargetVolumeSnapshotName string `json:"targetVolumeSnapshotName"`        // Name of the target VolumeSnapshot.  Required.
	VolumeId       string `json:"volumeId,omitempty"`     // optional
    Mode           string `json:"mode,omitempty"`         // Default "Block"
	StartOffset    string `json:"startOffset,omitempty"`  // Logical offset from beginning of disk/volume.
	                                                      // Use string instead of uint64 to give vendor
	                                                      // the flexibility of implementing it either
	                                                      // string "token" or a number.
	Parameters map[string]string `json:"parameters,omitempty"` // Vendor specific parameters passed in as opaque key-value pairs.  Optional.
}

// Status is the status for a DifferentialSnapshot resource
type DifferentialSnapshotStatus struct {
	State           string         `json:"state"`
	Error           string         `json:"error,omitempty"`
	URL             string         `json:"url,omitempty"`
}

// List is a list of DifferentialSnapshot resources
type DifferentialSnapshotList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`
	Items []GetChangedBlocks `json:"items"`
}

### DiffSnap Controller
DiffSnap Controller is implemented as a sidecar in the CSI external-snapshotter. It watches for the create event of DifferentialSnapshot CR. The DifferentialSnapshot object will contain BaseVolumeSnapshotName and TargetVolumeSnapshotName. The DiffSnap Controller will then fetch these VolumeSnapshots and the associated VolumeSnapshotContents to validate them first. From VolumeSnapshotContent objects, the controller gets the handle of backend snapshots and CSI Driver name. The controller will also validate if the target CSI Driver does support DIFFERENTIAL_SNAPSHOT_SERVICE.

The controller then uses the CSI Driver name and snapshot handles to create DifferentialSnapshotResource object. Then a URL will be generated based on the info in this DifferentialSnapshotResource in following format: AddressOfDiffSnapServer:DiffSnapPort/DifferentialSnapshotNamespace/DifferentialSnapshotName. The controller then send this DifferentialSnapshotResource to the DiffSnap Server and put the URL in the DifferentialSnapshot object so the user/client can start getting the result from DiffSnap Server.

### DiffSnap Server

DiffSnap Server is a REST server that runs as another thread in the DiffSnap Controller and listens to specific port (configurable) at its service address (configurable).

When DiffSnap Server receives a DifferentialSnapshotResource from DiffSnap Controller, it will put this resource in the appropriate map in memory.

When DiffSnap Server receives GET request, it will use the namespace and DifferentialSnapshot name to look up the DifferentialSnapshotResource object in the map. Since all the info in the DifferentialSnapshotResource have been validated by the DiffSnap Controller, the DiffSnap Server simply send gRPC call GetDifferentialSnapshot to the corresponding CSI Driver. When the controller receives the gRPC response from CSI Driver, it will put the content of the response inside the body of the GET response and send it back to the user/client.

DiffSnap Controller must have Get permission to access VolumeSnapshot, VolumeSnapshotContent and the Get, Update, List, Delete permission to GetChangedBlocks.

### Security

Appropriate rBAC will be created to grant permission to access DifferentialSnapshot CR. So only the backup software instance and DiffSnap Controller would be able to access the DifferentialSnapshot CR to read URL. Additional rBAC may be also created to grant access to the DiffSnap Server only to specific backup software instance or data mover pod.

### CSI Driver

Storage vendors who wish to support CSI Differential Snapshot would enhance their CSI Driver to support gRPC GetDifferentialSnapshot service. gRPC server will create the differential snapshot for the 2 specified snapshots. If neccessary, gPRC server then convert the vendor specific differential snapshot's metadata into format of DifferentialSnapshotResponse. Vendor specific information could also be packaged in the Context field of the ChangedBlock.
For consecutive changed blocks, the gRPC may combine them into 1 ChangedBlock with the Offset is the offset of the first changed block and the size is the total size of all consecutive changed blocks.

## Sample usages
### Backup Block PVC with data mover
Below is an example of a backup workflow that utilizes the CSI Differential Snapshots to increase backup efficiency:
* Create a VolumeSnapshot of the Block PVC to be backed up. (The use already have previous VolumeSnapshot of the same PVC).
* Create a new Block PVC (PVC2) using the VolumeSnapshot as Source
* Create a data mover pod with the new PVC mounted as a raw block device.
* Create DifferentialSnapshot CR on Kubernetes API Server and watch its status until either Success or Failure.
* If the Status is success, send the URL to the data mover to start moving data to backup target.
* The data mover pod then issue GET command to the URL specified in the Status of the DifferentialSnapshot CR. When the data mover receives the GET response, it will get the list of changed blocks in the body of the response then the data mover will proceed to backup only the changed data blocks.

### Backup Block PVC without data mover
Below is another example of a backup workflow that does not involve creating PVC from Snapshot if the backup solution can access the storage device directly.
* Create a VolumeSnapshot of the PVC to be backed up.
* Similarly create DifferentialSnapshot CR on Kubernetes API Server and watch its status until either Success or Failure.
* If the Status is success, issue GET command to the URL specified in the Status of the DifferentialSnapshot CR. The GET response will contain the list of changed blocks in the body.
* Based on this list and the Context field, the backup software can then connect directly backend storage to fetch specific data blocks.

### Backup FileSystem PVC with data mover

### Restore FileSystem PVC with data mover

### Test plan

## Implementation History:

## Alternatives:

### DifferentialSnapshotStatus As Kubernetest Subresource via API Aggregation
DifferentialSnapshotStatus could be a Kubernetes subresource to be used in Kubernetes API Aggregation Layer. The DiffSnap Controller would directly issue gRPC command and result of the DifferentialSnapshot could be put directly on the DifferentialSnapshotStatus:
- Since all traffic related to the DifferentialSnapshotStatus is done via the Aggregation server instead of etcd, this would not put etcd server on the datapath. This approach also simplify the datapath.
- However, it still requires an additional server with persistent storage to temporarily save all of the differential snapshot info until it is clean up.

### DifferentialSnapshot Proxy Service
Another alternative deisng is to eliminate both the DifferentialSnapshot CRD and DiffSnap Controller, simply having the DiffSnap Server listens to request at known service address and port. The user/client simply send GET request command with BaseVolumeSnapshotName and TargetVolumeSnapshotName. GET request may have additional info in parameters or body. The DiffSnap Server will verify VolumeSnapshots, fetch VolumeSnapshotContent to get snapshot handle and CSI Driver name. Then it will invoke gRPC call to the CSI Driver, get the gRPC response and forward it to the GET response from the user/client.
- This approach would have simplier datapath, only a REST server running in a sidecar of CSI external-snapshot. The user/client can simply issue REST GET commands until no more data.
- However, the CRD + Controller are more familiar with the Kubernetes user.