# Introduction

There is [external health monitoring](https://github.com/kubernetes-csi/external-health-monitor/) which does mostly similar stuff, but actions are not in the scope of it, all it can do is log the health condition of the mounts.

# Volume Healer Controller

Before we begin it is note worthy to know that the existing healer logic remains the same.

**Now the obvious question we expect is "what do we achieve with the new controller?"**
* Moving the k8s specific code out of ceph-csi, to make ceph-csi orchestrator independent.
* Achieve a way to monitor the health of NBD volumes.
* We currently do not have a way to act on failed/crashed NBD volumes with the existing healer logic. Once the controller detects that the volume is not healthy, it will have an opportunity to act on it.
* This will be a generic framework for any userspace mounters added in the future, example ceph-fuse

# Details

.-------------------.
| CSI-Addons |
| Healer Controller |
'-------------------'
|
| gRPC
|
.---------+------------------------------.
| | |
| .------------. .------------. |
| | CSI-Addons | gRPC | CSI | |
| | side-car |--------| Controller | |
| '------------' | NodePlugin | |
| '------------' |
| CSI-driver Pod |
'----------------------------------------'

## CSI-Addons Healer Controller
* The healer controller will be implemented as part of the CSI-Addons controller driver. It is the one who triggers controller RPC to check the health condition of the CSI volumes.
* Watch for the `CSIAddonsNode` object, if the object is created/modified by the side-car do below operations
* Fetch all the volume attachments (VA) list
* Filter the volume attachments list through matching driver name and status attached
* For each volume attachment get the respective PV information and check the criteria of PV Bound (also we can have configuration options like mounter type)
* Send the request for side-car to check if the volume is healthy along with the PV details details
* Log the final response returned from the side-car
* The healer controller periodically triggers the health status checks of the CSI volumes via the gRPC calls.

## CSI-Addons side-car
* The side-car registers itself by creating a `CSIAddonsNode` CR that the CSI-Addons Controller can use to connect to the side-car and execute operations.
* When a nodeplugin container is restarted the side-car will update/create(if not present) [CSIAddonsNode](https://github.com/csi-addons/kubernetes-csi-addons/blob/main/api/v1alpha1/csiaddonsnode_types.go) object.
* The side-car works as relay, upon receiving a gRPC call from healer controller, the CSI-Addons side-car will then check volume's mounting conditions collected via the CSI NodeGetVolumeStats RPC.
* In case if any volume condition is reported abnormal, the CSI-Addons side-car will act on it, it does some internal operation like fetching secrets, contructing staging path and finally sends NodeStageVolume request to the CSI driver and finally returns back the response to the controller.

PS: Some modification might be required in side-car to update the `CSIAddonsNode` object if its already present and its minimal change.

## Add Controller Changes
Watch for the `CSIAddonsNode` object, if the object is created by the side-car do below operations
* Fetch the volume attachments (VA) list and
* Send the `ControllerGetVolumeStats` request per volume

Add `ControllerGetVolumeStats` RPC

rpc ControllerGetVolumeStats (ControllerGetVolumeStatsRequest)
    returns (NodeGetVolumeStatsResponse) {}

// Make PV parameters (required) as input paramters of the RPC message b/w side-car and healer controller.
message ControllerGetVolumeStatsRequest {
  option (alpha_message) = true;

  // The ID of the volume to fetch current volume information for.
  // This field is REQUIRED.
  string volume_id = 1;
}

message ControllerGetVolumeStatsResponse {
  option (alpha_message) = true;

  // Normal volumes are available for use and operating optimally.
  // An abnormal volume does not meet these criteria.
  // This field is REQUIRED.
  bool abnormal = 1;
}

Add `GET_VOLUME` controller capabilities, If a driver supports `GET_VOLUME` capability, it MUST implement the `ControllerGetVolumeStats` RPC.

## CSI side-car changes

A new node capability is added. Add a `VOLUME_CONDITION` node capability. If driver supports this capability, it MUST fetch volume condition information in `ControllerGetVolumeStats`. On a `ControllerGetVolumeStatsRequest` from Controller, for any PV that is mounted, the side-car calls `NodeGetVolumeStats` to see if volume is still mounted; and returns its condition.

i.e. Instead of adding a new RPC, we can leverage the existing `NodeGetVolumeStats` RPC by extening the existing `NodeGetVolumeStats` CSI function.

rpc NodeGetVolumeStats (NodeGetVolumeStatsRequest)
    returns (NodeGetVolumeStatsResponse) {}


message NodeGetVolumeStatsRequest {
  // The ID of the volume. This field is REQUIRED.
  string volume_id = 1;

  // It can be any valid path where volume was previously
  // staged or published.
  // It MUST be an absolute path in the root filesystem of
  // the process serving this request.
  // This is a REQUIRED field.
  string volume_path = 2;

  // The path where the volume is staged, if the plugin has the
  // STAGE_UNSTAGE_VOLUME capability, otherwise empty.
  // If not empty, it MUST be an absolute path in the root
  // filesystem of the process serving this request.
  // This field is OPTIONAL.
  string staging_target_path = 3;
}

message NodeGetVolumeStatsResponse {
  // This field is OPTIONAL.
  repeated VolumeUsage usage = 1;
  // Normal volumes are available for use and operating optimally.
  // An abnormal volume does not meet these criteria.
  // This field is REQUIRED.
  bool abnormal = 1;
}

In a line `GET_VOLUME` and `VOLUME_CONDITION` controller capabilities will be implemented. If a driver supports `GET_VOLUME` capability, it MUST implement the `ControllerGetVolumeStats` RPC. If a driver supports `VOLUME_CONDITION`, it MUST supports the volume condition flag in the status field in `NodeGetVolumeResponse`.

Also please note its the side-car's duty to act by issuing a `NodeStageVolume` call if the volume is reported abnormal as part of `NodeGetVolumeStatsResponse`.