# Krustlet Device Plugin Support
###### tags: `Wasm`
[Krustlet](https://github.com/deislabs/krustlet) is a open source project that enables WebAssembly workloads to run natively on Kubernetes. It does this by implementing the kubelet API, scheduling WASM modules instead of Pods upon requests from the Kubernetes scheduler.
Kubernetes' [Device Plugin](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/) framework enables Kubernetes workloads to request extended resources, such as hardware, advertised by Device Plugins. While krustlet currently supports CSI plugins, it does not support [Device Plugins](https://github.com/deislabs/krustlet/blob/811fbf871ca7f6955d1018b6cb61db7670583c6b/docs/topics/plugin_system.md#what-is-not-supported). This documentation goes through the work needed to be done to enable device plugin support.

## Investigation
Upon startup, the Kubernetes kubelet [registers two Plugin Handlers](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/kubelet.go#L1403), one for CSI plugins and the other for for device plugins. Krustlet has currently implemented the equivalent of the CSI Plugin Handler as a `PluginRegistry` in its [`plugin_watcher` module](https://github.com/deislabs/krustlet/blob/main/crates/kubelet/src/plugin_watcher/mod.rs). The `PluginRegistry` implements both the [plugin registration API](https://github.com/kubernetes/kubelet/blob/v0.19.2/pkg/apis/pluginregistration/v1/api.proto) -- which despite its agnostic name, is not for Device Plugins -- and [`PluginHandler`](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/cache/types.go#L47) interface.

Kubernetes' kubelet has one [pluginManager](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/plugin_manager.go#L83) to which the CSI and DP `PluginHandlers` register. The [`pluginManager.reconciler`](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/reconciler/reconciler.go#L73) makes sure that all plugins are in the correct (registered or deregistered) state. For example, it checks to see if a socket no longer exists for a plugin that is in the registered plugin list of the`pluginManager.reconciler.actualStateOfWorld`. If so, it will [unregister the plugin](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/reconciler/reconciler.go#L116) by calling the associated `PluginHandler's` `deregisterPlugin` function.

Therefore, plugin management in Kubernetes consists of the following:

1. A single `PluginManager` maintains list of plugins and reconcile's state of plugins.
2. CSI and DP `PluginHandlers` implement the [`PluginHandler` interface](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/cache/types.go#L47), so the `PluginManager` can ask them to register/deregister plugins when reconciliation is needed.
3. CSI and DP plugins implement their own APIs for interacting with their plugins ([CSI plugin API](https://github.com/kubernetes/kubelet/blob/v0.19.2/pkg/apis/pluginregistration/v1/api.proto) and [DP Plugin API](https://github.com/kubernetes/kubelet/blob/v0.19.2/pkg/apis/deviceplugin/v1beta1/api.proto#L23).

The only shared code between the two `PluginHandlers` appears to be the reconciliation that happens by the `PluginManager`.

The CSI and DP `PluginHandlers` handle registration differently. CSI plugins advertise a `Registration` service that the CSI `PluginHandler` calls when it notices a new socket under `/var/lib/kubelet/plugins_registry/`. The inverse happens for DPs. The DP `PluginHandler` advertises a `Registration` service that DPs call.

## Kubelet Proposed Restructuring
Currently, in krustlet, there is a `plugin_watcher` module that contains support for CSI plugins. It also contains a `PluginType` enum with both `CsiPlugin` and `DevicePlugin` options. However, the remaining code is specific to CSI plugin support. This `plugin_watcher` module may be best renamed `csi_plugin_handler` and moved to be a submodule under a `plugins` module. A new `device_plugin_handler` submodule would also live under the `plugins` module. The `plugin_watcher's` `valdiate`, `register`, and `remove_plugin` functions, which appear to be an implmentation of the [`PluginHandler` interface](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/cache/types.go#L47), can be moved to be an implementation of a `PluginHandler` trait, implemented in both the `csi_plugin_handler` and `device_plugin_handler` modules and called out to by the `plugin_manager`.

>The Kubernetes' Device Plugin Manager implements the `PluginHandler` interface [here](https://github.com/kubernetes/kubernetes/blob/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/cm/devicemanager/manager.go#L341).

The rust modules inside the krustlet kubelet (`./krustlet/crates/kubelet`) could be divided like so:

kubelet
└── src
    └── plugins
        ├── plugin_manager 
        ├── plugin_handler_interface
        ├── reconciler
        ├── device_plugin_handler
        │   └── ...
        └── csi_plugin_handler

> Would these sub-modules be better named `device_plugin_manager` and `csi_plugin_manger`, since the `PluginHandler` interface is only a subset of the functionality they implement? They also each implement their plugin APIs in order to manage plugins. The name `plugin_registry` works for CSI, since its [API](https://github.com/kubernetes/kubelet/blob/v0.19.2/pkg/apis/pluginregistration/v1/api.proto) is focused on registering the plugins; however, kubelet implements both the `Registration` service and device plugin client in the [Device Plugin API](https://github.com/kubernetes/kubelet/blob/v0.19.2/pkg/apis/deviceplugin/v1beta1/api.proto#L23).

Or, to better match the Kubernetes repository layout where plugins are placed under the directory of their area of concern:

kubelet
└── src
    ├── plugin_manager
    │   ├── plugin_handler_interface
    │   └── reconciler
    ├── cm (container manager)
    │   ├── device_plugin_handler 
    │   └── ... later add support for memory manager, topology manager, etc, as in K8s cm dir
    └── volume
        └── csi_plugin_handler

> Note: these are the K8s equivalent for the `csi_plugin_handler` can be found [here](https://github.com/kubernetes/kubernetes/blob/fd74333a971e2048b5fb2b692a9e043483d63fba/pkg/volume/csi/csi_plugin.go).

This latter layout propses adding a directory for container managers (cm) as done in [Kubernetes](https://github.com/kubernetes/kubernetes/tree/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/cm). It contains the device plugin manager (equivalent to the `PluginHandler`), memory manager, topology manager, etc.

## Futher investigation
The device plugin manager hosts the registration service of the device plugin interface. After a device plugin (DP) calls [this register function](https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/kubelet/cm/devicemanager/manager.go#L439) the DP manager adds the DP's endpoint to a list it maintains. Then the DP manager creates a DP client to interact with the DP (call `list_and_watch` and `allocate`). However, the DP manager also implements the [`RegisterPlugin` function](https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/pkg/kubelet/cm/devicemanager/manager.go#L319) of the `PluginHandler` interface, which seems to behave the same (also kicks off the creation of a device plugin client) and is called by the `PluginManager`. The inline documentation in [the container manager](https://sourcegraph.com/github.com/kubernetes/kubernetes@2112bddae101b35934f1d66288449adf31243033/-/blob/pkg/kubelet/cm/container_manager.go#L106) says this is to enable having a single module for plugin registration. However, I don't see what scenario would cause the [`reconciler` of the `PluginManager`](https://github.com/kubernetes/kubernetes/tree/ea0764452222146c47ec826977f49d7001b0ea8c/pkg/kubelet/pluginmanager/reconciler) to call the DP Manager's `RegisterPlugin`.