Device Plugin register options

# Device Plugin register options ###### tags: `K8s` ## DevicePluginServer * In device plugin server, it defines gRPC interfaces: ```protobuf= // DevicePlugin is the service advertised by Device Plugins service DevicePlugin { // GetDevicePluginOptions returns options to be communicated with Device // Manager rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {} // ListAndWatch returns a stream of List of Devices // Whenever a Device state change or a Device disappears, ListAndWatch // returns the new list rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {} // GetPreferredAllocation returns a preferred set of devices to allocate // from a list of available ones. The resulting preferred allocation is not // guaranteed to be the allocation ultimately performed by the // devicemanager. It is only designed to help the devicemanager make a more // informed allocation decision when possible. rpc GetPreferredAllocation(PreferredAllocationRequest) returns (PreferredAllocationResponse) {} // Allocate is called during container creation so that the Device // Plugin can run device specific operations and instruct Kubelet // of the steps to make the Device available in the container rpc Allocate(AllocateRequest) returns (AllocateResponse) {} // PreStartContainer is called, if indicated by Device Plugin during registeration phase, // before each container start. Device plugin can run device specific operations // such as resetting the device before making devices available to the container rpc PreStartContainer(PreStartContainerRequest) returns (PreStartContainerResponse) {} } ``` * We have to implement GetDevicePluginOptions(), GetPreferredAllocation() and PreStartContainer() to make device plugin options available. * GetPreferredAllocation() * GetPreferredAllocation() let kubelet knows, from device plugin point of view, which device(s) is prefered to allocate. * This is just a "suggestion", kubelet will determine by itself. * GetPreferredAllocation() needs PreferredAllocationRequest as input arguement. ```protobuf= // PreferredAllocationRequest is passed via a call to GetPreferredAllocation() // at pod admission time. The device plugin should take the list of // `available_deviceIDs` and calculate a preferred allocation of size // 'allocation_size' from them, making sure to include the set of devices // listed in 'must_include_deviceIDs'. message PreferredAllocationRequest { repeated ContainerPreferredAllocationRequest container_requests = 1; } message ContainerPreferredAllocationRequest { // List of available deviceIDs from which to choose a preferred allocation repeated string available_deviceIDs = 1; // List of deviceIDs that must be included in the preferred allocation repeated string must_include_deviceIDs = 2; // Number of devices to include in the preferred allocation int32 allocation_size = 3; } ``` * available_deviceIDs indicates currently available device(s) from kubelet. * must_include_deviceIDs indicates in above devices, which device(s) is mandatory. * allocation_size indicates kubelet needs how much of device(s). * GetPreferredAllocation() will also return PreferredAllocationResponse to kubelet. ``` protobuf= // PreferredAllocationResponse returns a preferred allocation, // resulting from a PreferredAllocationRequest. message PreferredAllocationResponse { repeated ContainerPreferredAllocationResponse container_responses = 1; } message ContainerPreferredAllocationResponse { repeated string deviceIDs = 1; } ``` * deviceIDs indicates the result which device plugin prefers to allocate. * PreStartContainer() * kubelet will call PreStartContainer() to let device plugin knows finally which device(s) is chosen by kubelet, and give a chance to device plugin to do some operation before container creating. * PreStartContainer() will pass an arugement called PreStartContainerRequest. ```protbuf= // - PreStartContainer is expected to be called before each container start if indicated by plugin during registration phase. // - PreStartContainer allows kubelet to pass reinitialized devices to containers. // - PreStartContainer allows Device Plugin to run device specific operations on // the Devices requested message PreStartContainerRequest { repeated string devices_ids = 1 [(gogoproto.customname) = "DevicesIDs"]; } ``` * devices_ids indicates which device(s) kubelet will allocate. * GetDevicePluginOptions() * GetDevicePluginOptions() is the function let kubelet knows GetPreferredAllocation() or/and PreStartContainer() are available. * GetDevicePluginOptions() will return DevicePluginOptions ```protobuf= message DevicePluginOptions { // Indicates if PreStartContainer call is required before each container start bool pre_start_required = 1; // Indicates if GetPreferredAllocation is implemented and available for calling bool get_preferred_allocation_available = 2; } ``` * Both elements indicate the corresponed option is available or not. * However, in real environment testing(kubelet v1.23.4), GetDevicePluginOptions() dosen't be automatically called at all, we have to register these two calls at registeration phase. ## Device Manager Registration Service * Register request defines the following: ```protobuf= message RegisterRequest { // Version of the API the Device Plugin was built against string version = 1; // Name of the unix socket the device plugin is listening on // PATH = path.Join(DevicePluginPath, endpoint) string endpoint = 2; // Schedulable resource name. As of now it's expected to be a DNS Label string resource_name = 3; // Options to be communicated with Device Manager DevicePluginOptions options = 4; } ``` * DevicePluginOptions is the options indicate GetPreferredAllocation() or/and PreStartContainer() is available * DevicePluginOptions contains followings: ```protobuf= message DevicePluginOptions { // Indicates if PreStartContainer call is required before each container start bool pre_start_required = 1; // Indicates if GetPreferredAllocation is implemented and available for calling bool get_preferred_allocation_available = 2; } ``` ## Summary procedure ![](https://i.imgur.com/AiLiI0f.png)