Kernel Server Conversation - related to https://github.com/jupyter-server/jupyter_server/issues/1187

Attendees: Zach Sailer, Kevin Bates (IBM), and Omar Jarjur (Google)

* Rather than call these a _Kernel Server_, it could be misleading since these (within the Jupyter Server instance) are not "servers" - so we thought _Kernel Provider_ was more appropriate.

* Kernel Providers
* Provide a list of provided kernelspecs
* Allow you to create a kernel using one of these
* Allow you to connect to the provided kernels

* How do you load the configs?
* Can't leverage traitlets for this.
* Could use traitlets for defaults, and a loader
function to configure/override specific instances.
* Maybe set a config on KernelProviders that
tells where to load configs from
- This could be a class-based value, with the default reading files from the config location, but others could BYO their own loader (should _config location_ be a trait of this "loader class")

* Backward compatibility goals
* Have the default case be the local kernel provider - NO config required
* This implies that local kernel names and display_names are not modified (which would allow use of local-based notebooks via other applications like papermill)
* Does the display name compatibility have
anyone relying on it?
- I don't think so. Since local kernels won't incur name changes, the only existing "compatibility concern" would be existing gateway users.
* For cases where we will modify display names,
the modification needs to be configurable so
that the user can match their language.
- I was assuming the modification would be the configured name for the KernelProvider, which will be unicode.
- So, something like `<provided-kernel spec-display-name> (<kernel-provider-name>)` ?
- yes. I think using a "suffix approach" to display names is appropriate (as you have done).
- One (minor?) hiccup I've found in practice is that the suffixed display names get too long and the suffix is truncated by the JupyterLab launcher. It still shows up in hovertext and the drop-down kernel picker, though.
- Yeah, I agree. Realestate is an issue, but one I don't think we should worry about too much, especially given the hover and picker results.
- Perhaps we could make the "remote provider name location" a configurable item on the KernelProviders object? But only wrt display_name. I think we should always _prefix_ kernel name. (This may be overkill, but certainly doable)
* The LocalKernelProvider is always configured unless explicitly configured 'off'.
* The KernelProviders would forward any non-prefixed kernel names to the LocalKernelProvider.
* Prefixed kernel names appear in the kernelspec response and the kernel start request and (never?) in a url. Since they should never have a `/`, this would make for a good prefix-separator. (Hmm, '/' may be an issue on Windows servers where '\\' should "never appear". The separator needs to something that won't appear in a file name on a supported platform.)
* ~~Even '/' can appear in filenames on *nix OSes~~
* How does this happen? Just tried a bunch of `touch` commands and couldn't get this to occur. ([Omar] - it looks like I was wrong and you can't actually do this... although on OSX it "looks" like you can with Finder, the filename actually has a ":" in it when viewed from the shell) ([Kevin] interesting - I see what you mean)
* This might be interesting: https://github.com/takluyver/jupyter_kernel_mgmt/issues/5
* This repo was what eventually became [kernel-provisioners](https://jupyter-client.readthedocs.io/en/latest/provisioning.html) and it was called kernel-providers (but different than what we're talking about here) - which required a prefix of the provider name.
* From the issue discussion is this: "The URLs for accessing kernelspec resources (icons etc.) " - so kernel name does appear in URLs - but I think we could look for that encoding (but this implies Lab/clientApp would need to encode the `/` when it exists (which they may already do in general)).
* Once a kernel_id exists, _all_ lookups would go through the id-to-provider map to locate its provider (even in the local-only case)
* If KernelProviders implements the MappingKernelManager interface, we should be able to replace it directly within the handlers (same goes for KernelspecManager - implying that KernelProviders _also_ implement that interface)

- Other thoughts
* We'd probably want some form of kernelspec caching in the KernelProviders implementation, otherwise each "kernel server" will get hit every 10 seconds by Lab.
* This is a good reason to implement kernelspec events - where _remote_ KernelProvider implementations are consumers, then forward the event to Jupyter Server's consumers. Some kernel servers, however, may not implement that, so those KernelProviders would likely need to fallback to a polling mechanism [configurable?] much like today.
- I haven't found the list kernelspecs to be a big issue with regard to caching (it's the same behavior as the existing GatewayClient implementation), but what definitely *is* an issue is kernelspec resource loading. *Especially* (re)loading the kernelspec icons is fairly wasteful.

- One area I'm a bit unsure about is how to "plug-in and multi-plex" the handling of the **websocket connection**. I think the work Zach did to abstract out the sockets handler will come in handy here. Is this just a matter of "intercepting" the open request to a "gateway handler" (if kernel_id indicates a remote provider) and then, from then on interaction with _that_ socket will hit _that_ handler?
- Could someone take a look at that area?
- I (Zach) haven't spent much time looking at the [Gateway websocket handler and websocket](https://github.com/jupyter-server/jupyter_server/blob/main/jupyter_server/gateway/handlers.py) client, but after a quick glance, it looks like the Gateway APIs in Jupyter Server could be greatly reduced using the new Websocket Connection APIs.
- Yes, the WS multiplexing could happen inside a new Gateway implementation of `BaseKernelWebsocketConnection`. This layer has access to the [mapping kernel manager and kernel manager](https://github.com/jupyter-server/jupyter_server/blob/f3fe168bc3a30051d021953e6e3631c794675ef7/jupyter_server/services/kernels/connection/base.py#L116-L124), so it can reason about where the message should go.

- Thank you Zach