# Electron Offscreen Rendering (`viz`)

This document is an overview of the new implementation of offscreen rendering. We needed to reimplement the internals because of the new `viz` compositor, which is now enabled by default. Most of the problems arise from the fact that the compositor now lives in a separate process.

## How things work

To render a `WebContents` offscreen, we need to provide our own subclasses of classes that handle the displaying of the contents of the webpage.
To achieve this, we provide our own `WebContentsView` subclass `OffScreenWebContentsView` during the construction of the `content::WebContents`. The `WebContentsView` then later constructs `RenderWidgetHostView`s, which we also have a subclass of, `OffScreenRenderWidgetHostView`. We must supply these classes because we want to have the popup `RenderWidgetHostView`s rendered offscreen as well (and the childs, like `<webview>`). Popups in this context are things like `<select>` element dropdowns, which are represented as separate `views::Widgets` traditionally, and would be impossible to render offscreen otherwise, because these `Widgets` would show up on the screen when an offscreen `<select>` tag would be activated, even if the parent is not showing - and we also wouldn't be able to capture their contents into bitmaps. We overlay the bitmaps of the popup "windows" over the bitmaps of the main frame, so we maintain a copy of every `OffScreenRenderWidgetHostView`'s output in a backing bitmap to avoid unnecessary recomposition requests (we use the last image of everything else if something changes).
The `RenderWidgetHostView` also handles sizing and input event routing tasks.
We also send `BeginFrame`s from the `OffScreenRenderWidgetHostView` which allows us to control the frame rate of the rendering of the webpage. This frame rate is a maximum number, if the page is static, there is no need to render new frames, so none are rendered.

### GPU rendering

Getting bitmap results from a `WebContents` rendered by the GPU involves readback from the GPU (copy from GPU memory to system memory), which is costly, so traditionally this method was - and is - slower than software rendering. We use a well established way in `chromium` code to get an image stream from the `WebContents`. We already use the `viz::mojom::FrameSinkVideoConsumer` to implement the `webContents.beginFrameSubscription` API, so we decided to implement GPU offscreen rendering with this class as well. All we need to do is ask for the `viz` side `viz::FrameSinkVideoCapturer` to capture the page, and almost everything is handled for us. We receive the frames as shared memory regions, which we can convert back to bitmaps that we can use.

### Software rendering (`Windows` and `Linux`)

While we could use the same method as the GPU rendering method does, it wouldn't be as fast as directly using the rendered output of the `SoftwareRenderer`, since capturing the page requires at least one copy of the output, while in the software rendering case the output is already a bitmap. To capture the output directly, we need to provide our own `SoftwareOutputDevice`, which was relatively easy before the transition to the `viz` compositor, but not quite as trivial with that enabled. We can't just provide a patched hook somewhere to supply our own subclass of `SoftwareOutputDevice`, since it lives in a different process now. Instead, we generalized a `Windows` specific class that does software rendering by sending the rendered bitmaps using shared memory back to the browser process. To do this, we need to make that `SoftwareOutputDevice` work on `Linux` using a patch, and we need to implement the browser side class to receive the results.
That class is the `LayeredWindowUpdater` that lives alongside the `OffScreenHostDisplayClient` class that we also have. We need our own `HostDisplayClient` because that is the class that constructs the `LayeredWindowUpdater`, and there is a 1:1 relationship between the `HostDisplayClient` and the `ui::Compositor`, which we also construct ourselves, so we can easily create a hook to create and associate `OffScreenHostDisplayClient` with our `ui::Compositor`.
We have a flag on the `HostDisplayClient` to indicate that we need a `SoftwareOutputDeviceProxy`, that is how we decide whether to create one or let `chromium` handle the `SoftwareOutputDevice` creation. This is convenient since the `HostDisplayClient` is available at the method that creates the `SoftwareOutputDevice`s, and the flag can be read cross process.
The `SoftwareOutputDeviceProxy` class that is the generalization of the `Windows` specific one creates an `SkCanvas` backed by shared memory, `chromium` renders into that canvas, and then at the end the bitmap is sent via a `SharedMemoryHandle` to our `LayeredWindowUpdater`, where it is converted back to a bitmap.

### Software rendering (`Mac`)

On `Mac` things are a tiny bit different. Instead of creating our own `SoftwareOutputDeviceProxy`, we let `chromium` go with the default behavior. We instead use the `OnDisplayReceivedCALayerParams` method of the `HostDisplayClient`, which is called every time a new frame is rendered, because a new `IOSurface` is swapped in to be displayed by the window. The `IOSurface`'s purpose is to supply frames cross process, so we can use that and extract the image data from it into a bitmap.