RenderDoc for Mozilla hacking

RenderDoc Website: https://renderdoc.org/
Documentation https://renderdoc.org/docs/index.html

The launch application tab

From inside renderdoc, find the "Launch Application tab"
Don't forget to save the settings.

Capturing WebRender

Most of the following info comes from https://github.com/servo/webrender/wiki/Debugging-WebRender#renderdoc

Attaching RenderDoc to Firefox directly

To set the executable path and command-line argument fields, look at what's printed when towards the beginning of a ./mach run invocation.

For me it looks like:

• Executable path: /home/nical/dev/mozilla/objdir/dbg/dist/bin/firefox
• CLI arguments -no-remote -profile /home/nical/dev/mozilla/objdir/dbg/tmp/profile-default

Disable sandboxing via the following environment variables (Set from within RenderDoc):

• MOZ_DISABLE_GPU_SANDBOX=1
• MOZ_DISABLE_CONTENT_SANDBOX=1
  For more sandboxing tweaks refer to https://wiki.mozilla.org/Security/Sandbox

On windows, tick "Capture Child Processes" in the "Launch application" tab.

Part of the difficulty of getting a WebRender workload captured is invalidation (we don't re-render content that hasn't changed since the previous frame). Setting the gfx.webrender.debug.force-picture-invalidation pref helps there.
RenderDoc also allows capturing N consecutive frames which also helps a lot.

Sometimes Firefox is successfully launched from renderdoc but the connection closes (timeout) before you get a chance to capture a frame in renderdoc. If this happens, it is still possible to re-attach to the Firefox process by selecting File > Attach to running instance in renderdoc.

On Windows

Start firefox with the --wait-for-browser command-line option. Otherwise the launcher process starts firefox and terminates right away which RenderDoc understands as the program terminating before anything interesting happened.

These prefs may also help:

• Set pref gfx.webrender.enabled-no-gpu-process-with-angle-win to true, and
• Set pref browser.launcherProcess.enabled to false;
• Set pref gfx.webrender.compositor to false. RenderDoc can't see frames when rendering to DirectComposition.

Via a wrench replay

Another way is to capture a trace and replay it in renderdoc via wrench.

Set the pref gfx.webrender.debug.enable-capture to true.

Multi-account containers have a keyboard shortcut which conflicts with the one used for WebRender captures. To disable the conflicting shortcut, go to about:addons > click the gear icon > Manage Extension Shortcuts and make sure that ctrl + shift + 3 is not used by the extension.

Note that WebRender captures contain most of WebRender's internal state. To ensure that the frame is re-rendered when replayed (rather than only compositing already-rendered picture cache tiles), delete the frame-*.ron file in the recording's folder. Or hit one of the arrow keys to make wrench re-render the frame.

Winit applicaitons

Renderdoc does not like wayland. Thankfully, renderdoc itself works under wayland, as long as the application being debugged uses x11. To do that for winit applications:

• For applications using winit versions up to 0.29.1, set the WINIT_UNIX_BACKEND env var to x11.
• For applications using winit versions 0.29.2 and later, set WAYLAND_DISPLAY to an empty string.

Using RenderDoc

Moving through time with the event browser and texture viewer

The event browser acts as a timeline of interactions with the API and drives what most of the other UI elements point at. I usually start by using the event browser in conjunction with the texture viewer tab to find the part of the frame I am interested in. Select various events in the event browser and see the render target updating with the result of each event.

In the texture viewer it can help to change the overlay (see the drop down menu towards the top). "Wireframe Mesh" is the overlay I use most. "Depth test" and "Stencil test" are sometimes very useful to understand why a particular draw call is not shown.

By default the texture viewer shows the render target, but you can focus on other textures like depth/stencil or inputs using the panel on the right. Sometimes it's useful to check that we are indeed sampling from the texture we expect.

If content is rendered upside-down, there is a green vertical double arrow button next to the overlay drop down menu that can be used to flip the y axis.

Mesh viewer

The mesh viewer is very useful when trying to understand what goes in and out of the vertex shader.

The tables show information for a single instance at a time. You can visit other instances using the control above the table.

In a lot of cases adding varyings and inspecting them in the mesh viewer is the moral equivalent of printf-debugging.

Pipeline state

There is a lot of information in there. Is the render target configured incorrectly? Is it an issue with the cull mode, the depth function or scissor that's preventing content from being shown? You'll probably find out in the Rasterizer or FB (Framebuffer) sections of the pipeline state tab.

You can also see which shader is used at each stage (see the "View" and "Edit" buttons in the corresponding sections).

Shader debugging

Does not work with OpenGL, but works with Vulkan and D3D.
When recording from a wrench trace on windows, pass --angle to the wrench invocation so that D3D11 is used under the hood for shader debugging to work.

Fragment shader

In the texture viewer, right-click on a pixel and click the "Debug" button on the bottom right of the UI.

Vertex shader

In the mesh viewer, right-click a vertex in the table, then "Debug this vertex".

Measuring performance

Not very precise so take results with more than a pinch of salt, but there is a little clock icon at the top of the event browser which re-runs the frame and measures the time spent in each draw call. I'd use it only as a very rough proxy.

Misc

If RenderDoc is started from the terminal, the application's stdout will be visible in the terminal that started renderdoc.

Apitrace

A different tool which, unlike RenderDoc, captures all frames, so it can be used in a few cases where RenderDoc is not a good fit.

Supports GL and D3D11 (but no modern API)

$ apitrace trace --api=egl <application>
$ qapitrace application.trace

On Mac

No Metal/RenderDoc for mac (at least not on m1 macs). But xcode's GPU debugging stuff is pretty good.

See https://github.com/gfx-rs/wgpu/wiki/Debugging-with-Xcode about how to set it up