# Getting WebGPU CTS actually working
Live reporting of the current progress of [Bug 1720941, AKA `webgpu-cts`](https://bugzilla.mozilla.org/show_bug.cgi?id=1720941).
## 2023-01-19 Kickoff meeting
:::spoiler
* Context:
* CTS is implemented in `gpuweb` upstream in a similar fashion to WebGL, which writes test cases as individual HTML files that has shared JS logic for common setup and teardown.
* Manual test runs can be performed with an HTML page that loads the above test case HTML files in an `iframe`.
* A CTS runner against the Deno ecosystem is located in the `cts_runner` folder in `wgpu` upstream. This currently _does not work_. :confused:
* Kelsey leads us on a tour of WebGL's current Mochitest implementation. It uses a tree of generated HTML files that wrap their correspondent upstream CTS files/cases.
:::spoiler
* These are generated by running [`generate-wrappers-and-manifest.py`](https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/generate-wrappers-and-manifest.py), which uses the following:
* Inputs:
* A big tree of test cases from Khronos upstream (which we put into [our fork](https://github.com/FirefoxGraphics/khronos-webgl))! `00_test_list.txt` files that list the more `txt` files (branch nodes) and HTML test case files (leaf nodes)
* Example: the [root `00_test_list.txt` file in our checkout](https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/checkout/00_test_list.txt)
* Corresponds to [this Khronos upstream file](https://github.com/FirefoxGraphics/khronos-webgl/blob/main/sdk/tests/00_test_list.txt).
* By default, all test cases are naively expected to pass as-is. [`mochitext-errata.ini`](https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/mochitest-errata.ini) lists exceptions (`skip-if`, `fail-if`, etc.).
* Outputs:
* [`generated-mochitest.ini`](https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/generated-mochitest.ini) becomes the final description of test cases for Mochitest to run.
* HTML files
* All of these cases is registered via [`dom/canvas/moz.build`](https://searchfox.org/mozilla-central/source/dom/canvas/moz.build#38).
* The wrapping logic does the following:
* Example upstream test case: <https://searchfox.org/mozilla-central/source/dom/canvas/test/webgl-conf/checkout/conformance/buffers/buffer-bind-test.html>
* `mochitest-single.html`: Uses the extensible `window.webglTestHarness` object to report up to the `iframe`
:::
* Should we try to run the CTS with Deno as the back end first? Or try for full Fx integration?
* @jgilbert: We should just go straight for Fx integration right now. WebGPU is already written, we just need to test it.
* Next steps:
- [ ] Actually get the CTS running against Firefox.
- [ ] @ErichDonGubler to run the CTS manually first.
- [ ] Validate that `wgpu.rs` examples actually run in a nightly or local build of Firefox first (with the right prefs. set).
- [x] `dom.webgpu.enabled` should be `true`.
- [x] Identify smoke-ish tests in the CTS that we can use to start getting interesting breakages in more advanced tests.
* Erich: This might also be interesting for aggregate statistics!
- [ ] Low-priority: see if there's an easy way to get stats from a run of tests against Fx.
- [ ] Break down later: generate Mochitest test cases from [the CTS](https://github.com/gpuweb/cts) a la WebGL.
- [ ] Don't try to de-dupe with the existing WebGL Mochitest implementation. Just dupe it and make adaptations to WebGPU.
* What else?
:::
## 2023-01-30 Analysis of the CTS
:::spoiler Erich's exhaustive tree-shaped notes on failures
```
webgpu
api
operation
adapter
requestAdapter: fails on [1]
requestDevice
default: fails on [1], seems like a foundational blocker
invalid: FREEZES
features
unknown: passes
known: fails on [1], seems like a foundational blocker
limits
unknown: fails on [7]
supported: fails on [1], seems like a foundational blocker
limit
better_than_supported: fails on [1]
supported: fails on [?]
buffers: fails on [?]
compute_pipeline: fails on [3]
device
lost
not_lost_on_gc: passes
lost_on_destroy: fails
same_object: fails
labels: fails, not implemented yet
memory_allocation: not implemented yet
memory_sync
buffer: fails on [1] and [2]
texture: fails on [1]
pipeline: not implemented yet
[1]: unable to fulfill requested features and limits
[2]: this.queue.onSubmittedWorkDone is not a function
[3]: EXCEPTION: GPUDevice.createComputePipelineAsync: 'layout' member of GPUPipelineDescriptorBase is not an object.
[4]: EXCEPTION: navigator.gpu.getPreferredCanvasFormat is not a function
[5]: EXCEPTION: Video never became ready
[6]: EXCEPTION: adapter.requestAdapterInfo is not a function
[7]: EXPECTATION FAILED: THREW NotSupportedError, instead of OperationError: NotVSupportedError: Unable to instantiate a Device
```
Failure bugs filed:
1. [1812353](https://bugzilla.mozilla.org/show_bug.cgi?id=1812353): seems particularly wide-ranging in what it's blocking
2. [1814088](https://bugzilla.mozilla.org/show_bug.cgi?id=1814088)
3. [1814090](https://bugzilla.mozilla.org/show_bug.cgi?id=1814090)
4. [1814091](https://bugzilla.mozilla.org/show_bug.cgi?id=1814091)
5. [1814094](https://bugzilla.mozilla.org/show_bug.cgi?id=1814094)
6. [1814101](https://bugzilla.mozilla.org/show_bug.cgi?id=1814101)
7. [1814098](https://bugzilla.mozilla.org/show_bug.cgi?id=1814098)
Freeze bug: [1812352](https://bugzilla.mozilla.org/show_bug.cgi?id=1812352)
:::
## 2023-01-31 How tests are organized
The easiest way to start explaining how CTS organizes tests in [`src/`](https://github.com/gpuweb/cts/tree/bd8450c3aaa35a48346674d2ecec98854882a7cd/src) is to show an example of a test case path:
```bash
webgpu:api,operations,adapter,requestAdapterInfo:limits,unknown:
# \____/ \_______________________________________/ \____________/
# | | |
# | | Test case name /
# | |
# | |
# | \ Test group path
# |
# \ Test suite name
```
:::spoiler
* Individual _test suites_ are top-level child directories inside of the `src/` folder. This is represented in a test case path as the first colon-terminated segment in each test case path (i.e., `webgpu:…`).
* Each test suite contains a `listing.ts` that returns a tree of _test groups_ as a module export.
* The tree of test groups is populated by recursively searching for:
* Directories, which become branch nodes.
* `*.spec.{js,ts}` files, which become leaf nodes containing test groups.
* `README.txt` files annotate the branch node associated with its parent directory with a description.
* Each test group is represented in a test case path as a comma-separated, colon-terminated path correspondent to discovered file path, (i.e., `…api,operations,adapter,requestAdapterInfo:…`)
* Each test group is a set of _test cases_ defined programmatically in a TypeScript file with a name of the form `*.spec.{js,ts}`. These are represented in a test case path as a final colon-terminated component, i.e., `…limits,unknown:`.
* N.B. that many tests conventionally use commas to further distinguish related concepts (such as running many similar tests with different parameters), but the commas are part of the string name in source, rather than an artifact of other structure.
Globbing is permitted as a final component after another component boundary (`:` or `,`), i.e., `webgpu:*`, or `webgpu:api,operations,device,*`.
:::
## 2023-02-02 First `try` build!
@jimblandy has recommended that @ErichDonGubler go forward with targeting WPT test cases.
## 2023-02-17 WPT tests running, but need chunking
:::spoiler
Since last update, the following interesting things have happened:
* Asked: Is stuff in `dom/webgpu/mochitest/` actually valuable to keep, with these CTS tests? [Asked](https://matrix.to/#/!AQDMTeppDxxlzBSBzM:mozilla.org/$JfKho4W9mdA-sRGcDl3n066w3xHHEnLh7AEn0NS1MCs?via=mozilla.org&via=matrix.org), waiting for an answer.
A from Kelsey: Don't worry about this for now.
- [x] Useful WPT execution was blocked on [1814745](https://bugzilla.mozilla.org/show_bug.cgi?id=1814745), because the `dom.webgpu.enabled` pref. was only set dynamically. Now it's fixed. Thanks, Kelsey!
:::
## 2023-02-22 More bugs filed!
:::spoiler
- [x] In `/webgpu/webgpu/web_platform/reftests/canvas_clear.https.html`:
```
JavaScript error: https://web-platform.test:8443/webgpu/webgpu/web_platform/reftests/canvas_clear.html.js, line 18: TypeError: GPUCommandEncoder.beginRenderPass: Missing required 'loadValue' member of GPURenderPassColorAttachment.
```
- [x] In `/webgpu/webgpu/web_platform/reftests/canvas_complex_bgra8unorm_copy.https.html`:
```
JavaScript error: https://web-platform.test:8443/webgpu/webgpu/web_platform/reftests/canvas_complex.html.js, line 34: TypeError: can't access property "height", ctx.canvas is undefined
```
- [x] In `webgpu:api,validation,buffer,mapping:mapAsync,state,mappingPending:*`:
```
Unhandled rejection: mapAsync rejected unexpectedly with: Error: mapAsync unexpectedly passed
assert@http://127.0.0.1:5000/webgpu/common/util/util.js:37:11
testMapAsyncCall@http://127.0.0.1:5000/webgpu/webgpu/api/validation/buffer/mapping.spec.js:48:15
```
- [x] In `webgpu:web_platform,canvas,getCurrentTexture:multiple_frames:*`:
```
Unhandled rejection: level (0) too large for base size (undefinedxundefined)
assert@http://127.0.0.1:5000/webgpu/common/util/util.js:37:11
physicalMipSize@http://127.0.0.1:5000/webgpu/webgpu/util/texture/base.js:42:13
getTextureCopyLayout@http://127.0.0.1:5000/webgpu/webgpu/util/texture/layout.js:42:34
expectSingleColor@http://127.0.0.1:5000/webgpu/webgpu/gpu_test.js:441:100
frameCheck@http://127.0.0.1:5000/webgpu/webgpu/web_platform/canvas/getCurrentTexture.spec.js:157:13
```
- [x] `webgpu/webgpu/web_platform/reftests/canvas_colorspace_bgra8unorm.https.html` fails ref test
- [x] `webgpu/webgpu/web_platform/reftests/canvas_colorspace_rgba8unorm.https.html` fails ref test
:::
## Current status
- [ ] Contact CI folks about potentially adding a significant amount of time to WPT tests in Taskcluster
- [x] Joel Mahar
- [ ] RyanVM
- [ ] Review CTS integration in Phabricator; tip of changes can be found at [D169953](https://phabricator.services.mozilla.com/D169953).
- [x] Work around current failures with test case expectation management, just like the failures we were already expecting anyway. Fun fact: `./mach wpt … --log-wptreport $report_json_path` and `./mach wpt-update $report_json_path` are great!
- [x] Consolidate vendoring workflow between Python and Rust, leaning towards all Rust
- [x] Use regex-based chunking, not HTML parsing and emitting (which is subject to the correctness of a lot of layers we don't need).
- [x] Figure out how to keep `wpt` chunks that handle `webgpu` tests from timing out.
- Run all tasks from `!talos !asan !tsan 'opt-web-platform-tests` and `!talos !asan !tsan 'debug-web-platform-tests`.
- Latest build of interest: <https://treeherder.mozilla.org/jobs?repo=try&revision=b04a0f2b7ca23a0c9ea7371c32d61eb649ba01f5>
- [x] Add more chunks?
- [ ] ~~Figure out how to profile and weigh longer execution times with chunks?~~ Only gonna do this if Joel or RyanVM actually have an objection.
- [x] Make sure we don't actually change `testing/web-platform/{meta,tests}` ([context](https://matrix.to/#/!AQDMTeppDxxlzBSBzM:mozilla.org/$wRyahPAWa9YAFY_JVfDj4dYTpye5LoOH9GxeoiScfPI?via=mozilla.org&via=matrix.org)); use `testing/web-platform/mozilla/{meta,tests}`, leave other stuff alone.
- [ ] ~~Replace expanded result groups with a single `expected: FAIL` or whatever applies, if possible~~
- 2023-03-06: Tried this recently, naively applying `expected: FAIL` on an entire section was too naive. Not sure how to make it pass yet.
- 2023-03-06: Nah, just skip this. This shouldn't block shipping.
Google Drive
Gist
Clipboard
Sign in with Wallet
