OBSOLETE - [WebGPU CTS](https://github.com/gpuweb/cts) Test Plan

# OBSOLETE - [WebGPU CTS](https://github.com/gpuweb/cts) Test Plan This document formerly held all of the unimplemented test plans. These have been moved into the gpuweb/cts repository. **Further changes should be made there. The guidelines below are outdated.** --- --- --- --- --- ## Contents of this document Everything in this document (once it is up-to-date with these guidelines) should be **unimplemented**. Once tests are in the process of being implemented, the plan **moves** to the `description` of a `.spec.ts` file (or to a `README.txt` file) in the CTS. This prevents multiple sources of information from getting out of sync. The contents of the `description`s and `README.txt` files appear as text in the CTS's standalone runner. The test implementation To-Do list is: - Anything in this document. - Any testing labeled TODO in the CTS. ## Test planning and implementation workflow ### Lifetime of new test plans For anything in the spec/API/ToC that is not currently covered by the test plan. Note that (the completed portions of) the initial version of this document, is based on parts of the planned [Table of Contents](https://github.com/gpuweb/gpuweb/wiki/Table-of-Contents) as of ~2020-08. 1. Test plan is added to this document. 1. Test plan is reviewed (reviewer should be the test implementer if different from the test planner). 1. Tests are implemented\*. ### Lifetime of test plan changes to match spec changes For changes that come through the specification process. TODO: Rename the columns to "Needs Test Plan" and "Specification Done". 1. Spec changes go through the [WebGPU spec project tracker](https://github.com/orgs/gpuweb/projects/1). 1. Once they reach the "Needs Test Plan" column, they can be added to this document (or directly into the CTS) and then moved to the "Specification Done" column. - Some features may have tests written before their specification is complete. If they are still in the "Needs Specification" column just make sure there is a note on the issue that tests are being written, and make sure any spec changes get reflected in the tests. - If adding to an unimplemented test plan in this document, just edit the test plan directly. If adding to a test plan that has already been moved into the CTS, either: - Leave a TODO in this document describing the changes needed, or - Open a PR leaving a TODO in the test description. 1. Test plan is reviewed (if needed). 1. Tests are implemented\*. ### Lifetime of additions to existing test plans For any new cases or testing found by any test plan author, WebGPU spec author, WebGPU implementer, WebGPU user, etc. For example, inspiration could come from reading an existing test suite (like dEQP or WebGL). These may have been filed as issues against the CTS repository on GitHub. 1. Plan is added to this document (or directly into the CTS). The issue, if any, can be closed so it's tracked in just one place. 1. Test plan is reviewed (if needed). 1. Tests are implemented\*. ### Requesting a review **To request a review on test plan edits, notify @kainino0x in a HackMD comment, or someone else via GitHub or email.** If comfortable, feel free to begin on test implementation before review; most review comments will just suggest additional corner cases or generalizations. ### \* Implementing tests 1. When a test implementer picks up any item in this document, they **move** it into the test description as part of their PR, leaving a note that it is in progress. - If not implemented in the same PR, the test plan item must be marked as "TODO" in the test description. 1. Once the PR is open (may be draft), update this document to **link to the PR.** ## Conventions ### Terms/Syntax - `Iff`: If and only if - `x=` or <code>×=</code>: cartesian product (for combinatorial test coverage) - Sometimes this will result in too many test cases;simplify as needed during planning or implementation. - `{x,y,z}`: list of cases to test - Control case: a case included to make sure that the rest of the cases aren't missing their target by testing some other error case. ### Validation tests Validation tests check the validation rules that are (or will be) set by the WebGPU spec. Validation tests try to carefully trigger the individual validation rules in the spec, without simultaneously triggering other rules. Validation errors *generally* generate WebGPU errors, not exceptions. But check the spec on a case-by-case basis. Test parameterization can help write many validation tests more succinctly, while making it easier for both authors and reviewers to be confident that an aspect of the API is tested fully. Examples: - [`webgpu:api,validation,render_pass,resolve:resolve_attachment:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/render_pass/resolve.spec.ts#L35) - [`webgpu:api,validation,createBindGroupLayout:bindingTypeSpecific_optional_members:*`](https://github.com/gpuweb/cts/blob/ded3b7c8a4680a1a01621a8ac859facefadf32d0/src/webgpu/api/validation/createBindGroupLayout.spec.ts#L68) Use your own discretion when deciding the balance between heavily parameterizing a test and writing multiple separate tests. ### Operation tests Operation tests test the actual results of using the API. They execute (sometimes significant) code and check that the result is within the expected set of behaviors (which can be quite complex to compute). Note that operation tests need to test a lot of interactions between different features, and so can become quite complex. Try to reduce the complexity by utilizing combinatorics and helpers, and splitting/merging test files as needed. These should be `GPUTest`s, so any validation errors are test failures. These tests should not explicitly try to trigger validation errors, but could choose to do so if it simplifies the combinatorics. When it's easier to write an operation test with invalid cases, use `ParamsBuilder.filter`/`.unless` to avoid invalid cases, or detect and `expect` validation errors in some cases. **Default value tests** (for arguments and dictionary members) should usually be operation tests - all you have to do is make sure the behavior with `undefined` has the same expected result that you have when the default value is specified explicitly.