# Guest Language Test Suite

### What is it?
* *It is a tool*
* The test suite provides test components, infrastructure, and result data to end users.
* *It can act like a "roadmap" that implementors can choose to follow*
* Guest language projects may find it useful to implement tests as a way to track their progress towards Component Model support.
* The test suite will be structured with tests of increasing complexity so that implementors can pass simpler tests and build up to harder ones.
* *It is an output of the Bytecode Alliance Guest Languages SIG*
* The SIG will develop test suite infrastructure and contribute test implementations for affiliated projects.

### What is it not?
* *It is not a specification or standard*
* Languages are not required to implement or pass tests.
* *It is not a benchmark*
* Users are discouraged from comparing the performance of tests implemented in different languages. The test suite is focused on correctness, not performance and implementors are not expected to optimize or tune their implementations.
* Test timings should only be used by test suite contributors to detect regressions and find opportunities to speed up the overall test suite.

### What is the test suite composed of?
* **Test cases** that define the type and behavior a component must implement to pass,
* **Source code implementations** of those test cases in various languages,
* Guest language **toolchain configurations** for building components,
* **Infrastructure** to automatically build, validate, and test implementations.

## Test Cases

Each test case represents a specific set of Component-level functionality.
For example a component exporting functions with numeric arguments and results.

### Test Type

The type of the test component will be represented by a world.

package test:numbers

interface api {
  roundtrip-u8: func(a: u8) -> u8
  roundtrip-s8: func(a: s8) -> s8
  ...
}

world test {
    export api
}

world runner {
    import api
    export run: func() -> bool
}

### Test Runner

For each test case, the logic for testing whether the component performed the correct behavior will be implemented as its own component. This component will import any functions that the test exports and invoke them in some way checking the results

assert_eq!(add_u32(1, 2), 3);
assert_eq!(add_i32(4, -2), 2);
...

<div style="text-align:center">
<img src="https://hackmd.io/_uploads/SyRE7FHKa.png" />
</div>

### Test Facilities

Some tests will involve imports and verifying that the test implementation invokes them in an appropriate way. To support this, test cases may also provide a Component implementing facilities for use by the test.

# key-value facility
a = dict()

def set(k: str, v: str):
    a[k] = v

def get(k: str) -> Optional[str]:
    return a.get(k)

<div style="text-align:center">
<img src="https://hackmd.io/_uploads/rJAH7FrKT.png" />
</div>

The test runner may also import exports of the tests facilities so that it can affect the way the test facilities behave during testing and observe the effects of import calls the test implementation makes.

## Source Code Implementations

There will be source code implementations for test runners, facilities, and implementations in the test-suite folder of SIG-Guest-Languages.

For each test case, each language may implement the test and exactly one language must implement the runner/facility.

test-suite
├───README.md
├───rust
│   ├───numbers
│   ├───numbers-runner
│   └───...
└───test-cases
    ├───wit
    │   └───numbers.wit
    └───config.json

## Toolchain Configurations

All code implemented in a given guest language

## Test Runner

The test runner takes the build outputs

### Test Results

Every test will be assigned one of the following outcomes.

* ✅ Passed
* Your test passed all checks
* ❌ Failed
* Your test failed at least one check
* Your test crashed
* Your implementation was not a valid component of the expected type
* Your implementation failed to build
* ⚠️ Not run
* Test runner and/or facility crashed
* Composer failed
* Test runner and/or facility was not a valid component of the expected type
* Test runner and/or facility failed to build
* ❓ Unimplemented
* Test not implemented for this language

e.g.
| Language | Test | Status |
|-|-|-|
| rust | numbers | ✅ Passed |
| rust | records | ❌ Failed (3 failures) |

### Language Results

The "scores" for a given language are the count of tests with each status.

e.g.
rust (1 ✅, 1 ❌, 0 ⚠️, 98 ❓)

> **Note**:
> Tests with any number of failed checks are failing tests. The UI and reports will never represent this as a completion percentage to discourage implementors from treating partial sucess like success. The number of failures will be reported to provide a way to see progress towards passing.

## Nix Questions

* How will we handle platforms?
* Can we easily include a common Nix library in the root of a repo that flakes within the repo use?
* I've heard Nix only likes referring to things within the current dir which might make things challenging
* How will we structure the Nix flakes?
* We could model it after [Nixify](https://github.com/rvolosatovs/nixify/blob/main/examples/rust-lib/flake.nix) such that each test implementation has its own flake that uses some `nixComponent.rust.mkFlake`-style function to define how to build itself.
* We could have each implementation just provide a Nix file that's an expression, but is not a flake in itself.

# Motivation & Alternatives Considered

* Using containers (rejected)
* Where should the test-suite live?