# Remote Supervisor On-Device Testing

## Introduction

### The why...

We want to test the supervisor codebase on a "real" device in order to more accurately test how applications under the control of the supervisor will behave in the field.

The benefits to this style of testing are extensive, but the main gain from this style of testing is that it ensures that user's applications will work and behave in a consistent way when interacting with the balena ecosystem. We would be using our own tools within the tests, such as the balena CLI and balenaCloud, so this also adds to the confidence that our suite of components works together prior to release.

### The what...

Extending the existing unit testing suite in the Supervisor to include a set of tests which involve pushing an application(s) to a remote device and those applications run a test and report the result back to the test suite. Prior to this we would need to sync the codebase to the device, as per our existing livepush workflow, and then open up a channel of communication with the test suite.

A simple example of this could be an SSH port forward, with a simple API bound to the remote host which a test application could `POST` a result to. The test suite would then `balena push ...` a test application to the device, which would run and report the response to the test API.

To make the Supervisor tests code-efficient and reduce duplication of effort, the test application could a base image which runs a single `bash` script and then idles. Each test scenario could then be programatically created and pushed.

## Technical

### Result Channel

Taking some cues from other languages and frameworks, we should look to have a simple JSON payload delivered over HTTP. The outer most structure should follow the schema:

{
    result: any,
    error: string | undefined
}

The script on the device should _only_ be used to gather information. The actual pass/fail decision made on the `result` should be done in the test suite. The `error` property can be used if gathering the `result` wasn't possible and a direct failure should be reported.

The result channel should be exposed to the test service via an environment variable as port-only which is bound on the device itself. This could be `SUPERVISOR_TEST_RESULT_CHANNEL_PORT`. The test will also be assigned a test ID to identify it's result, and this is passed in the environment as `SUPERVISOR_TEST_RESULT_ID`. This should be passed as a header when making the call, so that the test suit can identify which test context this applies to. This should allow for some tests to run in parallel.

>**NOTE:** Depending on the network context that the test service is run, the IP address of the result channel will be different and the service should have a means to determine the device' host IP to use.

As an example, a host-network bound service could use the following:

curl -X POST \
    "http://127.0.0.1:$SUPERVISOR_TEST_RESULT_CHANNEL_PORT \
    -H "balena-supervisor-test-id: $SUPERVISOR_TEST_RESULT_ID" \
    -d '{"result":"OK"}'

### Test Suite

The supervisor tests are written in TypeScipt and should look something like this:

it('should get a valid state from the /v1/device endpoint', async () => {
    
    // result will contain the JSON result directly from the supervisor API call...
    const result = await runTest('v1-api-device').asJSON<ApiDeviceState>();
    
    expect(result).to.have.property('status').which.equals('Idle');
});

#### `runTest(testName: string, appName?: string) => Promise<TestResult>`

The function should create a balena application context; from a single `Dockerfile` to a multi-container stack with `docker-compose.yml`. The value for `testName` defines a directory which contains the context. This pushed to the remote device using the CLI.

If a valid is provided for `appName` then this would indicate a push via balenaCloud. Without a value, the device address is used from the test environment and a local-mode push is used.

> **NOTE:** If a value is provided in the `error` field of the response then this method will thow a `RemoteTestException` with the `error` value provided as a message.

#### `TestResult`

The test result is accessed through this type. Helper functions are offered such as `asJSON()` which allow for marshalling the results into values which is more easily testable. If a helper function is called and the result cannot be marshalled then an exception is thrown.