{%hackmd @themes/orangeheart %}

# Notes On [Polkadot Ecosystem Tests](https://github.com/open-web3-stack/polkadot-ecosystem-tests/tree/master) (PET)

## Background on PET

These links are not required reading in order to get started with PET; they're helpful context on the origin, development and goals of the project.

See:
1. https://github.com/polkadot-fellows/runtimes/issues/171
2. https://forum.polkadot.network/t/polkadot-ecosystem-tests/9067
3. [Ecosystem Test Environment Bounty](https://polkadot.polkassembly.io/referenda/804)

TL;DR
* Testing initiatives had been, for some time, underway at Parity and elsewhere (see [`polkadot-integration-tests` repo](https://github.com/paritytech/polkadot-integration-tests))
* PET was inspired by Acala network's [`e2e-tests` repo](https://github.com/AcalaNetwork/e2e-tests/), created to identify regressions in XCM behavior
* It was created to satisfy an OpenGov bounty requesting creation of an ecosystem test environment (link no. 3)
* Main purpose is to be a visible central hub of testing efforts, allowing contributions by parachain teams and ecosystem builders

## Before starting

**HIGHLY RECOMMENDED**:
1. clone https://github.com/AcalaNetwork/chopsticks, and
2. skim the `README.md` for how to start a simple network

At times, it helps to have a local relay chain with one/several parachain(s) in order to locally verify some behavior, or confirm/disprove a hypothesis from a test scenario.
Having `chopsticks` ready to go will help save time.

Take note of the repository's `./configs/*.yml` directory and files.
This is where the created network's storage can be changed to match that of the test scenario, to reproduce the desired effect.

## Intro

To get started with the repository, see [the project's `README.md`](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/master/README.md) for guidance on how to install dependencies and run tests.

There's also an overview of [the project's structure](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/master/README.md), and how to add more tests, or [new kinds](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/master/README.md) of tests.

## While developing

### `chopsticks` + PJS

Chopsticks + polkadot.js is a useful combined work environment that should be used while preparing contributions to the PET.

If
* it's not clear how to structure the TypeScript for a test using `'@polkadot/types/` or `@polkadot-api`, or
* to further develop an idea for a scenario,

spin up a chopsticks network and mock the intended scenario with PJS.
This can be done either
1. manually building, signing and sending calls with the "Extrinsics" tab in the "Developer" dropdown menu, or
2. via the "`</>`JavaScript" option in the same menu

#### Inspecting running tests

This is already explained in PET's `README.md`: to debug a test _while_ it's running, use `await client.pause()`; below are some practical considerations on doing so.

Client pausing can be done on more than one chain - if the test scenario has both a relay chain and a parachain, one may:
1. insert `relayClient.pause()`
- note the lack of `await` - with it, the thread executing the `.js` would idle indefinitely without moving on to the next `pause()` calls, if any exist
3. for every parachain, add a `paraClient.pause()`
- if it's the **last** parachain that must be paused, use `await paraClient.pause()`
3. inspect `STDOUT` for port data on all chains
- the shown endpoint will be `https://polkadot.js.org/apps/?rpc=ws://[::]:<port_number>`. It may be necessary to change `[::]` to `[::1]` (IPv6 loopback) when pasting the address to PJS.
5. connect to relevant chains via PJS - as many tabs as desired can be used for each
- it is at times relevant to have "JavaScript" and "Chain State" open for each chain, to execute XCM calls and observe results origin-side and destination-side

**IMPORTANT**: temporarily modify Vitest's global variable `testTimeout` in `polkadot-ecosystem-tests/vitest.config.mts` if experimenting on PJS for prolonged periods - its default is 240 seconds.

### Concrete notes on testing

PET guide on most common testing pattern: [here](https://github.com/open-web3-stack/polkadot-ecosystem-tests?tab=readme-ov-file#key-testing-patterns).

The previous section won't be reproduced here in full; important takeaways below.

#### Block times

Calling `await client.dev.newBlock()` can take anywhere from 1 - 10 seconds, depending on conditions: CPU and network roundtrips (for interactions that require RPC fetches).

#### Snapshots

Prefer `vitest` [snapshots](https://vitest.dev/guide/snapshot) over `assert`s on concrete objects or values - both can coexist, but when in doubt or if compelled, prefer snapshots.

This is for a few reasons:
1. snapshots allow use of `chopsticks-testing` to easily [redact](https://github.com/AcalaNetwork/chopsticks?tab=readme-ov-file#testing-with-acala-networkchopsticks-testing) irrelevant information from chain data
- time-sensitive information such as timestamps/block and era counts should be excised to not cause spurious test failures
2. simultaneously, snapshots allow the storage of detailed information on the execution of an extrinsic/storage query - more practical than an `assert` for every field in a large object.
- In the event that e.g. the data or structure of an event or an extrinsic change, PET's CI will automatically raise a GH issue with the offending snapshot - this would be not easy to do with `assert`s.

##### Examples

Below are two examples of how snapshots serve as an automated error detection mechanism.

* [#61](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/61), [#63](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63) introduced E2E tests to the People chains
+ This PR found that `pallet_identity::{rename_sub, set_subs}` were missing events, so E2E tests for those extrinsics purposefully [include](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63/files#diff-c903afdb98d10e076ecab21537d36dac30e0b90dd0b62cada9033d7f1e96ed21R435) [snapshots](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/63/files#diff-c903afdb98d10e076ecab21537d36dac30e0b90dd0b62cada9033d7f1e96ed21R482) with the ([deliberately](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/3e5730a0b0d7ff192731f9fc515a3d0c43a94641/packages/kusama/src/__snapshots__/people.kusama.e2e.test.ts.snap#L317) [empty](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/3e5730a0b0d7ff192731f9fc515a3d0c43a94641/packages/kusama/src/__snapshots__/people.kusama.e2e.test.ts.snap#L331)) events emitted by the execution of those extrinsics.
+ The intent is that, when the upstream [`polkadot-sdk` PR](https://github.com/paritytech/polkadot-sdk/pull/6261) that fixes the uncovered problem reaches the production runtimes, it will trigger a sequence of failed automated CI runs that will raise an issue in the PET repo.

* [#160](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/160), [#115](https://github.com/open-web3-stack/polkadot-ecosystem-tests/pull/115) introduced E2E tests to governance features. As above:
+ a missing feature was detected (no events for `pallet_conviction_voting::{vote, remove_vote, remove_other_vote}`),
+ a [PR](https://github.com/paritytech/polkadot-sdk/pull/6544) was raised, and
+ [empty snapshots](https://github.com/rockbmb/polkadot-ecosystem-tests/blob/ab7735cb76eb512988e41331d8e028a0539b5f6d/packages/kusama/src/__snapshots__/kusama.governance.e2e.test.ts.snap#L165) were included to deliberately trigger a failure, so that
+ tests can then be fixed to include the new behavior (whose original error they uncovered)

##### Note

The above method of
1. having empty snapshots to stand in for coming fixes
2. awaiting for said fixes to be downstreamed and correcting PET as needed

also has a converse:
1. not-empty snapshots demonstrate the structure produced by the currently deployed runtimes
2. should a CI failure be caused by differing snapshots, this indicates (potentially unintended) changes introduced in `polkadot-sdk`.

TL;DR always inspect the snapshot diffs in CI failures.

## Scheduling tasks

It may be necessary to execute a call with a certain origin, or to manually schedule a call.

In these cases, check the `schedule*CallWithOrigin` helpers in `packages/shared` - they directly modify the chopsticks node's storage to inject an entry into the agenda of the scheduler pallet.

Recall that the scheduler's agenda is a key-value map keyyed by block numbers to lists of tasks scheduled for execution in that block:

	#[pallet::storage]
	pub type Agenda<T: Config> = StorageMap<
		BlockNumberFor<T>,
		BoundedVec<Option<ScheduledOf<T>>, T::MaxScheduledPerBlock>,
	>;

Internally, what this does is similar to the technique used in the guides linked in the last section:

const origin = ...
// Get the target scheduling block number.
// How this is done depends on the block provider - local, nonlocal.
const number = ...
const extrinsic = await client.api.tx.pallet.extrinsinc(...args)

await api.rpc('dev_setStorage', {
  scheduler: {
    agenda: [
      [
        [number], [
          {
            call: {
              Inline: extrinsic.method.toHex()
            },
            origin,
          }
        ]
      ]
    ]
  }
})

await api.rpc('dev_newBlock', { count: 1 })

#### Scheduling with different origins

See {%preview https://substrate.stackexchange.com/questions/12181/how-to-use-chopsticks-to-test-a-call-with-stakingadmin-origin %} for how to use origins other than `Root`.

#### Events resulting from manually scheduled tasks

If a task is executed as the result of a manual scheduling, then any events it emits - whether is succeeds or fails - will not be found with the otherwise functional method of

const events = await client.api.query.system.events()

const specificEvent = events.find((record) => {
  const { event } = record
  return event.section === 'pallet' && event.method === 'EventName'
})

Only `scheduler.Dispatched` will be found in this case.

To verify that the call did indeed succeed or fail in whichever way was expected, use `checkSystemEvents`, whose API is much like `checkEvent`'s: it allows filtering by pallet allone, or by section + method like in this example.

#### AHM

With the AHM, the scheduler pallet's agenda on Asset Hubs will be keyed by a non-local block provider - the relay chain's.

This introduces a semantic change in the above technique:

1. when at block `r` of a chopsticks relay, the task is to be manually scheduled for `r + 1`. After a `dev.newBlock`, it will be executed
2. when at block `p` of a chopsticks parachain, with `n` being the result of accessing `parachainSystem.lastRelayBlockNumber` at block `p`, the task is to be scheduled for `n` -- **and NOT** `p + 1` or `n + 1`

Note *also* that the Asset Hub system parachain's scheduler pallet may or may not have Async Backing (AB) enabled.

AB being enabled means [`chopsticks` will increment](https://github.com/AcalaNetwork/chopsticks/blob/d753769715cab36146ca6790260f7424cdca4251/packages/core/src/blockchain/inherent/parachain/validation-data.ts#L143-L152) `parachainSystem.lastRelayBlockNumber` in steps of 2 blocks.

---

Accounting for this requires giving tests a config to allow for runtime introspection, to calculate offsets depending on the environment they are running on (relay/para, AB enabled/disabled).

The alternative is to have 2 sets of test suites, mostly copy-pasted, to run with live production runtimes, and during AHM development/testing.

## Regarding the traversal of long time spans

Above, it was said that, in a `chopsticks` local network, block creation times are between 1 to 10 seconds.

In some cases, one may wish to test states or actions that require the passing of a large number of blocks.
Examples:

1. awaiting a referendum's timeout
2. observing an era change
3. fully unbonding bonded funds

To test these, it is not feasible to wait for the whole period:

// Unbonding period on Polkadot is 28 days
await client.dev.newBlock({ count: 28 * ((24 * 60 * 60) / 6) })

RPC timeout errors are certain, and even altering the timeout can still lead to `Abnormal closure` of the RPC endpoint the `chopsticks` instance is connected to.

### Mitigations

TODO

## Events

### Scrutinizing events

To check the most recent block's events, consider the following example:

const events = await baseClient.api.query.system.events()

const killedAccountEvent = events.find((record) => {
  const { event } = record
  return event.section === 'system' && event.method === 'KilledAccount'
})

// Good practice to check for this before proceeding.
expect(killedAccountEvent).toBeDefined()
// This has the added benefit of performing an automatic typecast 
assert(baseClient.api.events.system.KilledAccount.is(killedAccountEvent!.event))
// This object will be annotated with type information from the runtime's
// metadata regarding the event's fields - try it!
const killedAccountEventData = killedAccountEvent!.event.data
expect(killedAccountEventData.account.toString()).toBe(encodeAddress(alice.address, testConfig.addressEncoding))

### Events in snapshots

In case the need is instead to snapshot a test's event rather than just inspecting it, consider the following example:

const transferTx = ...
// For this to work, PJS' methods for sending transactions must be avoided in favor of chopstick's `sendTransaction`.
const transferEvents = await sendTransaction(transferTx.signAsync(alice))

await checkEvents(
  transferEvents,
  // Event of fee withdrawal from Alice
  { section: 'balances', method: 'Withdraw' },
  // Alice account is reaped, so dust is lost
  { section: 'balances', method: 'DustLost' },
)
  // In case any numbers in the event are irrelevant or unstable, causing spurious test failures, they can be redacted.
  .redact({ number: 0 })
  // Snapshots can be named, and each test can have many. This particular snapshot can be searched for by this name, and will be filed under the test's snapshot group.
  .toMatchSnapshot('unstable events when Alice `transfer_allow_death` to Bob')

Important points to remember:
1. `vitest` automatically creates snapshots, but does not automatically remove them
- for this, pass `-u` to `yarn test` to remove/update snapshot files when running tests
2. snapshots should be included in the PR of a contribution to PET

## Errors

### Scrutinizing PJS errors

See this for an example on how to catalyze a block's information for a failed extrinsic: https://polkadot.js.org/docs/api/cookbook/blocks/#how-do-i-determine-if-an-extrinsic-succeededfailed.

### Spurious `yarn test` errors locally/in CI

At times, there may be spurious test failures caused by underlying infrastructure/network issues.

Recent practical example: https://github.com/open-web3-stack/polkadot-ecosystem-tests/issues/161

Summary: in CI, errors such as `Internal RpcError: -32008: Response is too big: Exceeded max limit of ...` were caused by ([accidental](https://github.com/open-web3-stack/polkadot-ecosystem-tests/issues/161#issuecomment-2575665034)) changes in the configuration of RPC nodes.

Always inspect test failures carefully, as their cause might be orthogonal to the test's object, and reveal an issue with the scenario that must be addressed.

## Miscellanous

### `vitest` snapshots

If CI or local runs fail for a reason such as

 Error: Error: Snapshot `Kusama Asset Hub Bounties > All bounty failure tests > Bounty cannot be awarded if it has an active child bounty > child bounty added events 1` mismatched
 ❯ Object.toMatchSnapshot packages/shared/src/helpers/index.ts:19:17
 ❯ Checker.toMatchSnapshot node_modules/@acala-network/chopsticks-testing/dist/esm/check.js:240:51
 ❯ hasActiveChildBountyTest packages/shared/src/bounties.ts:2886:3
 ❯ testFn packages/shared/src/bounties.ts:3112:29

and there is no more information regarding the actual cause of the snapshot mismatch, the data which caused it, or a diff,

1. check the snapshot filename in the local repository
2. check that filename in the PR's latest commit
- test snapshots **must** be in the `git` tree
3. check the offending snapshot manually
- it's possible some local change wasn't commited, and the latest PR's latest commit has an empty snapshot, so there's nothing to compare the test results to, and `undefined` can't be printed or diffed against

### PJS

#### Executing `chopsticks` RPC calls in the JS console

Where in a PET test in a TypeScript module, one would write

relayClient.api.<pallet_name>.<extrinsic_name>(..args)

await paraClient.dev.newBlock()

await relayClient.dev.setStorage({
  Pallet: {
    storageItem: [
      ...
    ],
  },
})

in the JS console, it would instead be

api.<pallet_name>.<extrinsic_name>(..args)

await api.rpc('dev_newBlock()', ..args)

await api.rpc('dev_setStorage', {
    Pallet: {
      storageItem: [
        ...
      ],
    },
})

Documentation on these RPC methods, their arguments and how to call them:
1. [Acala's `chopsticks` docs](https://acalanetwork.github.io/chopsticks/docs/chopsticks/namespaces/DevRPC/README.html)
2. [Source code](https://github.com/AcalaNetwork/chopsticks/tree/master/packages/core/src/rpc/dev)

See @seadanda's HackMD [below](#Additional-reading) for a more comprehensive set of techniques to be used when manually testing in PJS, which make use of the above RPC endpoints.

#### PJS development accounts

By default, when using `chopsticks` through `polkadot-js` in a browser, development accounts - Alice through Ferdie - use the [Sr25519](https://github.com/polkadot-js/apps/blob/fd8eeb556a16bce1d4f7b9db247b188614f6ad64/packages/test-support/src/keyring/signers.ts#L9) signature scheme, whereas PET'S [testing keypairs](https://github.com/open-web3-stack/polkadot-ecosystem-tests/blob/master/packages/networks/src/defaultAccounts.ts#L5) use [Ed25519](https://github.com/AcalaNetwork/chopsticks/blob/master/packages/utils/src/index.ts#L257) by default.

Be mindful of this when `pause()`ing tests to then resume them manually in the browser, as `Alice` in the PET test will **not** be the same account as `Alice` in PJS.

### PR reviews

When awaiting for comments or reviews from @xlc, the maintainer of PET, please note that Bryan is on UTC+12/13.
Work on PET is best done async so as to not park yourself idle waiting for input.

Furthermor, be mindful of times for meetings/DMs.

## Additional reading

- Notes by @seadanda: {%preview https://hackmd.io/@seadanda/Sk3qcRlM0 %}
- shows how to dispatch `Root` origin calls from relay chain to parachain using above `scheduler` method + XCM
- showcases usefulness of `chopsticks` to debug/experiment
- exemplifies runtime authorization/upgrade via PJS
- Governance experiments by @xlc: {%preview https://hackmd.io/@xlc/H11BX5BLa %}
- shows how to dispatch `Root` origin calls via the `scheduler` pallet