---
# System prepended metadata

title: Disputes incident on Kusama 10-03-2023

---

# Disputes incident on Kusama 10-03-2023

A few validators started disputing a lot of included candidates

| Disputes initiated | AccountId |
|------|---------------------------|
| 5692 | GNptL51p7T4WEyL7qaXxpbW2Zzj2RKwimg1er3xZ6nJyoDx |
| 4314 | Ef1zUqDDDNME7FwfD2AXKUD5NTixnrmwaF8cfjKwYnsHqCn |
| 29   | FyL5TJXFEWDHC1yuBGwtngo3LRg4nEQ4ua47p1szMQdh8HR |
| 1    | G7Ur4BnMSfP2qE7ruSob5gwGQ5nzkGWu7Yqh14FcMqnDtgB |

Logs from `GNptL51p7T4WEyL7qaXxpbW2Zzj2RKwimg1er3xZ6nJyoDx`:
```
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 execution worker concluded, error occurred: InvalidCandidate(WorkerReportedError("execute: execute error: RuntimeConstruction(Other(\"cannot deserialize module: Module was compiled with incompatible Wasmtime version '6.0.0'\"))")) artifact_id=ArtifactId { code_hash: 0x3cf102a8162e4305af0da500073bd7fa5eea2af55d08cc36d38d4076f1cbacf8, executor_params_hash: 0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314 } worker=Worker(1v1) worker_rip=false
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 Failed to validate candidate para_id=Id(2096) error=InvalidCandidate(WorkerReportedError("execute: execute error: RuntimeConstruction(Other(\"cannot deserialize module: Module was compiled with incompatible Wasmtime version '6.0.0'\"))"))
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 Detected invalid candidate as an approval checker. reason=ExecutionError("execute: execute error: RuntimeConstruction(Other(\"cannot deserialize module: Module was compiled with incompatible Wasmtime version '6.0.0'\"))") candidate_hash=0xd167453242a6243eb6cd5c025a897bc357defcfd479e13779fe01a84beb37755 para_id=Id(2096) traceID=278344861163774145857439053264401824707
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 New dispute initiated for candidate. candidate_hash=0xd167453242a6243eb6cd5c025a897bc357defcfd479e13779fe01a84beb37755 session=28749 traceID=278344861163774145857439053264401824707
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 execution worker concluded, error occurred: InvalidCandidate(WorkerReportedError("execute: execute error: RuntimeConstruction(Other(\"cannot deserialize module: Module was compiled with incompatible Wasmtime version '6.0.0'\"))")) artifact_id=ArtifactId { code_hash: 0xc6b0fc455a18a70c8051abe73f616eca636f6f4225d0bce1c12d9ec19ce75bac, executor_params_hash: 0x03170a2e7597b7b7e3d84c05391d139a62b157e78786d8c082f29dcf4c111314 } worker=Worker(1v1) worker_rip=false
Mar 10 15:25:33 polkadot[2994789]: 2023-03-10 15:25:33 Failed to validate candidate para_id=Id(2087) error=InvalidCandidate(WorkerReportedError("execute: execute error: RuntimeConstruction(Other(\"cannot deserialize module: Module was compiled with incompatible Wasmtime version '6.0.0'\"))"))
```

The underlying reason was found by Jan:
> Another possibility could be that maybe they updated the polkadot binary while their node was still running.

## Next steps

1. Prevent node from issuing disputes if it fails to load/deserialize a PVF artifact.
We have pre-checking in place to make sure PVF is valid.

Suggested solutions:
a) Jan: Add Polkadot/wasmtime version to PVF cache path
b) Basti: We should recreate the artifact if it failed to load from disk. If this happens a second time we should stop and tell the operator to check its disk.
c) Dmitry: Check if the binary spawned by execution worker is of the same version as the spawner, and to restart node otherwise.

My personal preference is a) with maybe b)
Option c) doesn't sounds like it would completely solve it with a TOCTOU problem unless I'm missing something.

2. Investigate whether sender-side rate limiting is working as expected in dispute-coordinator/distribution. Is it not supposed to prevent us from disputing every single candidate?
3. Investigate dispute-coordinator spam resilience and performance issues (potentially network-bridge related). Disputes are important to resolve for security, but they should not substantially degrade liveness for no apparent reason.
We should not rely on slashing and disabling here as it's going to take time to implement properly with timed disputes.