---
tags: pvf
---

PVF Validation Overhaul
============

## Mechanics of Wasm execution

### Compilation

Before execution a wasm file should be compiled. Compilation takes time and memory.

Moderately sized wasm blob, may take up to 5s for compilation. Specially crafted malicious wasm blobs may take ridiculous amounts of memory and time. Compilation is heavily parallel and scales almost linearly with the number of threads.

Likely, we should do compilation in a separate process which has limits on memory consumption and execution time.

My observations are that wasmtime can produce blobs that are 10x of the input code. I suspect that debug info can make the images even more swollen.

_Jeff_: We should later switch to PVFs being special parachain blocks, so we ensure code distribution via availability, or maybe [use Rob's fast path with assuranced by availability](https://github.com/paritytech/polkadot/pull/2453). In this approach, approval checkers could verify PVF compilation times, admitadly 5s amounts to quite a slow "block execution time", but this gives assurances that no compliations [takes too long](https://github.com/paritytech/polkadot/pull/1656) or produce truly excessively sized artifacts.

### Execution

Execution of wasm is not always interruptible. Some engines do not support this. However, wasmtime does have API for [interruption][wasmtime::Config::interruptable]. It's implemented in a way that instruments the prologues and the loop headers. That is not that bad, but still has some overhead.

Another concern is stability: wasmtime is a big piece of machinery and it may contain bugs or we may misuse it, miss a changed configuration or something.

Perhaps it just better to hedge against all of this and move the whole execution into a separate process.

A PVF wasm is allowed to take a limited preconfigured amount of memory. We now agreed that it is set to 1GiB. I would change this to a lower amount for the time being.

[wasmtime::Config::interruptable]: https://docs.rs/wasmtime/0.21.0/wasmtime/struct.Config.html#method.interruptable

### Non Determinism

There was already a discussion in the github issue titled [STVF Determinism].

We should take some precautiouns for avoiding these sources of non determinism.

Specifically (may be not exhaustive):

- deterministic time at which the validation take place
- fixed upper bound linear memory size
- deterministic stack depth
- NaN cannonicalization or straight banning
- binary and syntatic limits

There are also some potential problems with sandboxed execution, since it doesn't require any determinism at the moment.

[STVF Determinism]: https://github.com/paritytech/polkadot/issues/1269

### Behavior of PVF Validation Clients

PVF system would get requests from backing, approvals and disputes.

We know what parachains could be potentially executed by backing, this is the set of all active parachains.

A word on parachains. Parachains make this set rather wide. There can be many parachains and we don't know much time in advance when they are going to be submitted for validation.

It is also known when parachains or parathreads offboard. We shouldn't be eager to remove the caches though, since we may need them for validation of disputes and approvals.

We should perform warming up caches for the cases where we know that we soon will validate.

There is a in form of complication is disputes. We may need to validate something that wasn't in the cache and pops up suddenly. It is OK to spend some time for compilation there, because disputes are not on the critical path.

Another caveat is that it is possible that the validator came offline in a middle of a session, and thus it should be able to start validation as soon as possible when it gets back up. This means that the order of compilation and the resource consumption should not prevent from validation of something out of order.

Execution of a wasm file should have an upper bound on execution time. Should it be exceeded, the execution should be terminated. There is also another, soft limit. If it is reached, the worker should notify the host about this.

### Problems of the Current Design

- We should not use shmem as we do now, because it is clunky to use, requires some raw memory manipulation, requires effort to harden against a malicious worker (in case it was hijacked), and is not striclty needed anyway. Instead, FIFO and files are preferred.
- Currently, sc-executor assumes the substrate execution model. I.e. it expects that there is a couple of runtimes active at the same time and that they compiled and instantiated once and then reused a lot of times. The substrate runtime execution cheats a bit because it doesn't care that much of non-determinism coming from the runtime (it is trusted). In contrast, PVFs are not trusted and it is important that their execution is deterministic. Moreover, there are way more PVFs and we cannot afford spending a lot of time before execution.
- Since there is no sharing of compiled artifacts, it can so happen that compilation of the same wasm module happens in multiple workers.

### Other Provisions

- We assume wasmtime or similar engine for PVF execution as primary. However, wasmi should be also supported for giving us a gold standard of execution. This is at least useful for diagnostics.
- We probably should strive to reuse sc-executor, in order to not duplicate code. That's a risk.
- It is benefitial to reuse the same version of wasmtime or otherwise we will get two instantiations of same crates. It may be even essential, since execution relies on custom signals which is a shared resource, and a potential source of bugs.
- We should be prepared to onboard another execution engine.
- There should be a limit on the number of workers and thus the number of PVF execution in flight, otherwise, we may exceed maximum available memory.

## Desired Design

### Overview

Logically execution is split on the following stages:

- Preparation
- Prevalidation. Check some conditions about the wasm file.
- Instrumentation. Modify the wasm file so that we have a
- Compilation. Take the instrumented bytecode and compile it into a machine code, optionally stashing it somewhere on the filesystem.
- Execution
- Take a compiled artifact and execute that.

candidate-validation system now manages a cache that consists of compiled machine code images. Periodically it attempts to foresee what code may be needed soon and what code likely won't be needed anymore.

Requests to this system should be changed so that they specify the validation code hash, rather than shooting validation code itself. If needed, the candidate-validation subsystem will fetch the code and compile it by itself.

### sc-executor and sc-executor-wasmtime

#### Instrumentation

There should be facilities to perform checks and instrumentation.

Instrumentation required depends on the engine. For example,

- stack depth metering can be performed via instrumentation or deferred to the engine and say be added directly in machine code
- wasmi allows reaching to non-exported mutable globals so that we could reset them. Wasmtime doesn't support that.

We could lift these instrumentation into a common place. We also don't need to perform globals instrumentation for the PVF validation path, since we are not going to reuse the wasm instances.

#### Create a WasmModule with a precompiled image

Another change that is required is ability to create a wasm instance from a serialized image. The serialized image is basically a compiled into machine code version of an instrumented wasm file. This file is specific to the version of the engine and can be invalidated by a wasmtime upgrade.

### candidate-validation Subsystem

Candidate Validation API should be altered in a way that it doesn't take the validation code itself. Instead, it should take "coordinates" of how to fetch the code, if it is not available in cache and requires preparation.

There should be logic that performs:

- pruning. Periodically (once in a session?) we should go over the cached images and remove the ones that are no longer needed.
- routing requests. Spinning up workers, tracking what worker does what.
- pre-compilation and pre-warming caches

It may so happen that there are several requests for the same code hash. Those should be debounced and only single compilation performed.

We should be careful in what way we communicate the errors happened on the preparation step: it certaintly shouldn't be eager, in the sense, that we should not shutdown a subsystem or anything of that sort if some step has failed. Instead, errors should be saved and reported at the execution stage, as if the prepartion had happened just before execution.

### Parachains Host Runtime API

ValidationCode and HistoricalValidationCode should get their "compact" versions, that only provide the hashes of the code. Alternatively, those requests and their responses should be altered in a way that the requester specifies what exactly it is interested in: hash or the preimage.

Likely, it makes sense to structure the runtime storage so that there is a mapping between the hash and the code preimage.

Besides that there should be API to get the code hashes of active parachains and parathreads. This is needed to seed the validation host with the code hashes it should compile upfront.

## Path to Implement

The changes outlined above are not insignificant, and should be done iteratively.

Right now, we already have `wasmtime` cache at our disposal, which already gives us a long way in performance. The caching key, is a several megabytes raw wasm byte blob, which is very heavy, but realistically, shuffling those won't be a business of more than a couple of milliseconds.

On the other hand, it is important is that we can warm up the compiled code caches with the relevant stuff and that we don't recompile the same file by several workers.

Another important point, is prevalidation and instrumentation. They are important for the security. Therefore, there should be priority for their implementation.

## Further Improvements

- Mitosis. In order to reduce latency of worker creation, we could designate a special reference process and `fork` it.
- Memory cache. Route candidates to the workers that recently had this candidate with a hope that it will still be able to process it.
- Hardening.
- Launching a worker under a different user,
- Giving up all capabilities except requried for validation. E.g. a worker should not be able to establish network connections or even have access to FS besides parts required to access the caches.
- Future proofing
- Be strict in which API modules export. Otherwise, they will export cruft.
- QoS
- Some work can be on the critical path, however, most of the work can be performed in the background. E.g. compilation should not take resources from, say, networking, unless we need to get the results ASAP.
- this could be regulated by chaning niceness value of the worker processes. In theory, if compilation is executed by a process that has high niceness, even though the CPU usage will be high, the other parts of the system won't notice the difference (except, of course, memory pressure).
- Deterministic stack metering in machine code.
- mmap of the wasmtime compiled image would allow to get rid of the copy inherent to `read` syscall. That maybe important because of the big sizes of compiled images.
- wasmtime optimizations:
- specify the host functions via the config
- use the linux primitive `userfaultfd` for user space page fault handling, so that we could avoid full initialization of the linear memory at the beginning and instead just initialize them on page faults. We could also keep a cache of zeroed pages to reduce latency of initialization.