Stage 2 Boogaloo

# Stage 2 boogaloo *This document is intended to lay out the rationale and a proposed path for the secondary Fault Proof project.* ## Background & Motivation The OP Stack is approaching stage one decentralization with the conclusion of the fault proof alpha chad project, which will lay the ground work for implementing the multi-proof architecture. The dependency graph of the system after stage one looks like:  ![mermaid-diagram-2023-12-29-181448](https://hackmd.io/_uploads/BJMbmR3Dp.svg) Because the current system contains several critical single points of failure, the risk of handing full challenger rights to it now is too high for acceptance. In order to reach stage two decentralization, we will need to introduce redundancy into the system by doubling up on the components above, enabling us to confidently revoke the existing override capabilities of the rollup operator. ## Goal Enable the OP Stack to reach stage two decentralization by delivering a secondary fault proof program and VM, ready to be integrated with the `op-challenger` and dispute game contracts for the multi-proof. ## Non-Goals 1. Multi-proof contracts, upgrade plans for the bridge, nor major audits. This is cut out of scope until the alternative components are in-place so that the security aspects of multiproofs can be front and center when the time comes. 2. Alternative implementations of components which are not a single point of failure, i.e. an alternative RISC-V VM to `asterisc` written in Rust or Zig. 3. Periphery modification to the FPVMs to support a wider range of programs, i.e. multithreading support or dynamic linking. ## Suggested Solution Warpath to a secondary FPP and FPVM directly after the release of the reference implementation with the fault proof alpha chad project. Adding the secondary implementations turns the above dependency graph into:  ![mermaid-diagram-2023-12-29-181059](https://hackmd.io/_uploads/B1mzGA3P6.svg) **Second Fault Proof Program** The fault proof program relies on several other components from the stack, namely: 1. A derivation pipeline 2. An EVM 3. Preimage Oracle ABI bindings Currently, the only ecosystem with existing implementations of these components, apart from the reference in the monorepo, is the growing Rust ecosystem. We have [`op-reth`][reth] & [`revm`][revm] which support OP execution, [`magi`][magi] with an existing implementation of the derivation pipeline, and [`cannon-rs`][cannon-rs] with an existing implementation of the preimage oracle ABI. A caveat to this is that [`magi`][magi] and [`cannon-rs`][cannon-rs]' respective subcomponents may not be reused for the Rust fault proof program due to their reliance on the standard library. Rust's standard `linux-gnu` and `linux-musl` targets for both MIPS and RISC-V use many currently unsupported syscalls, extensions of both of the FPVMs' base ISAs, as well as in the `gnu` targets' cases, require dynamic linking for access to `libc`. Supporting these features in our fault proof VMs will add enough complexity and security surface for this to be out of the question. <img src="https://hackmd.io/_uploads/Bybpnlaw6.png" style="height:1.5em" title="copium"> The solution to this is to color the Rust FPP with the `no_std` designation, which is much more portable, produces smaller binaries, enables finer-grained configurability of LLVM's codegen pipeline, and is intended for bare-metal or limited environments like our FPVMs. The bright side is, `revm`, `alloy`, and `reth-primitives` are already `no_std`, and `cannon-rs`' preimage oracle bindings have been ported to `no_std` as well. [`axos`][axos], the final dependency, will replace [`magi`][magi]'s current derivation pipeline. | Go (Reference) | Rust | Rust Component Status | | -------------------------------------- | -------------------------------------- | ---------------------------- | | `op-geth`'s state transition component | [`revm`][revm] + [`reth`][reth] | ✅ Implemented | | `op-node`'s derivation pipeline | [`axos`][axos] | 🚧 Under construction | | `op-preimage` | [`kona`][kona]'s preimage ABI bindings | ✳️ Implemented, need to port | **Second Fault Proof VM** `asterisc` is a fault proof VM based on the RISC-V ISA by [Proto](https://github.com/protolambda). As of the end of 2023, asterisc is near feature completion and will require a small<->medium sized lift in order to take to production. This component is included in this charter for several reasons: 1. It is nearly complete thanks to work in Proto's freetime, with estimates at 90% as of December 2023. 2. Work can be easily parallelized among the team on `asterisc`, `axos`, and `kona`. 3. Rust and LLVM's support for MIPS is dwindling, while its support for RISC-V is growing rapidly. Same goes for most programming languages which do not have financial incentive from benefactors to support MIPS, i.e. for legacy Android or PSP development. 4. The small<->medium amount of work required to bring `asterisc` to production pays off **bigtime**: We remove a secondary single point of failure from the fault proof system, decreasing the likelihood of an organization-ending event. ### Milestones The milestones for this project are split into two parallelizable workstreams: the FPVM (`asterisc`) and FPP (`kona`) squads. In addition, the EVM (`revm` + `op-reth`) and derivation pipeline (`axos`) will need to be maintained as ongoing protocol changes occur, such as the Ecotone upgrade. **Organization:** * At least 2 engineers on the `kona` workstream (owns `axos` as well, blocking `kona`'s `client` & `host` programs). * Owners: Andreas, Ben, new hire (?). * At least two engineers on the `asterisc` workstream, to join the `kona` or off-chain fault proof workstream after completion. Shouldn't take long. * Owners: Mofi, Adrian, new hire (?). * At least two engineers maintaining `op-reth` & `revm` during `kona` development, can be internal or external and will be working with the `reth` team. Once `kona` and `asterisc` are productionized, the internal Rust engineers will have enough bandwidth and will be expected to maintain this code. * Owners: Ben (prio kona), Base representative(s) (Roberto?) and/or new hire(s) (?). #### [`kona`][kona] **Workstream Total Estimate: 8 (+ 50%) engineering weeks**. `kona` is the Rust fault proof program, comprising of `revm` & `reth` for execution, `axos` for derivation, and `kona-preimage` for the preimage oracle ABI bindings. `kona` and `axos` are both greenfield projects, with `axos` completing `kona`'s required dependency graph.  ![mermaid-diagram-2023-12-29-203405](https://hackmd.io/_uploads/SyGoQx6wp.svg) **Milestone 1: [`axos`][axos]** Build out `axos`, the `no_std` implementation of the OP Stack's derivation pipeline. This dependency should be `no_std` and synchronous at its core, with abstractions that allow for integration in fully-fledged async environments, such as `magi`'s common targets. *Definition of Done*: `axos` implements a [to spec](https://github.com/ethereum-optimism/optimism/blob/develop/specs/derivation.md#l2-chain-derivation-pipeline) derivation pipeline for the current version of the OP Stack protocol (up to and including Ecotone.) Bonus points if we can coordinate with a16z and/or RiscZero for a quick turnaround on integration with `axos` over `magi`'s current derivation pipeline implementation. **Milestone 2: `kona` build pipelines** Create build pipelines for the current FPVM targets: `cannon` (MIPS32rel1, subset of Linux kernel) and `asterisc` (riscv64gc, subset of Linux kernel). *Definition of Done*: Docker images for compiling `no_std` Rust code down to the existing FPVM targets have been created with an accompanying release pipeline and test programs. **Milestone 3: `kona` common & preimage** Create a `common` and `preimage` crate within [`kona`][kona], where: - `kona-common` contains standard utilities and Kernel interfaces, such as syscall abstractions, common traits, a heap, a standard panic handler, types, etc. - `kona-preimage` contains a `no_std` implementation of the Preimage ABI, consumable by `client` programs that will be ran on top of the FPVMs. *Definition of Done*: 1. The `kona-common` crate contains complete implementations of utilities for performing common kernel interactions, bootstrapping the program, etc. 2. The `kona-preimage` crate contains a complete, client-side implementation of the preimage oracle ABI. 3. Accompanying test programs utilize these crates and successfully run on top of both `cannon` and `asterisc` in their respective e2e tests. (Can mirror existing minimal golang tests.) **Milestone 4: `kona` client & host programs** Implement the [fault proof program](https://github.com/ethereum-optimism/optimism/blob/develop/specs/fault-proof.md#fault-proof-program) specification using the crates created in milestone 3. This should encompass the `client` and `host` programs, and this is the largest milestone during `kona` development. This milestone specifically should be spiked and broken up much further, as there's a ton that can be parallelized within it. #### [`asterisc`][asterisc] **Workstream Total Estimate: 3 (+ 66%) engineering weeks**. High margin left for bugs found in Asterisc, learning RISC-V, any necessary refactoring in the `op-challenger` to better support switching VM backends based on the game type.  ![mermaid-diagram-2023-12-29-203245](https://hackmd.io/_uploads/ByiL7l6PT.svg) **Milestone 0: Transfer ownership** Move `asterisc` into the `ethereum-optimism` org on GitHub for easy access by the proofs team. *Note*: Pending Proto's approval, code belongs to him. Can work around it if he'd prefer to retain ownership. **Milestone 1: Binary** Add a binary to `asterisc` that can: 1. Load ELF files and run them, identically in functionality to `cannon`. 2. Patch ELF files. 3. Build a VM snapshot from an ELF file. This binary will be used by the `op-challenger` to invoke `asterisc`, as well as end-users to generate the absolute prestate snapshot, etc. *Definition of Done*: 1. The binary has been created with the above capabilities. 2. Each capability has been tested for correctness, similar to the `cannon` binary's tests. **Milestone 2: Contracts Integration** Pull in the `asterisc` solidity VM from the `rvsol` package as a dependency in `contracts-bedrock`, and register it as a new dispute game type within the `DisputeGameFactory` for testing. *Definition of Done:* 1. The `DisputeGameFactory` sets the implementation of a new game type to a version of the `FaultDisputeGame` that is backed by the `asterisc` VM in the `contracts-bedrock` deploy scripts. **Milestone 3: `op-challenger` Integration** Add a new `TraceProvider` for the `asterisc` VM to the `op-challenger`, enabling it to play a version of the `FaultDisputeGame` backed by the `asterisc` VM rather than `cannon`. The challenger should be able to decide which VM it uses based on the game type, and all other logic should remain identical. *Definition of Done:* 1. The `op-challenger` has a new `TraceProvider` that allows for generating state witness hashes from the `op-program` running on top of `asterisc`. 2. `op-e2e` includes tests for `FaultDisputeGame`s with the `asterisc` VM. **Milestone 4: Finish `c` extension support** The `c` extension is commonly enabled in Rust targets with LTS, specifically the bare-metal riscv64gc ELF target `kona` uses. Support for this extension has been added to the `fast` version of `asterisc` by clabby, and will need to be added to the `slow` and `solidity` version of the VM as well. Open PR: https://github.com/protolambda/asterisc/pull/6 Background: https://hackmd.io/pjMmcetGRqWvQb0uN0xgTg *Definition of Done:* 1. The RISC-V `c` extension has been added to the native fast/slow VM as well as the solidity VM and tested with the standard RISC-V compliance tests. 2. An example Rust program using `kona`'s build pipeline for the riscv64gc target is added to the `asterisc` tests. ## Risks The initial risk assessment of the project can be found [here](https://hackmd.io/@iQAf_6OqRmauDQ8jyj2J5Q/S1zNTsUXa). ## Security Considerations 1. **`reth` + `revm` execution engine integrity**: `op-reth` is a nascent Optimism execution client. Support and adoption for it has grown over the past few months, and it has been tested well at this point, though it is not as battle hardened as `op-geth`. `reth` itself will be receiving an audit from Sigmaprime in the coming months, and this will not include the Optimism changes, but this will ease concerns over bugs in the broader codebase. i. We should also be running `op-reth` nodes. 2. **`axos` integrity**: `axos` will be a new implementation of the derivation pipeline, intended to take over the role of `magi`'s implementation in the Rust ecosystem. To ensure `axos` has coverage, we should coordinate with `a16z` and possibly RiscZero to integrate with `axos` and assist in covering it. 3. **`kona` integration testing**: As `kona` will be pulling together several nascent components, it will be critical to integration test early to give it some shelf life. We'll need to set up scheduled jobs on Sepolia, similar to the ongoing Goerli FPP job that runs today in CircleCI. 4. **`asterisc` integrity**: This project charter takes `asterisc` into a "production ready" state, which should involve early e2e tests for a `FaultDisputeGame` backed by the `asterisc` VM. It will be important to integrate as early as possible so that we can run the `op-program` on top of it in both `op-e2e` and as a scheduled job with inputs from a live testnet. ## Success Evaluation By the end of this project, all components below should be complete, with full support the current protocol version as of the project's ending date, and ready to enter a scrutinous audit: - [ ] Alternative fault proof program (`kona`) - [ ] Alternative execution client (`op-reth` / `revm`) - [ ] Alternative derivation pipeline (`axos`) - [ ] Alternative preimage oracle bindings (`kona`) - [ ] Alternative fault proof VM (`asterisc`) - [ ] Integrated with the dispute game contracts - [ ] Integrated with the `op-challenger` - [ ] Supports the RISC-V `c` extension [magi]: github.com/a16z/magi [cannon-rs]: github.com/anton-rs/cannon-rs [revm]: github.com/bluealloy/revm [reth]: github.com/paradigmxyz/reth [kona]: github.com/ethereum-optimism/kona [axos]: github.com/ethereum-optimism/axos [asterisc]: github.com/protolambda/asterisc