# Benchmarking zkWASM performance with comparison to Risc0 and SP1

## Abstract
A zkVM is a virtual machine that adopts zero-knowledge-proof (ZKP) scheme, accepts general-purpose Turing complete language and generates Succinct Non-Interactive Argument of Knowledge (SNARK), which would be easy to verified. Nowadays there are quite many zkVMs emerging, yet few people knows their performance difference and applicable scenarios. This proposal is aiming to examine popular zkVMs by conducting a fair comparsion, including [risc0](https://github.com/risc0/risc0/), [sp1](https://github.com/succinctlabs/sp1) and [zkWASM](https://github.com/DelphinusLab/zkWasm), which are most popular and production-mature ones. Succinct Lab (creator of SP1) has provided a report for comparison between SP1 and RISC0 ([source](https://blog.succinct.xyz/introducing-sp1/)). Further, we complete last puzzle by benchmarking zkWASM in same scheme. By the end of this work, we hope to provide a comprehensive overview to today's blooming ZKP ecosystem.

## Objectives
We will conduct the comparison in following dimensions:
### Metrics
1. On provider side, metrics like proving time, proof size, energy cost would be collected;
2. On verifier side, metrics like verifying time, energy cost would be collected;
3. (Optional) For those providing on-chain verifier, the gas fee would be examined as well.
### Benchmark Workload
Benchmark tasks in small, medium and large size are being provided.
Small: Fibonacci
Medium: Merkle Proofs Verification
Large: [OP Fraud Proof](https://github.com/ethereum-optimism/optimism/tree/develop/op-program) or [Tendermint Light Client](https://docs.tendermint.com/v0.34/tendermint-core/light-client.html). Caution, this is might be optional since some zkVMs might not be feasible for holding too large source instructions.
### Environment
Two test platforms:
1. Personal laptop: MacBook Pro with M1/M2/M3 Chip (CPU benchmarking)
2. Server: equipped with over 32 cores, over 256GB RAM, Nvdia GeForce RTX 4090 and above (GPU benchmarking, if any)

## Deliverable
1. A report detailedly documenting metrics listed in `Objective` section.
2. A github repo including:
a. all reproducible benchmark test cases
b. all intermidate files and scripts associated with corresponding zkVMs.

## Grant Scope
In `Objective` section, scope is well described. Here it might be necessary to declaim what is NOT under scope of this grant:
1. Any optimization or customization is not included
2. A simple analysis on why performance differentiates could be present. While a detailed root cause analysis and binary profiling are not included
3. Some zkVMs are not included in this research, such as [Miden](https://github.com/0xPolygonMiden/miden-vm) and [Cario](https://github.com/starkware-libs/cairo). Miden is adopting [Miden Assembly](https://github.com/0xPolygonMiden/miden-vm/tree/main/assembly) for now and Cario itself is a [programing language](https://docs.cairo-lang.org/hello_cairo/intro.html). For those two, one has to write specifc assembly or high-level specific language to profile the performance. Thus a fair comparison between a specific language written program and a general rust program could be hard to conducted.

## Project Team
Qi Zhou, project lead, 20 hours per month for guidance and reviewing
Frank Liu, researcher, 40 hours per month for implementation and profiling
Po, researcher, 40 hours per month for implementation and profiling

## Background
Qi Zhou: Dr. Qi Zhou is a prominent contributor to the Ethereum ecosystem, known for authoring and co-authoring several EIPs, including EIP-4804, EIP-4972, EIP-5018, and EIP-5027, among others. His research interests revolve around building the next generation of Web3 infrastructure, with a focus on areas such as data availability, layer 2 scaling, decentralized storage, and token economics. Dr. Zhou has been recognized for his contributions with multiple grants from the Ethereum Foundation, which have supported his research into topics such as [data availability sampling](https://blog.ethereum.org/2022/12/07/esp-allocation-q3-22) and [proof of L2 storage using L1 contracts](https://blog.ethereum.org/2023/02/14/layer-2-grants-roundup) ([EthStorage](https://ethstorage.io/#/)).

Frank Liu: Frank is a core developer of ETHStorage and a researcher on ZKP. His work includes driving developing [zkGO](https://perfect-amphibian-929.notion.site/Advancing-Towards-ZK-Fraud-Proof-zkGo-Compiling-L2-Geth-into-ZK-Compatible-Wasm-315a878af5754c549e5003568e1ee124), [EthStorage L1 Protocol](https://github.com/ethstorage/storage-contracts), [zkWASM continuation](https://github.com/ethstorage/zkWasm/tree/frank/cont_script) (forked from [zkWASM](https://github.com/DelphinusLab/zkWasm)), among others. He also involves research on [data availability sampling](https://blog.ethereum.org/2022/12/07/esp-allocation-q3-22) and [proof of L2 storage using L1 contracts](https://blog.ethereum.org/2023/02/14/layer-2-grants-roundup). He has 8 years background on database industry and system programing, focusing on performance optimization, memory management, scheduling and hardware acceleration. Now his interest in decentralized protocol, tiken economics and zero-knowledge-proof. At leisure time, he writes blogs about [zkp things](https://hackmd.io/@FrankJLiu/Sk61V29Mp).

Po: Dr. Po is a core developer of ETHStorage. Po contributes to zk-related research to lower the cost of proof of storage, a critical component of ETHStorage. Additionally, he is an active member of [Antalpha-Labs](https://github.com/Antalpha-Labs), a prominent web3 developer community. He dedicates his time to sharing knowledge and writing about zk-related topics. Furthermore, he stays updated on recent zk-related research such as Nova, HyperNova and Protostar.

## Methodology
First, we establish general benchmarks in Rust that can be applied across all zkVMs. Rust is recognized for its utilization of LLVM as a codegen backend, capable of generating a variety of assembly forms, including WASM, MIPS, RISC-V, among others. This versatility meets the diverse instruction requirements of different zkVMs, providing a solid foundation for equitable comparison.
Then, we reproduce what report from Succinct Lab has documented, and verify if data matches.
Next, we use same rust program to generate WASM, which is the input of zkWASM. Profiling zkWASM with those tests, we could get a fair comparison between three types of zkVMs.
Last step we write report up.

## Cost
Researchers Cost: 1000 USD per week ([data point](https://www.glassdoor.com/job-listing/zero-knowledge-proofs-zkp-engineer-palo-verde-consulting-JV_IC1147420_KO0,34_KE35,56.htm?jl=1008321141789))
Hardware Cost: 600 USD per month (tax included) ([source](https://www.gpu-mart.com/rtx-4090-hosting))

## Timeline
Milestone 1: Benchmark write-up and test
Hours: 3 weeks
Budget: 1,000 x 3 = 3,000

Milestone 2: reproduce Succinct Lab [results](https://blog.succinct.xyz/introducing-sp1/)
Hours: 1 weeks
Budget: 1,000 x 1 + 600 = 1,600

Milestone 3: Profiling on zkWASM
Hours: 4 weeks
Budget: 1,000 x 4 + 600 = 4,600

Milestone 4: Data validation and reconciliation and doc write-up
Hours: 3 weeks
Budget: 1,000 x 3 + 600 = 3,600

## Budget
Support research: $12,800
Communication (software/cell) and traveling: $5,000
Other staff cost: $2,000
Toal: $19,800