# Benchmarking Performance of zkVMs

## Abstract
A zkVM is a virtual machine that adopts zero-knowledge-proof (ZKP) scheme, accepts general-purpose Turing complete language and generates Succinct Non-Interactive Argument of Knowledge (SNARK), which would be easy to verified. Nowadays there are quite many zkVMs emerging, yet few people knows their performance difference and applicable scenarios. This proposal is aiming to examine popular zkVMs by conducting a fair comparsion, including [risc0](https://github.com/risc0/risc0/), [sp1](https://github.com/succinctlabs/sp1), [zkWASM](https://github.com/DelphinusLab/zkWasm), [Miden](https://github.com/0xPolygonMiden/miden-vm) and [Cario](https://github.com/starkware-libs/cairo)(generally). By the end of this work, we hope to provide a comprehensive overview to today's blooming ZKP ecosystem.

## Objectives
We will conduct the comparison in following dimensions:
### Metrics
1. On provider side, metrics like proving time, proof size, energy cost would be collected;
2. On verifier side, metrics like verifying time, energy cost would be collected
### Benchmark Workload
Benchmark tasks in small, medium and large size are being provided.
Small: Fibonacci
Medium: Merkle Proofs Verification
Large: [OP Fraud Proof](https://github.com/ethereum-optimism/optimism/tree/develop/op-program) or [Tendermint Light Client](https://docs.tendermint.com/v0.34/tendermint-core/light-client.html). Caution, this is might be optional since some zkVMs might not be feasible for holding too large source instructions.
### Environment
Two test platforms:
1. Personal laptop: MacBook Pro with M1/M2/M3 Chip
2. Server: equipped with over 32 cores, over 256GB RAM, Nvdia GeForce RTX 4090 and above

## Deliverable
1. A report detailedly documenting metrics listed in `Objective` section.
2. A github repo including:
a. all reproducible benchmark test cases
b. all intermidate files and scripts associated with corresponding zkVMs.

## Grant Scope
In `Objective` section, scope is well described. Here it might be necessary to declaim what is NOT under scope of this grant:
1. Any optimization or customization is not included
2. A simple analysis on why performance differentiates could be present. While a detailed root cause analysis and binary profiling are not included
3. Miden is adopting [Miden Assembly](https://github.com/0xPolygonMiden/miden-vm/tree/main/assembly) and Cario itself is a [programing language](https://docs.cairo-lang.org/hello_cairo/intro.html). For those two, writing a larger program might not be feasible.

## Project Team
Qi Zhou, project lead, 20 hours per month for guidance and reviewing
Frank Liu, researcher, 40 hours per month for implementation and profiling
Po, researcher, 40 hours per month for implementation and profiling

## Background
Qi Zhou: Dr. Qi Zhou is a prominent contributor to the Ethereum ecosystem, known for authoring and co-authoring several EIPs, including EIP-4804, EIP-4972, EIP-5018, and EIP-5027, among others. His research interests revolve around building the next generation of Web3 infrastructure, with a focus on areas such as data availability, layer 2 scaling, decentralized storage, and token economics. Dr. Zhou has been recognized for his contributions with multiple grants from the Ethereum Foundation, which have supported his research into topics such as [data availability sampling](https://blog.ethereum.org/2022/12/07/esp-allocation-q3-22) and [proof of L2 storage using L1 contracts](https://blog.ethereum.org/2023/02/14/layer-2-grants-roundup) ([EthStorage](https://ethstorage.io/#/)).

Frank Liu: Frank is a core developer of ETHStorage and a researcher on ZKP. His work includes driving developing [zkGO](https://perfect-amphibian-929.notion.site/Advancing-Towards-ZK-Fraud-Proof-zkGo-Compiling-L2-Geth-into-ZK-Compatible-Wasm-315a878af5754c549e5003568e1ee124), [EthStorage L1 Protocol](https://github.com/ethstorage/storage-contracts), [zkWASM continuation](https://github.com/ethstorage/zkWasm/tree/frank/cont_script) (forked from [zkWASM](https://github.com/DelphinusLab/zkWasm)), among others. He also involves research on [data availability sampling](https://blog.ethereum.org/2022/12/07/esp-allocation-q3-22) and [proof of L2 storage using L1 contracts](https://blog.ethereum.org/2023/02/14/layer-2-grants-roundup). He has 8 years background on database industry and system programing, focusing on performance optimization, memory management, scheduling and hardware acceleration. Now his interest in decentralized protocol, tiken economics and zero-knowledge-proof. At leisure time, he writes blogs about [zkp things](https://hackmd.io/@FrankJLiu/Sk61V29Mp).

Po: Dr. Po is a core developer of ETHStorage. Po contributes to zk-related research to lower the cost of proof of storage, a critical component of ETHStorage. Additionally, he is an active member of [Antalpha-Labs](https://github.com/Antalpha-Labs), a prominent web3 developer community. He dedicates his time to sharing knowledge and writing about zk-related topics. Furthermore, he stays updated on recent zk-related research such as Nova, HyperNova and Protostar.

## Methodology
First, we establish general benchmarks in Rust that can be applied across all zkVMs. Rust is recognized for its utilization of LLVM as a codegen backend, capable of generating a variety of assembly forms, including WASM, MIPS, RISC-V, among others. This versatility meets the diverse instruction requirements of different zkVMs, providing a solid foundation for equitable comparison.
Then, we examine each zkVMs. Succinct Lab (creator of SP1) has provided a report for comparison between SP1 and RISC0 ([source](https://blog.succinct.xyz/introducing-sp1/)). For those two, we reproduce the workload and verify if data matches.
Different zkVM accepts different instructions. For risc0(RISC-V), sp1(rust) and zkWASM (WASM), the test case can be derived from same rust file. For Miden and Cario, we have to write specific programs. The fair comparison is merely based on semantical consistency among dedicated asm and general rust program.

## Cost
Researchers Cost: 1000 USD per week ([data point](https://www.glassdoor.com/job-listing/zero-knowledge-proofs-zkp-engineer-palo-verde-consulting-JV_IC1147420_KO0,34_KE35,56.htm?jl=1008321141789))
Hardware Cost: 600 USD per month (tax included) ([source](https://www.gpu-mart.com/rtx-4090-hosting))

## Timeline
Milestone 1: Benchmark write-up
Hours: 3 weeks
Budget: 1,000 x 3 = 3,000

Milestone 2: reproduce Succinct Lab [results](https://blog.succinct.xyz/introducing-sp1/)
Hours: 1 weeks
Budget: 1,000 x 1 + 600 = 1,600

Milestone 3: Profiling on zkWASM
Hours: 4 weeks
Budget: 1,000 x 4 + 600 = 4,600

Milestone 4: Profiling on Miden
Hours: 4 weeks
Budget: 1,000 x 4 + 600 = 4,600
Note: Only attempting writing small and medium size benchmark

Milestone 5: Profiling on Cario
Hours: 4 weeks
Budget: 1,000 x 4 + 600 = 4,600
Note: Only attempting writing small and medium size benchmark

Milestone 6: Data collection and write-up
Hours: 3 weeks
Budget: 1,000 x 3 + 600 = 3,600

## Budget
Support research: $22,000
Communication (software/cell) and traveling: $5,000
Other staff cost: $2,000
Toal: $29,000