# Extended Benchmarks Planning(v1)

## Goal
Basically, we want to build the comprehensive benchmarks. This should be helpful to anyone who wants to explore the popular and SOTA proving systems from the perspective of the client side proving.

## Existing zkID-Benchmark
1. Proving systems
- Criteria
- Transparent(no trusted setup)
- Small proof size
- Fast proving
- Post-quantum soundness

- List
- Binius (Binius)
- Plonky2 (TurboPlonk + FRI)
- Plonky3 (TurboPlonk + FRI)
- STWO(circle-STARK)
- ligero (ligero)
- Polyhedra Expandra(Libra + Orion PCS)

2. Target circuit
- Criteria
This benchmark is planned from the requirement of ZK ID project. In our setting, the issuer signs a SHA-256 hash of an SD-JWT containing the credential attributes. A typical SD-JWT in this application is about 2 kB in size, so we benchmarked SHA-256 circuits that hash 2 kB of input data.

- List
- SHA256(SHA2-256) on 2kB of input data

## What to do
1. Add more proving systems
- Selection criteria
- popular proving systems (we're missing)
- stable and production-level
- SOTA new releases

- List of candidates
- Groth16(circom)
- Halo2(Zcash-Halo2 & PSE-Halo2)
- Spartan(2)
- Nova
- HyperNova
- HyperPlonk
- LatticeFold
- Spartan-WHIR
- GKR-WHIR(Ceno)
- SpeedySpartan
- Ligerito
- Binary-GKR(polyhedra)
- ...
- One more category to consider is [zkVM projects](https://x.com/FlyingNobita/status/1925928483856785575)

2. Add more target circuits
- Selection criteria
- popular benchmark circuits (we're missing)
- popular cryptographic primitives used in the Internet
- popular cryptographic primitives used in blockchain industry

- List of candidates
- Hashing: Keccak, Poseidon, Poseidon2, Neptune, GMiMC
- EC operations: EC point addition, scalar multiplication, MSM(digital signature, key agreement)
- Digital Signature(circuits verify ECDSA, EdDSA)
- Merkle Tree Membership Proofs (privacy-focused app like mixer, rollup)
- Proof composition(circuits that verify other proofs)
- Proof Aggregation(combining multiple proofs into one)

3. Improve the benchmark function
- Goal
Make benchmarks more robust technically
- Criteria
- Independent of machine
- (semi-)auto release of quarterly report

## Action Plan
We are trying to reach the goal in multiple phases.

- 1st Phase
First, we need to improve the benchmark codebase to meet the goal.
Then, We'd like to cover the most essential stuff - proof systems and circuits which need no or less tweak.
Details: https://hackmd.io/ujg9XQvxToCG3jlMEyqo1A?view

- 2nd Phase
Here, we try to cover the remaining proving systems and circuits.
It needs extra consideration/planning since it can involve the implementation of proving system and circuits. (See NOTE 1.)
**IMPORTANT**: We can extend the benchmark by adding several features. For example, we can add "usability" criteria to benchmark table. It can show how usable are the certain proof system or circuits, in terms of compatibility, easy-to-use, etc.

## NOTE
1. Possible implementation of proving system and circuits
There is possibility that the target circuits and proving systems are not good for direct benchmarking.
In such case, we should be able to re-write or tweak the proving system or circuits as much as we expect.
For example, current Ligerito proving scheme is written in Julia language. Also, their implementation is not good for benchmarking any circuits other than their hand-written case.
Then, we need to re-write the Ligerito with Rustlang for benchmarking.

2. Reuse of existing benchmarks and frameworks
We can re-use the existing benchmarks, their open-source code, etc.
The best example is **zk-Harness**.
[zk-Harness](https://www.zk-bench.org/) is an existing third-party benchmark for a bit comprehensive ZK circuits.
Their benchmarks include the following:
- circuit benchmark
- various curves(bn254, bn128, bls12_377, bls24_315, bw6_761)
- frameworks(circom/snarkjs, gnark)
- backend(groth16, plonk)
- circuit(cubic, expo, exponentiate, sha256)
- metric(time, RAM)
- Arithmetic benchmark
- various curves(bn254, bw6_633, bn128, bw6_761, bls12_377, bls24_315, bls12_381)
- frameworks(gnark, snarkjs)
- field(base, scalar)
- op(add, inv, mul, sub, exp)
- metric(time, RAM)
- Elliptic Curve Benchmarks
- curve(same as above)
- frameworks(gnark, snarkjs)
- op(pairing, g1_scalar_mul, g1_MSM, g2_scalar_mul, g2_MSM)
- metric(time, RAM)

We can also reference the following:
- https://zkbenchmarks.com/
- https://blog.ezkl.xyz/post/benchmarks/
- https://medium.com/@ingonyama/zk-benchmark-toolkit-35b87e74621d
- https://github.com/codex-storage/zk-benchmarks/tree/master
- https://zkbench.dev/
- https://github.com/delendum-xyz/zk-benchmarkinghttps://github.com/delendum-xyz/zk-benchmarking
- https://hackmd.io/@Cysic/r1LoS0MmJg