# Understanding ZKsync Prover

## Introduction
In this blog, we explore the workflow of the ZKsync prover system. We break down how GPU proof generation works, including the roles of the witness generator, witness vector generator, and circuit prover. Additionally, we cover how different components interact to optimize proof generation while efficiently managing system resources.

## General Overview

First, we will examine the proof generation process using an A100 + RTX 4000 series GPU, coupled with GCS as the object store for storing proof generation artifacts.

To test proof generation, we generated a proof for the mainnet batch `496134`, with inputs fetched via the `external_proving_api`.

### Total Proof Generation Time
- **~3.5 hours**

### Batch Processing Statistics
- **Jobs per aggregation round:**
- **agg_round 0 (BasicCircuits):** 17,061
- **agg_round 1 (LeafAggregation):** 544
- **agg_round 2 (NodeAggregation):** 32
- **agg_round 3 (RecursionTip):** 1
- **agg_round 4 (Scheduler):** 1
- **Total jobs:** 17,639

### Job Prioritization
- **Heavy WVG:** Prioritizes node aggregation rounds first; processes other rounds if no node jobs are present.
- **Light WVG:** Processes all jobs except for node aggregation jobs.
- **Node Proofs:** Heavy jobs (~9GB RAM), while others are lighter (~2GB RAM).

### Communication and Storage
- **Communication:** Between the witness vector generator (WVG) and GPU prover via the MPSC channel.
- **Storage:**
- `object_store.put` to save data to object storage.
- `object_store.get` to retrieve data from object storage.

## Steps for ZKSync Proof Generation

In this section, we go through the proof generation process using the master and slave nodes.

The **master node** runs the following components:
1. Job Monitor
2. Prometheus
3. Grafana
4. PostgreSQL Database
5. Witness Generator
6. Compressor

The **slave node** runs only the `zksync_circuit_prover` (WVG + GpuCircuitProver).

### Master Node Setup

1. Follow the setup instructions [here](https://matter-labs.github.io/zksync-era/prover/latest/), ensuring that you complete steps 1, 2, and 5 up to the "Initializing System" section.
2. Download the `witness_input.bin` file from the following link and upload it to the `witness_inputs` folder in your storage bucket:

    https://prover-api.zksync.io/proof_generation_data

3. Fetch prover information and add it to the database using `prover_cli`:

    zkstack dev prover info

    prover_cli <DATABASE_URL> insert-version --version=<MINOR_VERSION> --patch=<PATCH_VERSION> --snark-wrapper=<SNARK_WRAPPER>

4. Insert the batch to be proven into the database:

    prover_cli postgres://*****:*****@localhost:5432/zksync_prover_localhost_gcsera insert-batch --number=***** --version=25 --patch=0

5. Log in to your GCloud account:

    gcloud auth application-default login

6. Start the monitor, witness generator, and compressor using the following commands:

    zkstack prover run --component=witness-generator --round=all-rounds
    zkstack prover run --component=compressor
    zkstack prover run --component=prover-job-monitor

### Slave GPU prover Setup

Ensure the following tools and dependencies are installed on your system:

1. **gcloud CLI**: [Installation Guide](https://cloud.google.com/sdk/docs/install#linux)
2. **Docker**: [Installation Guide](https://docs.docker.com/engine/install/ubuntu)
3. **CUDA Toolkit + Drivers (Minimum Version: v12.4)**: [Download CUDA](https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=22.04&target_type=deb_network)

#### Verifying CUDA Installation

After installation, verify that CUDA is correctly set up:

1. **Check if `nvcc` is available:**

    which nvcc

If `nvcc` is not found, add it to your PATH:

    echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
    source ~/.bashrc

2. **Verify `nvcc` version:**

    nvcc --version

Expected output:

    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2024 NVIDIA Corporation
    Built on Thu_Mar_28_02:18:24_PDT_2024
    Cuda compilation tools, release 12.4, V12.4.131
    Build cuda_12.4.r12.4/compiler.34097967_0

### Starting the Slave GPU Prover

1. **Log in to your GCloud account:**

    gcloud auth application-default login

2. **Clone the `era_prover` repository:**

    git clone https://github.com/marlinprotocol/era_prover.git

**Note:** All subsequent commands should be executed in the `era_prover` directory.
3. **Download the setup keys into the `era_prover` directory:**

    gsutil cp -r gs://matterlabs-setup-data-us/dc3bba7-gpu/* data/keys/

4. **Run the prover using Docker:**

    docker-compose up --build

## Internal Workings of ZKSync Circuit Prover (WVG + GpuCircuitProver)

### ZKSync Prover Flow

**Key Components:**
- **MetadataLoader:** Differentiates between heavy and light WVG runners.
- **Job Runner:** Manages picker, executor, and saver tasks.

### Steps:

1. **Load Resources:**
- Loads configuration files (`secret.yaml`, `general.yaml`, `object_store.config`).
2. **Start Witness Vector Generators:**
- **Initialize:**
- `WitnessVectorGeneratorExecutor`
- `WitnessVectorGeneratorJobPicker`
- `WitnessVectorGeneratorJobSaver`
- **Run Job Runner:**
1. `JobPickerTask` fetches new jobs and sends data over the MPSC channel (`input_tx`).
2. `WorkerPool` processes jobs, generates witnesses, and forwards them via `result_tx`.
3. `JobSaverTask` receives witness vectors and forwards them to `JobPicker` for the `circuit_prover_runner`.
3. **Start `circuit_prover_runner`:**
- **Initialize:**
- `GpuCircuitProverExecutor`
- `GpuCircuitProverJobPicker`
- `GpuCircuitProverJobSaver`
- **Run Job Runner:**
1. `JobPickerTask` loads setup data and sends jobs to the executor.
2. `WorkerPool` generates proofs and forwards them via `result_tx`.
3. `JobSaverTask` stores proofs in `object_store` and updates the database.

The prover gateway just pools a endpoint to find new witness input data and submit the proof once the proofs is generated.

## Internal workings of BasicCircuitWitnessGenerator

1. **Initializing the Circuit:**
- The process starts with calling `Circuit::new()` in `w_g/main.rs`.
- This is followed by invoking `Circuit::run()`, which subsequently calls the `run()` function for **BasicCircuitWitnessGenerator** in `queued_job_monitor/lib.rs`.

2. **Fetching the Next Job:**
- Inside `WitnessGenerator::<BasicCircuits>::run()`, the function `get_next_job()` (located in `w_g/rounds/mod.rs`) is invoked.
- This function further calls:
1. **`get_metadata()`** (`w_g/basic_circuits/mod.rs`):
- Calls `get_next_basic_circuit_witness_job`, which retrieves the **l1_batch_number** via an SQL query.
2. **`prepare_job()`** (`w_g/basic_circuits/mod.rs`):
- Calls `get_artifacts()` to fetch the required job data (**artifacts/WitnessInputData**) from the object store, using the **batch_number** obtained from `get_metadata()`.

3. **Processing the Job:**
- The function `process_job()` is executed, which takes the **job ID**, the retrieved **job data (artifacts/WitnessInputData)**, and the **current timestamp** as inputs.
- This process involves calling `generate_witness()`, which creates a **task handle** and performs the following tasks:
1. Prepares the input data.
2. Runs a **blocking** witness generation function.
3. Handles circuit and recursion queue processing asynchronously.
4. Saves **ClosedFormInputs** and **Circuits** into the object store.
5. Returns the following outputs:
- `circuit_urls`
- `recursion_urls`
- `scheduler_witness`
- `block_aux_witness`

4. **Waiting for Task Completion:**
- The `wait_for_task()` function is called to **poll the task handle** and **save its outcome**.
- It performs the following tasks:
1. Saves `scheduler_witness` and `block_aux_witness` to the object store.
2. Inserts **prover jobs** into the `prover_jobs_fri` table.
3. Creates **aggregation jobs** for the next round in the `leaf_aggregation_witness_jobs_fri` table.
4. Marks **witness jobs** as successful in the `witness_inputs_fri` table, completing the round.

## Conclusion

This blog explored the ZKSync Prover system, detailing its proof generation workflow, job prioritization, and system architecture across master and slave nodes. By understanding the internal workings of the Witness Vector Generator (WVG) and GPU Circuit Prover, we gain valuable insights into optimizing proof generation and resource management.

These findings will significantly aid in integrating ZKSync Prover with Kalypso, allowing us to streamline proof generation, enhance job scheduling, and efficiently manage GPU resources. Leveraging this knowledge, we can improve Kalypso’s compatibility with ZKSync, ensuring a more scalable and performant ZK proof generation pipeline.