# Prover Integration Guide

*This document is meant for infrastructure providers who intend to run a prover in the Aztec Network testnet. It covers the architecture of the prover node, suggestions for integration, and instructions on running Aztec's prover node client.*

## Overview

Both sequencing and proving in the Aztec Network is intended to be fully decentralized. We expect sequencers to submit blocks to L1 every ~30 seconds, and provers to prove batches of 32 blocks to finalize the L2 chain. Sequencers will be chosen via a random election, while provers will be selected by sequencers via an out-of-protocol sequencer-prover coordination mechanism.

For the purpose of proving testing, the network will be led by a single sequencer, that will choose the prover for each epoch based on the bids submitted.

## Rollup Circuits

Aztec's circuits are structured such that a single **root rollup proof** is produced that attests to the correct execution of a sequence of transactions throughout an **epoch**. An **epoch** is defined as 32 consecutive slots.

Proofs in Aztec are initially split into two kinds: **client** and **server** proofs.

**Client proofs** are generated by users in their own devices when they send a transaction, and use the ClientIVC proving system. These proofs are zero-knowledge, and prove the correct execution of smart contract functions on the client, attesting to the validity of the transaction side effects (emitted note hashes, nullifiers, encrypted logs, etc).

**Server proofs** are generated by provers. These are Honk proofs, and can be roughly split into 3 domains:
- **Transaction proving**: This proves correct execution of a transaction's public functions, via the **AVM** and **public kernel** circuit. It then verifies and converts the ClientIVC proof into a Honk proof using a **tube** circuit that can be aggregated into the rollup tree.
- **Rollup proving**: The final set of side effects of each transaction is fed into the **base rollup** circuit which forms the leaf level of a binary tree of aggregated proofs, which get iteratively **merged** and ultimately result in a single **root rollup** proof that is verified on L1.
- **Parity circuits**: This tree-like set of circuits is used exclusively for cross-chain messaging, and its output is then fed into the block root rollup circuit.

Note that generating each proof requires two steps:
- The circuit is first simulated given a set of inputs to produce the witness, using the ACVM or AVM simulator.
- The partial witness is then provided, along with the circuit bytecode and optionally a proof aggregation object, to the barretenberg binary to generate the actual proof.

### Parity proving

The parity circuits attest to cross-chain messaging. All L1 to L2 messages in a given block are batched and processed by individual **base parity** circuits, whose output is then aggregated into a **root parity** circuit. This process is repeated for every block in the epoch.

graph BT
    I0((Msgs 00-03))
    I1((Msgs 04-07))
    I2((Msgs 08-11))
    I3((Msgs 12-15))
    
    I0 --> B0[Base Parity]
    I1 --> B1[Base Parity]
    I2 --> B2[Base Parity]
    I3 --> B3[Base Parity]
    
    R[Root Parity]
    B0 --> R
    B1 --> R
    B2 --> R
    B3 --> R

### Transaction proving

Each transaction submitted by a user has an associated ClientIVC proof. The first step in proving is to transform this proof into a Honk proof that can be recursively fed into other circuits. This is handled by the initial **tube** circuit.

Then, a transaction may have multiple **public function calls** (or none) that are executed by the sequencer. Each of these public executions need to be proven using the **AVM circuit**, the result of each then being passed onto a **public kernel** circuit, and then ultimately into **public kernel tail** circuit.

If the transaction has no public function calls, then the output of the tube circuit is fed directly into the next stage of proving.

graph BT
    ClientIVC(ClientIVC) --> Tube[Tube]
    Tube --> Pub1[Public Kernel]
    AVM1[AVM] --> Pub1
    AVM2[AVM] --> Pub2[Public Kernel]
    Pub1 --> Pub2
    AVM3[AVM] --> Pub3[Public Kernel]
    Pub2 --> Pub3
    Pub3 --> PubT[Public Kernel Tail]

### Rollup proving

The output of each transaction is passed onto a **base rollup circuit**. These circuits then get aggregated into a binary tree via **merge circuits**, until a **block root rollup circuit** that attests to the validity of an entire block.

graph BT
    Tube1[Tube\nTx=0x1] --> B1[Base Rollup]
    Pub2[Public Kernel Tail\nTx=0x2] --> B2[Base Rollup]
    Pub3[Public Kernel Tail\nTx=0x3] --> B3[Base Rollup]
    Pub4[Public Kernel Tail\nTx=0x4] --> B4[Base Rollup]
    PR[Root Parity] --> BR
    B1 --> M1[Merge Rollup]
    B2 --> M1
    B3 --> M2[Merge Rollup]
    B4 --> M2
    M1 --> BR[Block Root Rollup]
    M2 --> BR

Block root rollups are then again merged into a binary tree using **block merge rollup circuits**, until they are merged into a **root rollup circuit** that proves the validity of the entire epoch.

graph BT
    B1[Block Root 1] --> M1[Block Merge Rollup]
    B2[Block Root 2] --> M1
    B3[Block Root 3] --> M2[Block Merge Rollup]
    B4[Block Root 4] --> M2
    M1 --> BR[Root Rollup]
    M2 --> BR

### Putting it all together

This is the entire set of circuits that need to be recursively proven in order to complete the proof for a given epoch.

graph BT
    subgraph "Block 1"
    direction BT
        BR[Block Root 1]

        subgraph "Parity"
        direction BT
            I0((Msgs 00-03))
            I1((Msgs 04-07))
            I2((Msgs 08-11))
            I3((Msgs 12-15))

            I0 --> BPar0[Base Parity]
            I1 --> BPar1[Base Parity]
            I2 --> BPar2[Base Parity]
            I3 --> BPar3[Base Parity]

            RPar[Root Parity]
            BPar0 --> RPar
            BPar1 --> RPar
            BPar2 --> RPar
            BPar3 --> RPar
        end

        subgraph "Tx with no public fns"
            ClientIVC1(Client IVC\nTx=0x1) --> Tube1[Tube Circuit]
        end
        Tube1 --> B1[Base Rollup]

        subgraph "Tx with public fns"
            ClientIVC2(Client IVC\nTx=0x2) --> Tube2[Tube Circuit]
            Tube2 --> Pub1[Public Kernel]
            AVM1[AVM] --> Pub1
            AVM2[AVM] --> Pub2[Public Kernel]
            Pub1 --> Pub2
            Pub2 --> PubT2[Public Kernel Tail]
        end
        PubT2[Public Kernel Tail\nTx=0x2] --> B2[Base Rollup]

        PubT3[Public Kernel Tail\nTx=0x3] --> B3[Base Rollup]
        PubT4[Public Kernel Tail\nTx=0x4] --> B4[Base Rollup]
        B1 --> M1[Merge Rollup]
        B2 --> M1
        B3 --> M2[Merge Rollup]
        B4 --> M2
    end
    
    RPar --> BR
    M1 --> BR
    M2 --> BR
    
    Block2[Block Root 2]
    Block3[Block Root 3]
    Block4[Block Root 4]
    
    BlockMerge1[Block Merge 1]
    BlockMerge2[Block Merge 2]
    
    Root[Root Rollup]
    
    BR --> BlockMerge1
    Block2 --> BlockMerge1
    Block3 --> BlockMerge2
    Block4 --> BlockMerge2
    
    BlockMerge1 --> Root
    BlockMerge2 --> Root
    

## Prover Node

The Aztec client can be run as a **prover node**. In this mode, the client will automatically monitor L1 for unproven epochs and propose bids for proving them. Should a bid get accepted and posted to L1 by a sequencer, the prover node will kick off an **epoch proving job** which performs the following tasks:
- Downloads the transaction hashes in the epoch and all L1 to L2 messages from L1.
- Downloads the transaction objects with their ClientIVC proofs from a remote node (to be replaced by loading them from the P2P pool).
- Executes transactions in the epoch in order, generating proving jobs for each of them.
- Generates the inputs for each circuit and kicks off individual proving jobs to prover agents, recursively proving until it gets to the root rollup proof.
- Submits the root rollup proof to L1 to advance the proven chain.

graph
    style prover-node stroke:#333,stroke-width:4px

    prover-node[Prover Node]
    proving-job[Proving Job]
    tx-provider[Tx Provider]
    l1-publisher[L1 Publisher]
    l2-block-source[L2 Block Source]
    world-state[World State DB]
    tx-processor[Tx Processor]
    prover-client[Proving Orchestrator]
    proving-queue[Proof Broker]
    prover-agent[Prover Agent]
    bb[Barretenberg]

    prover-node --trigger--> proving-job
    proving-job --"process-tx"--> tx-processor --"add-tx"--> prover-client
    proving-job --start-batch--> prover-client
    proving-job --get-tx-hashes--> l2-block-source
    proving-job --"advance-to"--> world-state
    proving-job --"get-txs"--> tx-provider
    tx-processor --rw--> world-state
    world-state --"get-blocks"--> l2-block-source
    prover-client --"rw"--> world-state
    proving-job --publish-proof--> l1-publisher
    prover-client --"push-job"--> proving-queue
    %%prover-agent --"pull-job"--> proving-queue
        proving-queue <--"pull-jobs"--o prover-agent
    subgraph "Prover Agent"
        prover-agent --"prove"--> bb
    end

    %% p2p-client --> tx-pool --"save-tx"--> tx-db
    %% p2p-client --get-blocks--> l2-block-source

To achieve this, the prover node relies on the following components:

- **L2 block source**: Tracks L2 by monitoring the Rollup contract on L1 for new L2 blocks and proofs published, and downloads them.
- **Transaction provider**: Connects to a remote Aztec Node or to the P2P network directly to retrieve transactions with their client proofs.
- **World state**: Collection of merkle trees that represent the chain global state. It is updated from data collected from the L2 block source as well as from the execution of transactions.
- **Transaction processor**: Uses the Aztec VM simulator to execute public function calls from transactions and update world state accordingly.
- **L1 publisher**: Handles the submission of blocks and proofs to the L1 rollup contract.
- **Proving orchestrator**: Accepts executed transactions in a given block and dispatches proving jobs to a broker to produce the resulting root rollup proof.
- **Proof broker**: Takes individual proving requests from the Orchestrator and dispatches it to a fleet of prover agents.
- **Prover agent**: Multiple stateless processes that poll the proving queue for new jobs, execute them, and submit back the corresponding proof and outputs.

### Proving Orchestrator

The **proving orchestrator** is the main component of the prover node. The orchestrator receives each transaction as they are executed by the **transaction processor**, executes the tube, AVM, public kernel, and base rollup circuits for it, and then adds it to the rollup tree where it gets merged until the root rollup proof is obtained. The orchestrator also handles passing the generated proof from a circuit to the next one that aggregates it.

In short, **the orchestrator is responsible for turning an epoch proving job into a set of individual proving requests** that are fulfilled by a fleet of proving agents.

The orchestrator applies changes to the world-state as it goes through the received transactions (this is subject to change) and outputs proving jobs to a **proving broker**. Each individual proving job is comprised of an identifier of the circuit to be run and its inputs.

The types of proving jobs are the following:

- `PRIVATE_KERNEL_EMPTY`
- `PUBLIC_VM`
- `PUBLIC_KERNEL_NON_TAIL`
- `PUBLIC_KERNEL_TAIL`
- `BASE_ROLLUP`
- `MERGE_ROLLUP`
- `BLOCK_ROOT_ROLLUP`
- `BLOCK_MERGE_ROLLUP`
- `ROOT_ROLLUP`
- `BASE_PARITY`
- `ROOT_PARITY`
- `TUBE_PROOF`

### Prover Agent

:::warning
This interface will change with the new prover broker implementation being built [here](https://github.com/AztecProtocol/aztec-packages/pull/8609).
:::

The **prover agent** is a stateless process that consumes proving job requests and runs them through a prover (implementation [here](https://github.com/AztecProtocol/aztec-packages/blob/provernet/yarn-project/prover-client/src/prover-agent/prover-agent.ts)). The agent can run within the same process as the orchestrator, or on a separate machine, in which case it polls for new jobs via JSON-RPC over HTTP. Multiple prover agents can poll from the same queue to distribute the load.

See [here](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/circuit-types/src/interfaces/proving-job.ts#L140) for the proving queue interface, which can be called via an HTTP POST with the following JSON payloads as an example:

{ "method": "provingJobSource_getProvingJob" }
{ "method": "provingJobSource_heartbeat", "params": ["job-id"] }

The prover agent uses first the **ACVM simulator** to simulate the circuits and generate the witnesses based on the inputs provided. Refer to [this file](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/simulator/src/providers/acvm_native.ts) as a reference on how to invoke the ACVM CLI.

The prover agent then relies on **barretenberg**, Aztec's cryptographic library, for generating proofs. Barretenberg can be executed directly as an executable via its CLI. Refer to [this file](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/bb-prover/src/bb/execute.ts) as a reference on how to call barretenberg for generating each type of proof needed.

The relevant circuits can be found in the `noir-projects/noir-protocol-circuits` folder in the repository.

### Quote Provider

In order to start proving an epoch, a prover node must first submit a **quote** to the network, and it needs to be accepted. Quotes for proving a given epoch are generated and broadcasted once the epoch is finished. The sequencer is then responsible for selecting a quote and publishing it on L1 along with a block. The prover node monitors L1 and triggers the proving job only if they see their quote as accepted.

The **quote provider** is the component, within the prover node, responsible for generating a quote for a given epoch. The quote is expected to contain at least the following fields:
- `basisPointFee`: The fraction of the block rewards that the prover gets when the proof of epoch lands.
- `bondAmount`: The number of tokens that the prover must provide as a bond when it is selected for proving an epoch. The bond is taken from funds escrowed by the prover on L1. Must be at least `PROOF_COMMITMENT_MIN_BOND_AMOUNT_IN_TST=1000` tokens.

:::info
In development networks, provernet included, the prover node will automatically mint test tokens for itself to put in the escrow so they can be provided as bond.
:::

The quote provider defaults to always sending a quote, with `QUOTE_PROVIDER_BASIS_POINT_FEE=100` and `QUOTE_PROVIDER_BOND_AMOUNT=1000`, both configurable via environment variables.

Alternatively, the quote provider can issue an HTTP POST to a configurable `QUOTE_PROVIDER_URL` to get the quote. The request body is JSON-encoded and contains the following fields:
- `epochNumber`: The epoch number to prove
- `fromBlock`: The first block number of the epoch to prove
- `endBlock`: The last block number (inclusive) of the epoch to prove
- `txCount`: The total number of txs in the epoch
- `totalFees`: The accumulated total fees across all txs in the epoch

The response is expected also in JSON and to contain `basisPointFee` and `bondAmount` fields. If no response is sent, no quote will be sent by the prover for the epoch.

## Running a Prover Node

The Aztec client needed to run a prover node is shipped as a [docker image `aztecprotocol/aztec`](https://hub.docker.com/r/aztecprotocol/aztec). The image exposes the Aztec CLI as its `ENTRYPOINT`, which includes a `start` command for starting different components. You can download it directly or use the [sandbox scripts](https://docs.aztec.network/getting_started#install-the-sandbox) which will automatically pull the image and add the `aztec` shell script to your path.

Once the `aztec` command is available, you can run a **prover node** via:

aztec start --prover-node --archiver

To run a **prover agent**, either run `aztec start --prover`, or add the `--prover` flag to the command above to start an in-process prover.

### Configuration

The Aztec client is configured via environment variables, the following ones being relevant for the prover node:

- `ETHEREUM_HOST`: URL to an Ethereum node.
- `L1_CHAIN_ID`: Chain ID for the L1 Ethereum chain.
- `DATA_DIRECTORY`: Local folder where archive and world state data is stored.
- `AZTEC_PORT`: Port where the JSON-RPC APIs will be served.
- `PROVER_COORDINATION_NODE_URL`: URL to an Aztec node that acts as transaction provider and as source of truth for the L1 contract addresses. **Formerly `TX_PROVIDER_NODE_URL`.**
- `PROVER_PUBLISHER_PRIVATE_KEY`: Private key used for publishing proofs to L1. Ensure it corresponds to an address with ETH to pay for gas.
- `PROVER_AGENT_ENABLED`: Whether to run a prover agent within the node. We recommend setting to `false` and running prover agents in separate hosts.
- `PROVER_ID`: An identifier for the prover, which gets embedded into the proofs. Can be an arbitrary hex string.

The prover agent, on the other hand, relies on the following environment variables:

- `PROVER_JOB_SOURCE_URL`: URL to the prover node that acts as a proving job source. **Formerly `AZTEC_NODE_URL`.**
- `PROVER_AGENT_CONCURRENCY`: Maximum concurrency for this given prover agent. Defaults to `1`.

Both the prover node and agent also rely on the following:

- `PROVER_REAL_PROOFS`: Whether to generate actual proofs, as opposed to only simulating the circuit and outputting fake proofs. **Set to `true` for the scope of the integration.**
- `LOG_LEVEL`: One of `debug`, `verbose`, `info`, `warn`, or `error`.
- `LOG_JSON`: Set to `true` to output logs in ndJSON format (**unreleased**).
- `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`: Optional URL for pushing telemetry data to a remote OpenTelemetry data collector.

### Running a ProverNet locally

:::info
This docker compose template is to be replaced by a Kubernetes template to be runnable locally.
:::

To run a local network that emulates Provernet, we suggest using docker compose with the [template](https://github.com/AztecProtocol/aztec-packages/blob/provernet/docker-compose.provernet.yml) `docker-compose.provernet.yml` from the `AztecProtocol/aztec-packages` repository. This will kick off the following containers:
- An anvil instance that acts as L1
- A single Aztec sequencer node
- A bot that will send token transfers as test transactions as regular intervals using its own PXE
- A prover-node that will look for unproven blocks in anvil and attempt to prove them
- A single prover-agent that will pull proving jobs from the prover node, which can be scaled to multiple ones to speed up proving of larger blocks
- A block watcher that will periodically report the latest block and latest proven block in the chain

The docker compose template will accept the following env vars:
- `IMAGE`: Image and tag within the `aztecprotocol` docker registry to use. Defaults to `aztec:master`. **Use the `aztec` image `sha256:41ab5110d03cacb6850dab47df5871ecb136cf001b50e7dc01a02075c11a75eb` here.**
- `PROVER_REAL_PROOFS`: Whether to generate actual proofs. Set to false to speed up testing.
- `PROVER_TEST_DELAY_MS`: Artificial delay in ms for every proof generated. Consider setting to a non-zero value if `PROVER_REAL_PROOFS` is disabled.

To further tweak the behaviour of the network, consider modifying the following environment variables on the template:
- `BOT_TX_INTERVAL_SECONDS`: How many seconds should the bot wait between sent transactions
- `BOT_PRIVATE_TRANSFERS_PER_TX`: How many private transfers are sent in each tx (a higher number will mean larger transactions with more side effects, but more local proving time for the bot)
- `BOT_PUBLIC_TRANSFERS_PER_TX`: How many private transfers are sent in each tx (a higher number will mean more effort in proving the AVM and public kernel circuits, with almost the same effort for the bot)
- `SEQ_MAX_TX_PER_BLOCK`: Up to how many txs should a sequencer fit in a block.
- `SEQ_MIN_TX_PER_BLOCK`: Minimum number of txs to kick off a new block.

### Connecting to ProverNet

:::warning
Provernet is currently offline.
:::

To connect to ProverNet, you'll need to use an API Key that will be shared with all selected participants of the integration program.

Make sure to use the `provernet` version of Aztec's docker image to avoid compatibility issues. This can be retrieved by executing `VERSION=provernet aztec-up` if you have the `aztec` shell installed via `bash -i <(curl -s install.aztec.network).`

Before running a prover node for ProverNet, you'll need to seed your Ethereum address with funds to pay for gas. We have set up a faucet to do this, which can be called via the CLI:

aztec drip-faucet -a $ADDRESS -u "https://api.aztec.network/provernet/aztec-faucet/$APIKEY"

You can test that you have received funds using for example [`cast`](https://book.getfoundry.sh/cast/) to check your balance:

cast balance -r "https://provernet-mainnet-fork.aztec.network:8545/$APIKEY" $ADDRESS

Once you have funded your account, you can start your prover node using the command `aztec start --prover-node --archiver` and the following environment, replacing `$APIKEY` with the one provided, and `$PRIVATE_KEY` with the one that corresponds to your funded `$ADDRESS`.

AZTEC_PORT=8080
LOG_LEVEL=verbose
DATA_DIRECTORY=./data
ARCHIVER_POLLING_INTERVAL=1000
L1_CHAIN_ID=677692
ETHEREUM_HOST=https://provernet-mainnet-fork.aztec.network:8545/$APIKEY
PROVER_COORDINATION_NODE_URL=https://api.aztec.network/provernet/aztec-node-1/$APIKEY
PROVER_REAL_PROOFS=true
PROVER_AGENT_ENABLED=false
PROVER_PUBLISHER_PRIVATE_KEY=$PRIVATE_KEY
PROVER_NODE_MAX_PENDING_JOBS=32
PROVER_ID=$ADDRESS

Then connect one or more prover agents to your prover node, so they can start pulling proving jobs and executing them. Run `aztec start --prover` with the environment, replacing `$PROVER_NODE_URL` with the url to the prover node you started (localhost or host.docker.internal if running on the same host).

LOG_LEVEL=verbose 
PROVER_REAL_PROOFS=true
PROVER_JOB_SOURCE_URL=http://$PROVER_NODE_URL:8080
SKIP_PORT_ASSIGNMENT=1

If successful, after a while you should start seeing logs in the prover-node such as:

aztec:prover:proving-orchestrator [INFO] Orchestrator finalised block 16
aztec:block-proving-job [INFO] Finalised proof for block range fromBlock=16 toBlock=16
aztec:sequencer:publisher [INFO] SubmitProof

### Infrastructure-as-Code

:::warning
We are working on Kubernetes templates for deployment, replacing the current terraform and docker compose recipes.
:::

At Aztec we rely on Terraform templates for deploying Aztec nodes to AWS using ECS. While these templates are meant for internal use, they are open source like the rest of the code for the Aztec network, and you are welcome to use them as guidance for your own IaC if you find them helpful.

You can find the templates for running a sequencer and proving node [here](https://github.com/AztecProtocol/aztec3-packages/blob/master/yarn-project/aztec/terraform/node/main.tf), and a prover agent connected to it [here](https://github.com/AztecProtocol/aztec3-packages/blob/master/yarn-project/aztec/terraform/prover/main.tf).

## Extending the Prover Node

While it is possible to generate proofs for the Aztec network using the prover node and agents without any further changes, we expect provers with existing proving infrastructure to extend the prover node to leverage their provers.

Based on the existing architecture, we recommend the following options for extending the prover node and integrating it with an existing system. Bear in mind that these are just recommendations. If you feel a different approach is a better fit for integrating your existing proving infrastructure, go for it, and do not hesitate to reach out to us via the forum for further guidance.

### Custom Prover Agent

The simplest component to tweak is the prover agent. It should be feasible to build a custom one entirely from scratch, that consumes jobs from the proving queue using its HTTP JSON RPC interface, and in turn calls barretenberg's CLI to generate the proof. The prover agent is defined in `prover-agent` [(link)](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/prover-client/src/prover-agent/prover-agent.ts), and started in the main `aztec` package in `start_prover_agent` [(link)](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/aztec/src/cli/cmds/start_prover_agent.ts).

Alternatively, it should be possible to modify the existing prover agent to use a different `ServerCircuitProver` interface rather than the `BarretenbergNativeProver` [(link)](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/bb-prover/src/prover/bb_prover.ts) that calls directly to the CLI, and in turn calls to another infrastructure. The `ServerCircuitProver` is injected in the constructor of the `ProverAgent`.

### Custom Proving Job Queue

:::warning
This interface will change with the new prover broker implementation being built [here](https://github.com/AztecProtocol/aztec-packages/pull/8609).
:::

Instead of using the prover node's in-memory proving job queue, it's possible to provide the proving orchestrator [(link)](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/prover-client/src/orchestrator/orchestrator.ts) with a different implementation that pushes the job requests to a coordinator managed by existing infrastructure, which then dispatches jobs using custom logic. To provide a different implementation, inject it as the `ServerCircuitProver` of the orchestrator's constructor, which gets called in `createBlockProver` ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/prover-client/src/tx-prover/tx-prover.ts)).

The current queue is a simple priority queue implemented in memory ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/prover-client/src/prover-agent/memory-proving-queue.ts)), which sorts jobs based on their associated epoch (block) number, to ensure that jobs from an older block are processed first.

:::info
Note that proving jobs for a given epoch are produced incrementally: eg a *base rollup* proof job requires its corresponding *tube* proof to be completed first, so the orchestrator only enqueues the base rollup once it receives the response from the tube. A FIFO ordering would mean that jobs from a new block would get precedence over completing an existing block.
:::

The queue must implement the `ProvingJobSource` interface ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/circuit-types/src/interfaces/proving-job.ts)), which is required by the `createProvingJobSourceServer` call ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/aztec/src/cli/cmds/start_prover_node.ts)) to create the endpoint that the prover agents pull jobs from.

### Custom Orchestrator

A more difficult and involved approach is to modify the [proving orchestrator](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/prover-client/src/orchestrator/orchestrator.ts) itself. This requires additional work since the orchestrator is responsible for generating the sequence of individual proving jobs from a sequence of Aztec transactions, but this allows for greater control of how proof data is passed from one circuit to the next.

## Development setup

To set up a local development environment for modifying the prover node or agent, you will need to pull and build the [Aztec packages monorepo](https://github.com/AztecProtocol/aztec-packages). The `provernet` branch should match the deployed version on provernet.

The monorepo includes several dependencies to the prover node and agent, such as cryptography libraries, circuits, the noir compiler, contracts, etc. As instructed [here](https://github.com/AztecProtocol/aztec-packages?tab=readme-ov-file#development-setup), run `bootstrap.sh full` to build all projects in the monorepo, including the `yarn-project` that has the typescript code for prover node and agent. Any subsequent changes can then be handled by running `yarn tsc -b` within `yarn-project`. Note there's a [devbox image](https://hub.docker.com/r/aztecprotocol/devbox) published with all build tools already bundled to simplify dev setup.

Alternatively, you can run `bootstrap.sh fast` which will download build artifacts from a private ECR that acts as build cache. You will need credentials to access the registry and run this command, contact as privately for them.

Once any changes have been made to the code and built, you can start the prover node and agent via the `aztec` CLI which sits on the `yarn-project/aztec` folder, by changing directory into it and running `yarn aztec --help`.

## Metrics

Aztec components can optionally push OpenTelemetry data to an [OTEL collector](https://opentelemetry.io/docs/collector/), so they can then be fed into other platforms. This is enabled by setting a `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` environment variable with the URL to the collector, and an optional `OTEL_SERVICE_NAME` with the name of the service pushing the data.

Refer to the `docker-compose` template ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/docker-compose.yml)) in the repo for an example using an OpenTelemetry collector, Prometheus, Grafana, and Jaeger. You can also find the definition of the Grafana dashboard we use internally for monitoring our own infrastructure ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/grafana_dashboards/aztec/aztec-dashboard-all-in-one.json)).

The list of metrics currently generated can be found in the `metrics` file ([link](https://github.com/AztecProtocol/aztec3-packages/blob/provernet/yarn-project/telemetry-client/src/metrics.ts)). The `CIRCUIT_*` metrics are generated by the prover agent, and arguably the most relevant for the integration.

### Observed proving times

:::warning
These numbers are out of date.
:::

Based on our own deployments, using proving agents with 32 cores and 128 GB memory, we are seeing the followign times for witness and proof generation for each circuit. Note that these times may change with changes to the circuits and improvements to the proving backend:

| Circuit | Witness generation | Proof generation |
| ------------------ | ------------------ | ---------------- |
| Tube | - | 4min |
| Public Kernel App | 4s | 22s |
| Public Kernel Tail | 37s | 2min |
| Base Parity | 20ms | 2s |
| Root Parity | 50ms | 1min |
| Base Rollup | 7s | 2min |
| Merge Rollup | 30ms | 30s |
| Root Rollup | 60ms | 37s |