Prover Integration Guide

This document is meant for infrastructure providers who intend to run a prover in the Aztec Network testnet. It covers the architecture of the prover node, suggestions for integration, and instructions on running Aztec's prover node client.

Overview

Both sequencing and proving in the Aztec Network is intended to be fully decentralized. We expect sequencers to submit blocks to L1 every ~30 seconds, and provers to prove batches of 32 blocks to finalize the L2 chain. Sequencers will be chosen via a random election, while provers will be selected by sequencers via an out-of-protocol sequencer-prover coordination mechanism.

For the purpose of proving testing, the network will be led by a single sequencer, that will choose the prover for each epoch based on the bids submitted.

Rollup Circuits

Aztec's circuits are structured such that a single root rollup proof is produced that attests to the correct execution of a sequence of transactions throughout an epoch. An epoch is defined as 32 consecutive slots.

Proofs in Aztec are initially split into two kinds: client and server proofs.

Client proofs are generated by users in their own devices when they send a transaction, and use the ClientIVC proving system. These proofs are zero-knowledge, and prove the correct execution of smart contract functions on the client, attesting to the validity of the transaction side effects (emitted note hashes, nullifiers, encrypted logs, etc).

Server proofs are generated by provers. These are Honk proofs, and can be roughly split into 3 domains:

• Transaction proving: This proves correct execution of a transaction's public functions, via the AVM and public kernel circuit. It then verifies and converts the ClientIVC proof into a Honk proof using a tube circuit that can be aggregated into the rollup tree.
• Rollup proving: The final set of side effects of each transaction is fed into the base rollup circuit which forms the leaf level of a binary tree of aggregated proofs, which get iteratively merged and ultimately result in a single root rollup proof that is verified on L1.
• Parity circuits: This tree-like set of circuits is used exclusively for cross-chain messaging, and its output is then fed into the block root rollup circuit.

Note that generating each proof requires two steps:

• The circuit is first simulated given a set of inputs to produce the witness, using the ACVM or AVM simulator.
• The partial witness is then provided, along with the circuit bytecode and optionally a proof aggregation object, to the barretenberg binary to generate the actual proof.

Parity proving

The parity circuits attest to cross-chain messaging. All L1 to L2 messages in a given block are batched and processed by individual base parity circuits, whose output is then aggregated into a root parity circuit. This process is repeated for every block in the epoch.

Transaction proving

Each transaction submitted by a user has an associated ClientIVC proof. The first step in proving is to transform this proof into a Honk proof that can be recursively fed into other circuits. This is handled by the initial tube circuit.

Then, a transaction may have multiple public function calls (or none) that are executed by the sequencer. Each of these public executions need to be proven using the AVM circuit, the result of each then being passed onto a public kernel circuit, and then ultimately into public kernel tail circuit.

If the transaction has no public function calls, then the output of the tube circuit is fed directly into the next stage of proving.

Rollup proving

The output of each transaction is passed onto a base rollup circuit. These circuits then get aggregated into a binary tree via merge circuits, until a block root rollup circuit that attests to the validity of an entire block.

Block root rollups are then again merged into a binary tree using block merge rollup circuits, until they are merged into a root rollup circuit that proves the validity of the entire epoch.

Putting it all together

This is the entire set of circuits that need to be recursively proven in order to complete the proof for a given epoch.

Prover Node

The Aztec client can be run as a prover node. In this mode, the client will automatically monitor L1 for unproven epochs and propose bids for proving them. Should a bid get accepted and posted to L1 by a sequencer, the prover node will kick off an epoch proving job which performs the following tasks:

• Downloads the transaction hashes in the epoch and all L1 to L2 messages from L1.
• Downloads the transaction objects with their ClientIVC proofs from a remote node (to be replaced by loading them from the P2P pool).
• Executes transactions in the epoch in order, generating proving jobs for each of them.
• Generates the inputs for each circuit and kicks off individual proving jobs to prover agents, recursively proving until it gets to the root rollup proof.
• Submits the root rollup proof to L1 to advance the proven chain.

To achieve this, the prover node relies on the following components:

• L2 block source: Tracks L2 by monitoring the Rollup contract on L1 for new L2 blocks and proofs published, and downloads them.
• Transaction provider: Connects to a remote Aztec Node or to the P2P network directly to retrieve transactions with their client proofs.
• World state: Collection of merkle trees that represent the chain global state. It is updated from data collected from the L2 block source as well as from the execution of transactions.
• Transaction processor: Uses the Aztec VM simulator to execute public function calls from transactions and update world state accordingly.
• L1 publisher: Handles the submission of blocks and proofs to the L1 rollup contract.
• Proving orchestrator: Accepts executed transactions in a given block and dispatches proving jobs to a broker to produce the resulting root rollup proof.
• Proof broker: Takes individual proving requests from the Orchestrator and dispatches it to a fleet of prover agents.
• Prover agent: Multiple stateless processes that poll the proving queue for new jobs, execute them, and submit back the corresponding proof and outputs.

Proving Orchestrator

The proving orchestrator is the main component of the prover node. The orchestrator receives each transaction as they are executed by the transaction processor, executes the tube, AVM, public kernel, and base rollup circuits for it, and then adds it to the rollup tree where it gets merged until the root rollup proof is obtained. The orchestrator also handles passing the generated proof from a circuit to the next one that aggregates it.

In short, the orchestrator is responsible for turning an epoch proving job into a set of individual proving requests that are fulfilled by a fleet of proving agents.

The orchestrator applies changes to the world-state as it goes through the received transactions (this is subject to change) and outputs proving jobs to a proving broker. Each individual proving job is comprised of an identifier of the circuit to be run and its inputs.

The types of proving jobs are the following:

• PRIVATE_KERNEL_EMPTY
• PUBLIC_VM
• PUBLIC_KERNEL_NON_TAIL
• PUBLIC_KERNEL_TAIL
• BASE_ROLLUP
• MERGE_ROLLUP
• BLOCK_ROOT_ROLLUP
• BLOCK_MERGE_ROLLUP
• ROOT_ROLLUP
• BASE_PARITY
• ROOT_PARITY
• TUBE_PROOF

Prover Agent

This interface will change with the new prover broker implementation being built here.

The prover agent is a stateless process that consumes proving job requests and runs them through a prover (implementation here). The agent can run within the same process as the orchestrator, or on a separate machine, in which case it polls for new jobs via JSON-RPC over HTTP. Multiple prover agents can poll from the same queue to distribute the load.

See here for the proving queue interface, which can be called via an HTTP POST with the following JSON payloads as an example:

{ "method": "provingJobSource_getProvingJob" }
{ "method": "provingJobSource_heartbeat", "params": ["job-id"] }

The prover agent uses first the ACVM simulator to simulate the circuits and generate the witnesses based on the inputs provided. Refer to this file as a reference on how to invoke the ACVM CLI.

The prover agent then relies on barretenberg, Aztec's cryptographic library, for generating proofs. Barretenberg can be executed directly as an executable via its CLI. Refer to this file as a reference on how to call barretenberg for generating each type of proof needed.

The relevant circuits can be found in the noir-projects/noir-protocol-circuits folder in the repository.

Quote Provider

In order to start proving an epoch, a prover node must first submit a quote to the network, and it needs to be accepted. Quotes for proving a given epoch are generated and broadcasted once the epoch is finished. The sequencer is then responsible for selecting a quote and publishing it on L1 along with a block. The prover node monitors L1 and triggers the proving job only if they see their quote as accepted.

The quote provider is the component, within the prover node, responsible for generating a quote for a given epoch. The quote is expected to contain at least the following fields:

• basisPointFee: The fraction of the block rewards that the prover gets when the proof of epoch lands.
• bondAmount: The number of tokens that the prover must provide as a bond when it is selected for proving an epoch. The bond is taken from funds escrowed by the prover on L1. Must be at least PROOF_COMMITMENT_MIN_BOND_AMOUNT_IN_TST=1000 tokens.

In development networks, provernet included, the prover node will automatically mint test tokens for itself to put in the escrow so they can be provided as bond.

The quote provider defaults to always sending a quote, with QUOTE_PROVIDER_BASIS_POINT_FEE=100 and QUOTE_PROVIDER_BOND_AMOUNT=1000, both configurable via environment variables.

Alternatively, the quote provider can issue an HTTP POST to a configurable QUOTE_PROVIDER_URL to get the quote. The request body is JSON-encoded and contains the following fields:

• epochNumber: The epoch number to prove
• fromBlock: The first block number of the epoch to prove
• endBlock: The last block number (inclusive) of the epoch to prove
• txCount: The total number of txs in the epoch
• totalFees: The accumulated total fees across all txs in the epoch

The response is expected also in JSON and to contain basisPointFee and bondAmount fields. If no response is sent, no quote will be sent by the prover for the epoch.

Running a Prover Node

The Aztec client needed to run a prover node is shipped as a docker image aztecprotocol/aztec. The image exposes the Aztec CLI as its ENTRYPOINT, which includes a start command for starting different components. You can download it directly or use the sandbox scripts which will automatically pull the image and add the aztec shell script to your path.

Once the aztec command is available, you can run a prover node via:

aztec start --prover-node --archiver

To run a prover agent, either run aztec start --prover, or add the --prover flag to the command above to start an in-process prover.

Configuration

The Aztec client is configured via environment variables, the following ones being relevant for the prover node:

• ETHEREUM_HOST: URL to an Ethereum node.
• L1_CHAIN_ID: Chain ID for the L1 Ethereum chain.
• DATA_DIRECTORY: Local folder where archive and world state data is stored.
• AZTEC_PORT: Port where the JSON-RPC APIs will be served.
• PROVER_COORDINATION_NODE_URL: URL to an Aztec node that acts as transaction provider and as source of truth for the L1 contract addresses. Formerly TX_PROVIDER_NODE_URL.
• PROVER_PUBLISHER_PRIVATE_KEY: Private key used for publishing proofs to L1. Ensure it corresponds to an address with ETH to pay for gas.
• PROVER_AGENT_ENABLED: Whether to run a prover agent within the node. We recommend setting to false and running prover agents in separate hosts.
• PROVER_ID: An identifier for the prover, which gets embedded into the proofs. Can be an arbitrary hex string.

The prover agent, on the other hand, relies on the following environment variables:

• PROVER_JOB_SOURCE_URL: URL to the prover node that acts as a proving job source. Formerly AZTEC_NODE_URL.
• PROVER_AGENT_CONCURRENCY: Maximum concurrency for this given prover agent. Defaults to 1.

Both the prover node and agent also rely on the following:

• PROVER_REAL_PROOFS: Whether to generate actual proofs, as opposed to only simulating the circuit and outputting fake proofs. Set to true for the scope of the integration.
• LOG_LEVEL: One of debug, verbose, info, warn, or error.
• LOG_JSON: Set to true to output logs in ndJSON format (unreleased).
• OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: Optional URL for pushing telemetry data to a remote OpenTelemetry data collector.

Running a ProverNet locally

This docker compose template is to be replaced by a Kubernetes template to be runnable locally.

To run a local network that emulates Provernet, we suggest using docker compose with the template docker-compose.provernet.yml from the AztecProtocol/aztec-packages repository. This will kick off the following containers:

• An anvil instance that acts as L1
• A single Aztec sequencer node
• A bot that will send token transfers as test transactions as regular intervals using its own PXE
• A prover-node that will look for unproven blocks in anvil and attempt to prove them
• A single prover-agent that will pull proving jobs from the prover node, which can be scaled to multiple ones to speed up proving of larger blocks
• A block watcher that will periodically report the latest block and latest proven block in the chain

The docker compose template will accept the following env vars:

• IMAGE: Image and tag within the aztecprotocol docker registry to use. Defaults to aztec:master. Use the aztec image sha256:41ab5110d03cacb6850dab47df5871ecb136cf001b50e7dc01a02075c11a75eb here.
• PROVER_REAL_PROOFS: Whether to generate actual proofs. Set to false to speed up testing.
• PROVER_TEST_DELAY_MS: Artificial delay in ms for every proof generated. Consider setting to a non-zero value if PROVER_REAL_PROOFS is disabled.

To further tweak the behaviour of the network, consider modifying the following environment variables on the template:

• BOT_TX_INTERVAL_SECONDS: How many seconds should the bot wait between sent transactions
• BOT_PRIVATE_TRANSFERS_PER_TX: How many private transfers are sent in each tx (a higher number will mean larger transactions with more side effects, but more local proving time for the bot)
• BOT_PUBLIC_TRANSFERS_PER_TX: How many private transfers are sent in each tx (a higher number will mean more effort in proving the AVM and public kernel circuits, with almost the same effort for the bot)
• SEQ_MAX_TX_PER_BLOCK: Up to how many txs should a sequencer fit in a block.
• SEQ_MIN_TX_PER_BLOCK: Minimum number of txs to kick off a new block.

Connecting to ProverNet

Provernet is currently offline.

To connect to ProverNet, you'll need to use an API Key that will be shared with all selected participants of the integration program.

Make sure to use the provernet version of Aztec's docker image to avoid compatibility issues. This can be retrieved by executing VERSION=provernet aztec-up if you have the aztec shell installed via bash -i <(curl -s install.aztec.network).

Before running a prover node for ProverNet, you'll need to seed your Ethereum address with funds to pay for gas. We have set up a faucet to do this, which can be called via the CLI:

aztec drip-faucet -a $ADDRESS -u "https://api.aztec.network/provernet/aztec-faucet/$APIKEY"

You can test that you have received funds using for example cast to check your balance:

cast balance -r "https://provernet-mainnet-fork.aztec.network:8545/$APIKEY" $ADDRESS

Once you have funded your account, you can start your prover node using the command aztec start --prover-node --archiver and the following environment, replacing $APIKEY with the one provided, and $PRIVATE_KEY with the one that corresponds to your funded $ADDRESS.

AZTEC_PORT=8080
LOG_LEVEL=verbose
DATA_DIRECTORY=./data
ARCHIVER_POLLING_INTERVAL=1000
L1_CHAIN_ID=677692
ETHEREUM_HOST=https://provernet-mainnet-fork.aztec.network:8545/$APIKEY
PROVER_COORDINATION_NODE_URL=https://api.aztec.network/provernet/aztec-node-1/$APIKEY
PROVER_REAL_PROOFS=true
PROVER_AGENT_ENABLED=false
PROVER_PUBLISHER_PRIVATE_KEY=$PRIVATE_KEY
PROVER_NODE_MAX_PENDING_JOBS=32
PROVER_ID=$ADDRESS

Then connect one or more prover agents to your prover node, so they can start pulling proving jobs and executing them. Run aztec start --prover with the environment, replacing $PROVER_NODE_URL with the url to the prover node you started (localhost or host.docker.internal if running on the same host).

LOG_LEVEL=verbose 
PROVER_REAL_PROOFS=true
PROVER_JOB_SOURCE_URL=http://$PROVER_NODE_URL:8080
SKIP_PORT_ASSIGNMENT=1

If successful, after a while you should start seeing logs in the prover-node such as:

aztec:prover:proving-orchestrator [INFO] Orchestrator finalised block 16
aztec:block-proving-job [INFO] Finalised proof for block range fromBlock=16 toBlock=16
aztec:sequencer:publisher [INFO] SubmitProof

Infrastructure-as-Code

We are working on Kubernetes templates for deployment, replacing the current terraform and docker compose recipes.

At Aztec we rely on Terraform templates for deploying Aztec nodes to AWS using ECS. While these templates are meant for internal use, they are open source like the rest of the code for the Aztec network, and you are welcome to use them as guidance for your own IaC if you find them helpful.

You can find the templates for running a sequencer and proving node here, and a prover agent connected to it here.

Extending the Prover Node

While it is possible to generate proofs for the Aztec network using the prover node and agents without any further changes, we expect provers with existing proving infrastructure to extend the prover node to leverage their provers.

Based on the existing architecture, we recommend the following options for extending the prover node and integrating it with an existing system. Bear in mind that these are just recommendations. If you feel a different approach is a better fit for integrating your existing proving infrastructure, go for it, and do not hesitate to reach out to us via the forum for further guidance.

Custom Prover Agent

The simplest component to tweak is the prover agent. It should be feasible to build a custom one entirely from scratch, that consumes jobs from the proving queue using its HTTP JSON RPC interface, and in turn calls barretenberg's CLI to generate the proof. The prover agent is defined in prover-agent (link), and started in the main aztec package in start_prover_agent (link).

Alternatively, it should be possible to modify the existing prover agent to use a different ServerCircuitProver interface rather than the BarretenbergNativeProver (link) that calls directly to the CLI, and in turn calls to another infrastructure. The ServerCircuitProver is injected in the constructor of the ProverAgent.

Custom Proving Job Queue

This interface will change with the new prover broker implementation being built here.

Instead of using the prover node's in-memory proving job queue, it's possible to provide the proving orchestrator (link) with a different implementation that pushes the job requests to a coordinator managed by existing infrastructure, which then dispatches jobs using custom logic. To provide a different implementation, inject it as the ServerCircuitProver of the orchestrator's constructor, which gets called in createBlockProver (link).

The current queue is a simple priority queue implemented in memory (link), which sorts jobs based on their associated epoch (block) number, to ensure that jobs from an older block are processed first.

Note that proving jobs for a given epoch are produced incrementally: eg a base rollup proof job requires its corresponding tube proof to be completed first, so the orchestrator only enqueues the base rollup once it receives the response from the tube. A FIFO ordering would mean that jobs from a new block would get precedence over completing an existing block.

The queue must implement the ProvingJobSource interface (link), which is required by the createProvingJobSourceServer call (link) to create the endpoint that the prover agents pull jobs from.

Custom Orchestrator

A more difficult and involved approach is to modify the proving orchestrator itself. This requires additional work since the orchestrator is responsible for generating the sequence of individual proving jobs from a sequence of Aztec transactions, but this allows for greater control of how proof data is passed from one circuit to the next.

Development setup

To set up a local development environment for modifying the prover node or agent, you will need to pull and build the Aztec packages monorepo. The provernet branch should match the deployed version on provernet.

The monorepo includes several dependencies to the prover node and agent, such as cryptography libraries, circuits, the noir compiler, contracts, etc. As instructed here, run bootstrap.sh full to build all projects in the monorepo, including the yarn-project that has the typescript code for prover node and agent. Any subsequent changes can then be handled by running yarn tsc -b within yarn-project. Note there's a devbox image published with all build tools already bundled to simplify dev setup.

Alternatively, you can run bootstrap.sh fast which will download build artifacts from a private ECR that acts as build cache. You will need credentials to access the registry and run this command, contact as privately for them.

Once any changes have been made to the code and built, you can start the prover node and agent via the aztec CLI which sits on the yarn-project/aztec folder, by changing directory into it and running yarn aztec --help.

Metrics

Aztec components can optionally push OpenTelemetry data to an OTEL collector, so they can then be fed into other platforms. This is enabled by setting a OTEL_EXPORTER_OTLP_METRICS_ENDPOINT environment variable with the URL to the collector, and an optional OTEL_SERVICE_NAME with the name of the service pushing the data.

Refer to the docker-compose template (link) in the repo for an example using an OpenTelemetry collector, Prometheus, Grafana, and Jaeger. You can also find the definition of the Grafana dashboard we use internally for monitoring our own infrastructure (link).

The list of metrics currently generated can be found in the metrics file (link). The CIRCUIT_* metrics are generated by the prover agent, and arguably the most relevant for the integration.

Observed proving times

These numbers are out of date.

Based on our own deployments, using proving agents with 32 cores and 128 GB memory, we are seeing the followign times for witness and proof generation for each circuit. Note that these times may change with changes to the circuits and improvements to the proving backend:

Circuit	Witness generation	Proof generation
Tube	-	4min
Public Kernel App	4s	22s
Public Kernel Tail	37s	2min
Base Parity	20ms	2s
Root Parity	50ms	1min
Base Rollup	7s	2min
Merge Rollup	30ms	30s
Root Rollup	60ms	37s