# **Epoch Proving Integration Guide**
The Sequencer and Prover coordination mechanism will be put to test during the upcoming Sequencer & Prover Testnet, a permissioned Aztec rollup deployment where multiple sequencers coordinate with provers to build blocks, outsource proof production and advance the Aztec chain. S&P Testnet launch date is the first week of November.
Goals of sharing this:
1. We expect provers to be ready to prove full epochs in time for S&P testnet.
2. We would like to test proving at 1TPS for a short period of time during S&P Testnet
## Table of Contents
1. [Block Production Overview](#Block-Production-Overview)
2. [Proving Coordination Workflow](#Proving-Coordination-Workflow)
i. [P2P Client](#P2P-Client)
ii. [Prover Node](#Proving-Node)
iii. [Pricing Quotes](#Pricing-Quotes)
3. [Proving Workflow](#Proving-Workflow)
4. [Appendix I: Rollup Circuits Overview](#Appendix-I:-Rollup-Circuits-Overview)
## Block Production Overview
Both sequencing and proving in the Aztec Network are intended to be fully decentralized. We expect sequencers to submit blocks to L1 every ~36 seconds, and provers to prove batches of 32 blocks to finalize the L2 chain.
Sequencers will be chosen via a random election, while provers will be selected by sequencers via an out-of-protocol coordination mechanism. The plan is to steadily scale throughput to 1TPS (i.e. 36tx blocks posted every 36s), and we expect provers to be able to prove at 1TPS throughput by mid November.
The proposers in the first $C=13$ slots in epoch `N+1` will accept quotes to prove epoch `N` from provers. The winning prover will have until the end of epoch `N+1` to produce and submit the proof to L1. In the case where a quote is accepted in slot 1 of epoch `N+1`, the prover will have 18.6mins to compute the proof and land it on L1. In the case where a quote is accepted in slot 13, the prover will only have 11.4 mins.
At 1 TPS, each epoch is 1,152 txs. Based on timeliness requirements and depending on proof computational complexity, we should expect the number of prover agents to be up to 1,000 at 1TPS.
For more information on the block production mechanism, please see Aztec’s [RFC](#https://forum.aztec.network/t/request-for-comments-aztecs-block-production-system/6155/4) on block production.
## Proving Coordination Workflow
Proposers run RFQs to obtain quotes from provers. Quotes are binding promises from provers to prove an entire epoch. The exact channel over which provers send quotes to proposers is NOT enshrined by the protocol.
However, Aztec Nodes will support two optional mechanisms that provers can use to submit quotes to proposers.
- Gossip quotes via the P2P
- Send a quote directly via http (i.e. http://aztec-node:8000)
To send a quote via the p2p, do not set the environment variable `PROVER_COORDINATION_NODE_URL` and make sure that `P2P_ENABLED` is set to True.
> Note: For S&P Testnet, please make sure that you are gossiping quotes via the p2p. Set `P2P_ENABLED` to true and do not use `PROVER_COORDINATION_NODE_URL`.
```solidity=
struct EpochProofQuote {
Signature signature;
address prover;
uint256 epochToProve;
uint256 validUntilSlot;
uint256 bondAmount;
address rollupAddress;
uint32 basisPointFee;
}
```
To accomplish this coordination through the Aztec node software, we extend both the `P2PClient` and `ProverNode`.
### P2P Client
The `P2PClient` will be extended by:
```typescript
class P2PClient {
//...
public async addEpochProofQuote(quote: EpochProofQuote): Promise<void> {
// add quote to quote memory pool
this.epochProofQuotePool.addQuote(quote);
// propagate quote via p2p
this.broadcastEpochProofQuote(quote);
}
```
This is called by the Prover Node inside `ProverNode.sendEpochProofQuote()` after it detects an epoch has ended.
### Prover Node
As for the Prover Node, we add `QuoteProvider` and `BondManager` interfaces. Also an `EpochMonitor` which sits on the main `start` loop of the Prover Node. It fetches the most recent completed epochs and checks whether the proposer accepted an EpochProofQuote.
If no quote has been accepted yet, the `EpochMonitor` will call on `BondManager` and `QuoteProvider` to provide a valid quote. If the claim detected belongs to the prover, the monitor will kick off a `handleClaim()` to create proving jobs.
```typescript
interface BondManager {
ensureBond(amount: number): Promise<void>;
}
interface QuoteProvider {
getQuote(epoch: number): Promise<EpochProofQuote | undefined>;
}
```
When the prover node first starts up, it will call `BondManager.ensureBond` to ensure it has the minimum deposit amount `PROVER_MINIMUM_ESCROW_AMOUNT` deposited in the escrow contract. If it does not, it will top up to the target deposit amount `PROVER_TARGET_ESCROW_AMOUNT`.
> Both `PROVER_MINIMUM_ESCROW_AMOUNT` and `PROVER_TARGET_ESCROW_AMOUNT` are customizable env variables.
The `EpochMonitor` will then get the last completed, unproven epoch and will call on the `QuoteProvider` to generate a quote if the epoch has not been claimed by any provers yet. The QuoteProvider will be provided with all the blocks in the unproven epoch so it could perform any custom logic to determine the quote parameters i.e. `bondAmount`, `basisPointFee`.
Alternatively, the quote provider can issue an HTTP POST to a configurable `QUOTE_PROVIDER_URL` to get the quote. The request body is JSON-encoded and contains the following fields:
- `epochNumber`: The epoch number to prove
- `fromBlock`: The first block number of the epoch to prove
- `toBlock`: The last block number (inclusive) of the epoch to prove
- `txCount`: The total number of txs in the epoch
- `totalFees`: The accumulated total fees across all txs in the epoch
The response is also expected in JSON and to contain `basisPointFee` and `bondAmount` fields. Optionally, the request can include a `validUntilSlot` parameter which specifies for how many slots the quote remains valid. For example, an EpochProofQuote with parameters `epochToProve=100` and `validUntilSlot=5` means that any of the first 5 proposers in epoch 101 can "claim" this quote.
If no `QUOTE_PROVIDER_URL` is passed along to the Prover Node, then a SimpleQuoteProvider is used which always returns the same `basisPointFee` and `bondAmount` as set in the `QUOTE_PROVIDER_BASIS_POINT_FEE` and `QUOTE_PROVIDER_BOND_AMOUNT` environment variables.
>[!warning]
If the remote QuoteProvider does not return a `bondAmount` or a `basisPointFee`, the Prover Node will not generate nor submit a quote to the proposer.
Separately, the Prover Node needs a watcher on L1 to detect if its quote has been selected.
To this end, the `L1Publisher` will be extended with a new method:
```typescript
interface L1Publisher {
getProofClaim(): Promise<EpochProofClaim>;
}
```
The Prover Node will call this method at least once per L2 slot to check for unclaimed epochs or for whether its quotes have been accepted. You can update the polling interval using the environment variable `PROVER_NODE_POLLING_INTERVAL_MS`.
### Orchestrator
The Orchestrator is a component of the Prover node. It encodes the rules that govern how the tree of proofs is constructed and attempts to achieve maximum parallelism.
The Orchestrator enqueues jobs to the Prover Broker and periodically checks if any proofs have completed
### Prover Broker
The Prover Broker sits in between the Prover Node and the many Prover Agents that will be producing proofs to complete an epoch.
The broker's main responsibility is to take proving jobs from the Prover node and give them to agents in a robust way. As such it has the following properties:
- idempotency (won't enqueue the same proof twice)
- "at least once" delivery (if an agent goes away its job is re-enqueued)
- resilient against crashes
In the Aztec software, a Prover Broker is any service that implements the `ProvingJobProducer` and `ProvingJobConsumer` interfaces. Out of the box we ship with a simple service that maintains a couple of queues in memory. The proving jobs are also backed up to disk such that the broker is able to recover after a crash. More advanced queueing mechanism can be implemented as needed (Redis, Kafka), the interfaces we use are purposefully kept simple.
The built-in Prover Broker does not make any outgoing calls and it is accessible over JSON-RPC to both the Prover Node (specifically the Orchestrator) and to Prover Agents.
## Running a Prover Node
The Aztec client can be run as a Prover Node. In this mode, the client will automatically monitor L1 for unclaimed epochs and propose bids (i.e. `EpochProofQuote`) for proving them. The prover node watches the L1 to see when a bid they submitted has been accepted by a sequencer, the prover node will then kick off an epoch proving job which performs the following tasks:
* Downloads the transaction hashes in the epoch and all L1 to L2 messages from L1.
* Downloads the transaction objects with their ClientIVC proofs from a remote node or from P2P.
* Executes transactions in the epoch in order, generating proving jobs for each of them.
* Generates the inputs for each circuit and kicks off individual proving jobs to prover agents, recursively proving until it gets to the root rollup proof.
* Submits the root rollup proof to L1 to advance the proven chain.
```mermaid
graph
style prover-node stroke:#333,stroke-width:4px
prover-node[Prover Node]
proving-job[Proving Job]
tx-provider[Tx Provider]
l1-publisher[L1 Publisher]
l2-block-source[L2 Block Source]
world-state[World State DB]
tx-processor[Tx Processor]
prover-client[Proving Orchestrator]
proving-queue[Proof Broker]
prover-agent[Prover Agent]
bb[Barretenberg]
prover-node --trigger--> proving-job
proving-job --"process-tx"--> tx-processor --"add-tx"--> prover-client
proving-job --start-batch--> prover-client
proving-job --get-tx-hashes--> l2-block-source
proving-job --"advance-to"--> world-state
proving-job --"get-txs"--> tx-provider
tx-processor --rw--> world-state
world-state --"get-blocks"--> l2-block-source
prover-client --"rw"--> world-state
proving-job --publish-proof--> l1-publisher
prover-client --"push-job"--> proving-queue
%%prover-agent --"pull-job"--> proving-queue
proving-queue <--"pull-jobs"--o prover-agent
subgraph "Prover Agent"
prover-agent --"prove"--> bb
end
%% p2p-client --> tx-pool --"save-tx"--> tx-db
%% p2p-client --get-blocks--> l2-block-source
```
### Setup
The Aztec client needed to run a prover node is shipped as a [docker image `aztecprotocol/aztec`](https://hub.docker.com/r/aztecprotocol/aztec). The image exposes the Aztec CLI as its `ENTRYPOINT`, which includes a `start` command for starting different components. You can download it directly or use the [sandbox scripts](https://docs.aztec.network/getting_started#install-the-sandbox) which will automatically pull the image and add the `aztec` shell script to your path.
Once the `aztec` command is available, you can run a **prover node** via:
```bash!
aztec start --prover-node --archiver
```
To run a **prover agent**, either run `aztec start --prover`, or add the `--prover` flag to the command above to start an in-process prover.
### Configuration
The Aztec client is configured via environment variables, the following ones being relevant for the prover node:
- `ETHEREUM_HOST`: URL to an Ethereum node.
- `L1_CHAIN_ID`: Chain ID for the L1 Ethereum chain.
- `DATA_DIRECTORY`: Local folder where archive and world state data is stored.
- `AZTEC_PORT`: Port where the JSON-RPC APIs will be served.
- `PROVER_PUBLISHER_PRIVATE_KEY`: Private key used for publishing proofs to L1. Ensure it corresponds to an address with ETH to pay for gas.
- `PROVER_AGENT_ENABLED`: Whether to run a prover agent process on the same host running the Prover Node. We recommend setting to `false` and running prover agents on seperate hosts.
- `P2P_ENABLED` Set to True so that your node can discover peers, receive tx data and gossip quotes to sequencers.
- `PROVER_COORDINATION_NODE_URL` Send quotes via http. Only used if `P2P_ENABLED` is false.
- `BOOT_NODE_URL` The URL of the boot node for peer discovery.
- `AZTEC_NODE_URL` is used by the Prover Node to fetch the L1 contract addresses if they were not manually set via env vars.
>[!Note]
For S&P Testnet, we will be providing an Ethereum host, a Boot Node URL and a specific Aztec Image. Please refer to the
The prover agent, on the other hand, relies on the following environment variables:
- `PROVER_BROKER_HOST`: URL to the Prover Node that acts as a proving job source.
- `PROVER_AGENT_CONCURRENCY`: Maximum concurrency for this given prover agent. Defaults to `1`.
Both the prover node and agent also rely on the following:
- `PROVER_REAL_PROOFS`: Whether to generate actual proofs, as opposed to only simulating the circuit and outputting fake proofs. **Set to `true` for the scope of the S&P Testnet.**
- `LOG_LEVEL`: One of `debug`, `verbose`, `info`, `warn`, or `error`.
- `LOG_JSON`: Set to `true` to output logs in JSON format (**unreleased**).
- `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT`: Optional URL for pushing telemetry data to a remote OpenTelemetry data collector.
## Appendix: Rollup Circuits Overview
Aztec's circuits are structured such that a single **root rollup proof** is produced that attests to the correct execution of a sequence of transactions throughout an **epoch**. An **epoch** is defined as 32 consecutive slots.
Proofs in Aztec are initially split into two kinds: **client** and **server** proofs.
**Client proofs** are generated by users in their own devices when they send a transaction, and use the ClientIVC proving system. These proofs are zero-knowledge, and prove the correct execution of smart contract functions on the client, attesting to the validity of the transaction side effects (emitted note hashes, nullifiers, encrypted logs, etc).
**Server proofs** are generated by provers. These are Honk proofs, and can be roughly split into 3 domains:
- **Transaction proving**: This proves correct execution of a transaction's public functions, via the **AVM** and **public kernel** circuit. It then verifies and converts the ClientIVC proof into a Honk proof using a **tube** circuit that can be aggregated into the rollup tree.
- **Rollup proving**: The final set of side effects of each transaction is fed into the **base rollup** circuit which forms the leaf level of a binary tree of aggregated proofs, which get iteratively **merged** and ultimately result in a single **root rollup** proof that is verified on L1.
- **Parity circuits**: This tree-like set of circuits is used exclusively for cross-chain messaging, and its output is then fed into the block root rollup circuit.
Note that generating each proof requires two steps:
- The circuit is first simulated given a set of inputs to produce the witness, using the ACVM or AVM simulator.
- The partial witness is then provided, along with the circuit bytecode and optionally a proof aggregation object, to the barretenberg binary to generate the actual proof.
### Parity proving
The parity circuits attest to cross-chain messaging. All L1 to L2 messages in a given block are batched and processed by individual **base parity** circuits, whose output is then aggregated into a **root parity** circuit. This process is repeated for every block in the epoch.
```mermaid
graph BT
I0((Msgs 00-03))
I1((Msgs 04-07))
I2((Msgs 08-11))
I3((Msgs 12-15))
I0 --> B0[Base Parity]
I1 --> B1[Base Parity]
I2 --> B2[Base Parity]
I3 --> B3[Base Parity]
R[Root Parity]
B0 --> R
B1 --> R
B2 --> R
B3 --> R
```
### Transaction proving
Each transaction submitted by a user has an associated ClientIVC proof. The first step in proving is to transform this proof into a Honk proof that can be recursively fed into other circuits. This is handled by the initial **tube** circuit.
Then, a transaction may have multiple **public function calls** (or none) that are executed by the sequencer. Each of these public executions need to be proven using the **AVM circuit**. The outputs of the AVM and Tube circuits are then fed into the Public Base Rollup.
```mermaid
graph BT
ClientIVC(ClientIVC) --> Tube[Tube]
Tube --> Pub1[Public Base Rollup]
AVM1[AVM] --> Pub1
```
If the transaction has no public function calls, then the output of the tube circuit is fed directly into the next stage of proving.
```mermaid
graph BT
ClientIVC(ClientIVC) --> Tube[Tube]
Tube --> Pri[Private Base Rollup]
```
### Rollup proving
The output of each transaction is passed onto a **base rollup circuit**. These circuits then get aggregated into a binary tree via **merge circuits**, until a **block root rollup circuit** that attests to the validity of an entire block.
```mermaid
graph BT
Tube1[Tube\nTx=0x1] --> B1[Private Base Rollup]
Tube2[Tube\nTx=0x2] --> B2[Public Base Rollup]
AVM1[AVM] --> B2
Tube3[Tube\nTx=0x3] --> B3[Public Base Rollup]
AVM2[AVM] --> B3
Tube4[Tube\nTx=0x4] --> B4[Public Base Rollup]
AVM3[AVM] --> B4
B1 --> M1[Merge Rollup]
B2 --> M1
B3 --> M2[Merge Rollup]
B4 --> M2
M1 --> BR[Block Root Rollup]
M2 --> BR
```
Block root rollups are then again merged into a binary tree using **block merge rollup circuits**, until they are merged into a **root rollup circuit** that proves the validity of the entire epoch.
```mermaid
graph BT
B1[Block Root 1] --> M1[Block Merge Rollup]
B2[Block Root 2] --> M1
B3[Block Root 3] --> M2[Block Merge Rollup]
B4[Block Root 4] --> M2
M1 --> BR[Root Rollup]
M2 --> BR
```
### Putting it all together
This is the entire set of circuits that need to be recursively proven in order to complete the proof for a given epoch.
```mermaid
graph BT
subgraph "Block 1"
direction BT
BR[Block Root 1]
subgraph "Parity"
direction BT
I0((Msgs 00-03))
I1((Msgs 04-07))
I2((Msgs 08-11))
I3((Msgs 12-15))
I0 --> BPar0[Base Parity]
I1 --> BPar1[Base Parity]
I2 --> BPar2[Base Parity]
I3 --> BPar3[Base Parity]
RPar[Root Parity]
BPar0 --> RPar
BPar1 --> RPar
BPar2 --> RPar
BPar3 --> RPar
end
subgraph "Tx with no public fns"
ClientIVC1(Client IVC\nTx=0x1) --> Tube1[Tube Circuit]
end
Tube1 --> B1[Private Base Rollup]
subgraph "Tx with public fns"
ClientIVC2(Client IVC\nTx=0x2) --> Tube2[Tube Circuit]
Tube2 --> B2[Public Base Rollup]
AVM1[AVM] --> B2
end
B1 --> M1[Merge Rollup]
B2 --> M1
B3[Public Base Rollup] --> M2[Merge Rollup]
B4[Public Base Rollup] --> M2
end
RPar --> BR
M1 --> BR
M2 --> BR
Block2[Block Root 2]
Block3[Block Root 3]
Block4[Block Root 4]
BlockMerge1[Block Merge 1]
BlockMerge2[Block Merge 2]
Root[Root Rollup]
BR --> BlockMerge1
Block2 --> BlockMerge1
Block3 --> BlockMerge2
Block4 --> BlockMerge2
BlockMerge1 --> Root
BlockMerge2 --> Root
```
### Hardware Requirements
The number of gates figure is fluctuating as we push more code but is based on observations. The memory requirements are estimates on the other hand.
Proving times depend on thread availability. Efficient multi-threading is implemented for up to 16 cores. There are additional improvements from using 32 cores instead of 16 but speed-ups are marginal beyond that.
| Circuit | # of Gates | Memory Requirements |
| ---- | ----- | ----- |
| Tube Circuit | 4 million | 12 GiB |
| Private Base Rollup | 2.3 million | 9 $\text{ }$GiB |
| Public Base Rollup | 3.7 million | 12 GiB |
| Merge Rollup | 1.8 million |9 $\text{ }$GiB |
| Block Root | 4.4 million | 12 GiB |
| Root Rollup | 10 million | 30 GiB |
Sign in with Wallet
Connect another wallet