---
title: Expand the third phase in Oracle
tags: Oracle
status:
---

# Expand the third phase in Oracle


## 🎯 TL;DR
With the introduction of new modules, the Oracle third phase report might not fit into a single transaction.
This document presents an approach for implementing a multi-transactional third phase.

* Depending on the size of the third-phase report, it may be split into multiple transactions.
* The third-phase transaction consists of items, where each item represents a portion of the report data.
* The processing of each item must not exceed the gas limit defined for a single item. The item count per transaction limit specifies the maximum number of items in a single transaction. Therefore, the processing of items in each third-phase transaction will not exceed the gas limit, as determined by the product of the item gas cost limit and the item count limit.
* Each module within a third phase transaction will be processed independently. If the processing of any module fails, the finalization process for other unaffected modules will not be blocked.
* The integrity and sequence of the report data will be maintained across multiple transactions.

## 𓀀 Motivation

The third phase in the Accounting Oracle plays a crucial role in providing updated counts of stuck and exited validators for each node operator within the Lido protocol. This data required for distributing rewards and allocating new stake among node operators. Currently, the third phase is restricted to 200 Node Operators updates due to node operators limit in NOR contract and Ethereum block size limitations, which is sufficient for the Curated Module and Simple DVT modules.

However, with the introduction of the Community Staking Module, which don't impose any limits on node operator counts, updates for all node operators cannot be accommodated within a single transaction due to block gas limitation.

The impending launch of new staking modules within Lido necessitates a scalable solution for accommodating an increasing number of node operators.

### 🛑 Technical Constraints

The Accounting Oracle report can require an update for all Node Operators in Lido. A very large transaction could potentially take longer to process or even stall, as validators might delay including it due to its size and the commitment of a large portion of a block's gas limit.

The third phase of the Accounting report has two main parts:
- Provide Node Operator states to Staking Modules.
- Finalization hook.

The finalization hook is out of scope for this document. Detailed improvements for it are described in the [Reward distribution](https://hackmd.io/@lido/HJYbVq5b0) document.

As for Node Operator states updates, the [estimates for the CSM module](https://hackmd.io/SiwTUvtBSNKfoLyA_8LvtA?view) indicate that the report for 1000 Node Operators consumes approximately 37 million of gas. However, this estimate appears outdated. After reviewing the current code, 7.8 million gas for 1000 Node Operator updates seems like a more reasonable estimation.

::: spoiler Estimation
Per Operator Cost:

- 500 (decoding)
- 800 (storage read)
- 5000 (storage write)
- 500 (condition checks)
- 1000 (event emission)

Total per operator cost: 7800 gas.

For 1000 Operators:

 24,700 (base cost) + 1000 * 7800 (per operator cost)

 24,700 + 7,800,000 = 7,824,700

:::
<br/>

In the case of a growing CSM module and adding new SimpleDVT modules, we might be at risk that accounting oracle report will not fit into the single transaction.

## 📋 Proposed Solution

To address these challenges, a multi-transactional approach is proposed for the third phase of the Accounting Oracle report. The second-phase report is assumed to contain the hash of the first third-phase transaction, while each third-phase transaction will include the hash of the subsequent third-phase transaction. The last transaction will contain`ZERO_HASH`.


<br />

Proposed design ensures that each transaction of the third phase has been confirmed through the hash consensus of oracles during the first phase. It is not possible to make changes to any transaction without altering the report hash, thereby maintaining the integrity of the report. The process for verifying each transaction is described in the [Implementation Details](#Implementation-Details) section.

<br />


<br />

The current third phase single transaction data format is as follows:

|  3 bytes  | 2 bytes  | X bytes     | ...  
| itemIndex | itemType | itemPayload | ...  

The new version will include the `nextHash`:

| 32 bytes |    array of items
| nextHash |         ...    

Example

| 32 bytes |  3 bytes  |  2 bytes |   X bytes   |  ...  
| nextHash | itemIndex | itemType | itemPayload |  ...  


#### Finalization hook

Once the delivery of updates is complete, the Accounting Oracle triggers the Finalization hook (call Staking Router`onValidatorsCountsByNodeOperatorReportingFinished`). This signalizes that all Node Operator updates have been successfully delivered and the Accounting Oracle report is finalized.

Currently, in the Curated Module and Simple DVT modules, the finalization hook initiates reward distribution.

#### Empty third phase

No changes are required in the case when the third phase report is empty.

The second-phase report will contain `ZERO_HASH`

The AccountingOracle's `submitReportExtraDataEmpty` is called, which validates the report state and calls the finalization hook (Staking Router `onValidatorsCountsByNodeOperatorReportingFinished`).

## Implementation Details

### Processing third phase transactions
In the second phase, accounting oracle contract save in storage the hash of the first transaction's from the series of third-phase transactions.

During the third phase, we then verify that the hash of the report matches the expected hash previously stored in the state. Upon confirming a match, we proceed to process the items and update the storage with the next expected hash, as indicated by the report's `nextHash` field.

After processing the transaction portion of the report data (items), the `ExtraDataSubmitted` event will be emitted.

event ExtraDataSubmitted(uint256 indexed refSlot, uint256 itemsProcessed, uint256 itemsCount);

#### Sequence of the report data


Currently the third phase report data is split into items based on composite keys. These keys include the itemType, moduleId, and the first node operator ID in each item.

The proposed design for multi-transaction processing assumes that dividing the report into items will maintain the integrity and sequence of the report data. Report data must not be sent in an unsorted order, and each transaction must contain sorted items. The sorting order is maintained across all third phase transactions.

For more details, check the “Sanity checks” section below.

#### Count of items in third phase transaction


Each transaction consists of items, with each item representing a portion of the report data. A new parameter, `maxItemsPerExtraDataTransaction`, is suggested to define the maximum count of items in a single transaction.

// Maximum count of items in a third phase transaction
maxItemsPerExtraDataTransaction

The exact item count will be based on the following fundamental assumptions:

- Transaction processing should not exceed the maximum gas cost allocated for a third phase transaction.
- The processing of each item should not exceed the maximum gas cost allocated for item.

The gas-based calculations provided below are intended to illustrate our method for determining the specific value for the maximum item count. The Oracle will not store any gas limits; they are used solely for the purpose of calculating the maximum item count in transactions.

// More accurate values will be determined 
// after a precise calculation of gas consumption for each module update.

// Maximum gas cost for processing a third phase transaction
maxExtraDataTransactionGasCost = 8_000_000;

// Maximum gas cost for processing a single item
maxExtraDataTransactionItemGasCost = 1_000_000;

// Maximum item count in a third phase transaction
maxItemsPerExtraDataTransaction =
    maxExtraDataTransactionGasCost / maxExtraDataTransactionItemGasCost;

#### Third phase items payload


The proposed design does not restrict the format of the data that can be sent in an item's payload. However, as mentioned in the previous section, the processing of each item must not exceed the maximum gas cost allocated for an item.

Each item type may implement its own sanity checks to ensure data integrity and adherence to gas limitations.

Currently, for all modules (Curated, SimpleDVT, and CSM), the oracle's third phase supports only two types of items:

- Updating the count of exited validators for node operators.
- Updating the count of stuck validators for node operators.

##### Update validators count for Node operators
Due to the similarity between updating exited and stuck validators count, a single `maxNodeOperatorsPerExtraDataItem` can be used to ensure that the data processing for an item does not exceed the gas limit for an item.

// Maximum node operator updates per item
maxNodeOperatorsPerExtraDataItem

The gas-based calculations provided below are intended to illustrate our method for determining the specific value for the maximum node operator updates per item. The Oracle will not store any gas limits; they are used solely for the purpose of calculating the maximum node operator updates per item.

// Maximum node operator updates per item
maxNodeOperatorsPerExtraDataItem = 
    maxExtraDataTransactionItemGasCost / maxExtraDataTransactionNodeOperatorUpdateGasCost

Where `maxExtraDataTransactionNodeOperatorUpdateGasCost` is defined as the maximum gas cost for updating the count of stuck or exited validators for a single node operator across all modules.

Let's assume that M1, M2, and M3 represent the gas consumption for updating a single node operator (either stuck or exited validators count) in the respective modules with IDs 1, 2, and 3. The maximum gas consumption across these modules for node operator count updates (stuck and exited) is then calculated as follows:

maxExtraDataTransactionNodeOperatorUpdateGasCost = max(
  M1_nodeOperatorUpdateStuckValidatorsCountGasCost,
  M1_nodeOperatorUpdateExitedValidatorsCountGasCost,
  M2_nodeOperatorUpdateStuckValidatorsCountGasCost,
  M2_nodeOperatorUpdateExitedValidatorsCountGasCost,
  M3_nodeOperatorUpdateStuckValidatorsCountGasCost,
  M3_nodeOperatorUpdateExitedValidatorsCountGasCost
) 

The concrete value of maximum operator update gas cost will be determined after a precise calculation of the gas consumption.

#### Distribute items across transactions


The proposed multi-transaction design specifies the maximum item count for each third-phase transaction without dictating the method by which the oracle will pack items within these transactions.

Suppose the maximum item count per transaction is 3, and the oracle's third-phase report contains 4 items that need to be packed into 2 transactions.

// maximum items count per transaction
maxItemsPerExtraDataTransaction = 3

// third phase report contais 4 items
totalItemsCount = 4

Total: 4 items, will be split on 2 transactions

The most advisable strategies are:

1. Including as many items as possible in each transaction.


2. Distributing items across transactions.


3. Distributing items evenly according to actual items payload


The first strategy is the easiest to implement, but the large transactions may take longer to be included in a block.

We might consider the even distribution approach because, in this scenario, the processing time for each transaction should be equal. However, implementation could be challenging because the algorithm must take into account both the number of items and the actual payload of these items.

### Sanity checks

#### Exited validators limit
Currently, the `churnValidatorsPerDayLimit` parameter from the `OracleReportSanityChecker` is used to validate both newly exited validators `checkExitedValidatorsRatePerDay` and newly deposited validators `_checkAppearedValidatorsChurnLimit`. This threshold is set at 20,000 validators, a suitable number for scenarios that involve adding a large volume of new validators. However, this threshold proves excessively high for the validator exit process, where the Ethereum churn limit should be taken into account.

We propose dividing this into two distinct parameters, each with more precise values tailored to the respective cases.

On the second phase the `checkExitedValidatorsRatePerDay` sanity check will use new `exitedValidatorsPerDayChurnLimit`. And the `_checkAppearedValidatorsChurnLimit` check will use the value from `appearedValidatorsPerDayLimit`.

After the third phase for each module we check that total count of exited validators is equal to the exited validators count which we got on the second phase. (StakingRouter line 567)

// https://docs.lido.fi/guides/verify-lido-v2-upgrade-manual#depositsecuritymodule
// The maximum number of validators that Lido can deposit per day.

/**
 * const BLOCKS_PER_DAY = (24 * 60 * 60) / 12 = 7200
 * const DSM_MAX_DEPOSITS_PER_BLOCK = 150
 * const DSM_MIN_DEPOSIT_BLOCK_DISTANCE = 25
 *
 * const appearedValidatorsPerDayLimit = BLOCKS_PER_DAY / DSM_MIN_DEPOSIT_BLOCK_DISTANCE * DSM_MAX_DEPOSITS_PER_BLOCK // 43200
 */
appearedValidatorsPerDayLimit: 43200

/**
 * const currentValidatorsNumber = 1385929
 * const ACTIVATION_CHURN_LIMIT = 8
 * const VALIDATORS_PER_DAY = (BLOCKS_PER_DAY / 32) * ACTIVATION_CHURN_LIMIT
 * const VALIDATORS_AFTER_TWO_YEARS = VALIDATORS_PER_DAY * 365 * 2 + currentValidatorsNumber // 1385929 + (1800 * 365 * 2)
 *
 * const INITIAL_CHURN_LIMIT = 7
 * const CHURN_LIMIT_QUOTIENT = 65_536
 * const VALIDATORS_WITH_INITIAL_CHURN_LIMIT = 500_000
 *
 * const exitedValidatorsPerDayChurnLimit =
 *   Math.floor((VALIDATORS_AFTER_TWO_YEARS - VALIDATORS_WITH_INITIAL_CHURN_LIMIT) / CHURN_LIMIT_QUOTIENT) +
 *   INITIAL_CHURN_LIMIT // 40
 *
 * const maxExitedValidatorsPerDay = exitedValidatorsPerDayChurnLimit * (BLOCKS_PER_DAY / 32) // 9000
 */
exitedValidatorsPerDayChurnLimit:  9000

##### appearedValidatorsPerDayLimit
Above is an example calculation and recommended configuration. In our calculations we rely on the DSM limits. This number can be specified much lower, but its calculation will be dynamic, where the limits can vary greatly due to the number of rewards, withdrawals. We therefore recommend the use of static limits based on data from the DSM.

Will be used as a value in the check: `_checkAppearedValidatorsChurnLimit`
##### exitedValidatorsPerDayChurnLimit
In the calculation of this parameter, we calculate the growth of CHURN_LIMIT two years ahead. This parameter will be valid until the implementation of MEB.

Will be used as a value in the check: `checkExitedValidatorsRatePerDay`

#### Integrity and sequence of the report data

Currently, during the third phase of report processing, the Accounting Oracle validates that items are sorted according to composite keys. These keys include the itemType, moduleId, and the first node operator ID in each item.

// | 2 bytes | 19 bytes | 3 bytes | 8 bytes |
// | itemType | 00000000 | moduleId | firstNodeOpId |
uint256 sortingKey

The Accounting Oracle stores the `lastSortingKey` in `ExtraDataIterState` and updates it during processing the third phase transaction.

The proposed design for multi-transaction processing assumes that dividing the report into items will maintain the integrity and sequence of the report data. The `lastSortingKey` will be reused to validate the order of items within multiple transactions.

#### Split report data to items.
Currently the `OracleReportSanityChecker` contains parameters to establish the maximum number of node operators that can be updated during the third phase of the report.

Here is the current configuration for the on-chain sanity check, designed to accommodate two modules: Curated and SimpleDVT. It is configured to handle the delivery of up to four data lists. Each list represents either stuck or exited validators for node operators within a specific module, with no single list exceeding 50 node operators. In total, this allows for up to 200 updates concerning node operators.

   "maxAccountingExtraDataListItemsCount": 4,  
   "maxNodeOperatorsPerExtraDataItemCount": 50

interface IOracleReportSanityChecker {
    function checkAccountingExtraDataListItemCount(uint256 _extraDataListItemsCount) external view;
    function checkNodeOperatorsPerExtraDataItemCount(uint256 _itemIndex, uint256 _nodeOperatorsCount) external view;
}

These parameters and related sanity checks will be changed as the multi-transaction approach no longer imposes limits on the entire report size. Rather than validating the entire report size, we will independently validate each transaction during the third phase.

   "maxItemsPerExtraDataTransaction": 8
   "maxNodeOperatorsPerExtraDataItem": 24

interface IOracleReportSanityChecker {
    function checkExtraDataItemsCountPerTransaction(uint256 _extraDataListItemsCount) external view;
    function checkNodeOperatorsPerExtraDataItemCount(uint256 _itemIndex, uint256 _nodeOperatorsCount) external view;
}

##### Determining Limit Values
The gas consumption for updating a single node operator:

- CSM:
- ~16_650 Average
- ~41_150 Max (unstuck keys in specific condition)
- Curated-based: ~15_500

The processing of each item should not exceed the specified gas limit of 1_000_000 gas. Each transaction can contain up to 8 items, each consuming 1_000_000 gas (totaling 8_000_000 gas).

Using the higher value:
CSM: 41_150 gas

This results in: 24 operators per item (1_000_000 / 41_150 = 24.3)

- maxItemsPerExtraDataTransaction = 8
- maxNodeOperatorsPerExtraDataItem = 24

By setting these values, we ensure that each transaction stays within the 8_000_000 gas limit.