# Lido Report and incidents Off-Chain ###### tags `Lido`, `Ethereum` [TOC] ## Findings Report ### CRITICAL No issues found ### MAJOR #### [ACKNOWLEDGED] Abnormal CL rebase condition of the bunker mode depends on the manipulatable value of the `WithdrawalVault` balance in `abnormal_cl_rebase.py` ##### Description The function [`_calculate_cl_rebase_between_blocks`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py#L181) in the [`AbnormalClRebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py) class uses the `WidthrawalVault` balance to calculate the difference in the rebase between blocks. `WithdrawalVault` is a contract, which balance can be manipulated by sending ether directly to the contract. This will increase the expected rebase value at the end of the frame and allow the attacker to prevent bunker mode from activation. ```python def _calculate_cl_rebase_between_blocks( self, prev_blockstamp: BlockStamp, ref_blockstamp: ReferenceBlockStamp ) -> Gwei: # <...> # # Get Lido validators' balances with WithdrawalVault balance ref_lido_balance_with_vault = self._get_lido_validators_balance_with_vault( ref_blockstamp, self.lido_validators ) prev_lido_balance_with_vault = self._get_lido_validators_balance_with_vault( prev_blockstamp, prev_lido_validators ) ``` ``` python def _get_lido_validators_balance_with_vault( self, blockstamp: BlockStamp, lido_validators: list[LidoValidator] ) -> Gwei: """ Get Lido validator balance with withdrawals vault balance """ real_cl_balance = AbnormalClRebase.calculate_validators_balance_sum(lido_validators) withdrawals_vault_balance = int( self.w3.from_wei(self.w3.lido_contracts.get_withdrawal_balance_no_cache(blockstamp), "gwei") ) return Gwei(real_cl_balance + withdrawals_vault_balance) ``` The scenario of the attack may be as follows: 1. Malicious actor has a withdrawal request for a large amount of ETH in the withdrawal queue sent in the current frame. 2. The CL conditions show abnormal situation that provokes a bunker mode activation based on the abnormal CL rebase condition. 3. Let's say there is only a slight difference in the inequality `frame_cl_rebase < normal_cl_rebase`. 4. By sending this difference to the `WithdrawalVault` the attacker may obtain 24 more hours for withdrawing the funds from the protocol. ##### Recommendation We recommend to take the `WithdrawalVault` balance changes into account accurately, so that an unexpected increase in the balance can be matched against a calculated invariant. ##### Update ###### LIDO's response A risk’s reailization requires, **on top** of highly unlikely “bunker mode” premises, **additional preconditions with a low probability** (high amount of ETH in buffer and low cost for performing attack **simultaneously**) whereas the assessed impact on `stETH` holders is on par with the risk of APR decreasing due to new stake (when funds idle temporary) and **less than the standard deviation of APR**. Even though, considered mitigations either decrease user experience drastically or amplify the compexity of the protocol. #### [ACKNOWLEDGED] Abnormal CL rebase condition of the bunker mode depends on the manipulatable value of the validators real balances in `abnormal_cl_rebase.py` ##### Description The function [`_calculate_cl_rebase_between_blocks`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py#L181) in the [`AbnormalClRebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py) class uses real balances of lido validators to calculate the condition of the bunker mode activation. Validator's balance can be increased by depositing more ETH, thereby affecting the value of the specific CL rebase, which is proportional to the mean total lido real balance taken from the `REBASE_CHECK_NEAREST_EPOCH_DISTANCE` or `REBASE_CHECK_FAR_EPOCH_DISTANCE` and the current epoch of the oracle frame. By adding funds to validators balance, it is possible to prevent bunker mode at the condition 3 from activation, allowing the withdrawal queue to process the withdrawal requests for one more oracle frame. ``` python def _calculate_cl_rebase_between_blocks( self, prev_blockstamp: BlockStamp, ref_blockstamp: ReferenceBlockStamp ) -> Gwei: # <...> # # Get Lido validators' balances with WithdrawalVault balance ref_lido_balance_with_vault = self._get_lido_validators_balance_with_vault( ref_blockstamp, self.lido_validators ) prev_lido_balance_with_vault = self._get_lido_validators_balance_with_vault( prev_blockstamp, prev_lido_validators ) ``` ``` python def _get_lido_validators_balance_with_vault( self, blockstamp: BlockStamp, lido_validators: list[LidoValidator] ) -> Gwei: """ Get Lido validator balance with withdrawals vault balance """ real_cl_balance = AbnormalClRebase.calculate_validators_balance_sum(lido_validators) ``` ``` python def calculate_validators_balance_sum(validators: Sequence[Validator]) -> Gwei: return Gwei(sum(int(v.balance) for v in validators)) ``` The scenario may be as follows: 1. Malicious actor has a withdrawal request for a large amount of ETH in the withdrawal queue sent in the current frame. 2. The CL conditions show abnormal situation that provokes a bunker mode activation based on the condition 3. 3. Let's say the `nearest_cl_rebase` or `distant_cl_rebase` in the following inequalities are only slightly lesser than `0`: ``` python nearest_cl_rebase = self._calculate_cl_rebase_between_blocks(nearest_blockstamp, blockstamp) logger.info({"msg": f"Nearest intraframe sampled CL rebase {nearest_cl_rebase} Gwei"}) if nearest_cl_rebase < 0: return True distant_cl_rebase = self._calculate_cl_rebase_between_blocks(distant_blockstamp, blockstamp) logger.info({"msg": f"Distant intraframe sampled CL rebase {distant_cl_rebase} Gwei"}) if distant_cl_rebase < 0: return True ``` 4. By depositing funds to Lido validators' balances the attacker may obtain 24 more hours for withdrawing the funds from the protocol. ##### Recommendation We recommend taking into account changes in validators balances that may happen outside of the protocol, so that an unexpected increase in balances can be matched against a calculated invariant. ##### Update ###### LIDO's response A risk’s reailization requires, **on top** of highly unlikely “bunker mode” premises, **additional preconditions with a low probability** (high amount of ETH in buffer and low cost for performing attack **simultaneously**) whereas the assessed impact on `stETH` holders is on par with the risk of APR decreasing due to new stake (when funds idle temporary) and **less than the standard deviation of APR**. Even though, considered mitigations either decrease user experience drastically or amplify the compexity of the protocol. #### [ACKNOWLEDGED] Normal CL rebase calculation depends on the manipulatable value of validators' effective balances in `abnormal_cl_rebase.py` ##### Description The function [`_calculate_lido_normal_cl_rebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py#L181) in the [`AbnormalClRebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py) class uses effective balances of lido validators to calculate the condition of the bunker mode activation. ``` python def _calculate_lido_normal_cl_rebase(self, blockstamp: ReferenceBlockStamp) -> Gwei: #<...> mean_sum_of_all_effective_balance = AbnormalClRebase.get_mean_sum_of_effective_balance( last_report_blockstamp, blockstamp, last_report_all_validators, self.all_validators ) mean_sum_of_lido_effective_balance = AbnormalClRebase.get_mean_sum_of_effective_balance( last_report_blockstamp, blockstamp, last_report_lido_validators, self.lido_validators ) ``` Method [`calculate_active_effective_balance_sum`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/utils/validator_state.py#L90): ``` python def calculate_active_effective_balance_sum(validators: Sequence[Validator], ref_epoch: EpochNumber) -> Gwei: """ Return the combined effective balance of the active validators from the given list """ effective_balance_sum = 0 for validator in validators: if is_active_validator(validator, ref_epoch): effective_balance_sum += int(validator.validator.effective_balance) return Gwei(effective_balance_sum) ``` Validator's balance can be increased by depositing more ETH, thereby affecting the value of the normal CL rebase, which is proportional to the mean total Lido real balance taken from the previous and the current epochs of the oracle frame. By adding funds to validators balance, it is possible to prevent the bunker mode at the condition 3 from activation, allowing the withdrawal queue to process the withdrawal requests for one more oracle frame. The attack scenario may be as follows: 1. Malicious actor has a withdrawal request for a large amount of ETH in the withdrawal queue sent in the current frame. 3. The CL conditions show abnormal situation that provokes a bunker mode activation based on the condition 3. 4. Let's say there is only a slight difference in the inequality `diff_current_with_normal > self.b_conf.normalized_cl_reward_mistake_rate` in [`is_abnormal_cl_rebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/bunker_cases/abnormal_cl_rebase.py#L60) function. ``` python normal_report_cl_rebase = self._calculate_lido_normal_cl_rebase(blockstamp) diff_current_with_normal = 1 - current_report_cl_rebase / normal_report_cl_rebase if diff_current_with_normal > self.b_conf.normalized_cl_reward_mistake_rate: ``` 5. By depositing funds to Lido validators' balances the attacker may obtain 24 more hours for withdrawing the funds from the protocol. ##### Recommendation We recommend to take into account the changes in validators balances that may happen outside of the protocol, so that an unexpected increase in balances can be matched against a calculated invariant. ##### Update ###### LIDO's response A risk’s reailization requires, **on top** of highly unlikely “bunker mode” premises, **additional preconditions with a low probability** (high amount of ETH in buffer and low cost for performing attack **simultaneously**) whereas the assessed impact on `stETH` holders is on par with the risk of APR decreasing due to new stake (when funds idle temporary) and **less than the standard deviation of APR**. Even though, considered mitigations either decrease user experience drastically or amplify the compexity of the protocol. #### [ACKNOWLEDGED] The `finalization_share_rate` value can be artificially inflated in `accounting.py` ##### Description In the function [`_calculate_report`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/modules/accounting/accounting.py#L168) in [`accounting.py`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/modules/accounting/accounting.py) the value of `finalization_share_rate` is obtained through the call to the [`simulate_full_rebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/modules/accounting/accounting.py#L251) function, which accumulates the `WithdrawalVault` and `LidoExecutionLayerRewardsVault` balances and balances of Lido validators, obtained by calling the [`_get_consensus_lido_state`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/modules/accounting/accounting.py#L213) function: ```python def _get_consensus_lido_state(self, blockstamp: ReferenceBlockStamp) -> tuple[int, Gwei]: lido_validators = self.w3.lido_validators.get_lido_validators(blockstamp) count = len(lido_validators) total_balance = Gwei(sum(int(validator.balance) for validator in lido_validators)) logger.info({'msg': 'Calculate consensus lido state.', 'value': (count, total_balance)}) return count, total_balance ``` By sending ETH to any of `WithdrawalVault`, `LidoExecutionLayerRewardsVault`, or validators' balances (real balances, which are not capped), the share rate obtained after the rebase simulation can be inflated. While the funds used for the attack will be counted towards the protocol income, the scheme allows to predictably manipulate the value of the rebase, which can be used in complex attacks with other protocols that use stETH as a collateral or keep it in liquidity pools. Predictability of the oracle report frames combined with the possibility to manipulate the reported rebase values opens a wide range of opportunities of malicious value extraction. ##### Recommendation We recommend to take into account the changes of the `WithdrawalVault`, `LidoExecutionLayerRewardsVault`, and validators' balance accurately, so that an unexpected increase in balances can be matched against a calculated invariant. If possible, the effective balance of validators should be used for calculations, the real balance should be accounted separately and matched against an invaritant to account for abrupt changes. ##### Update ###### LIDO's response The possible attack vectors can be divided into the following groups: 1. Top-up validators balance on the side of Consensus Layer 2. Top-up vaults balances on the side of Execution Layer. There is a sanity check exists that prevents CL balance increase above the allowed level expressed in APR (and takes into account the elapsed time since the previous report). To address the second vector, there is a positive token rebase limiter which prevents oracle report sandwiching and also suitable for this case (the limiter just smoothes rewards over multiple oracle reports instead of step-wise addition). Worth noting that the Lido-participating validators collect MEV which has outliers during the network congestion and market instability periods. It means that the provided scenario of extremely high EL rewards can be not an attack at all. See also relevant GitHub issues [#405](https://github.com/lidofinance/lido-dao/issues/405) and [#428](https://github.com/lidofinance/lido-dao/issues/428) in the lido-dao repo. #### [NEW] Bunker mode condition calculation is based on average values over skipped frames in `bunker.py` ##### Description The function [`get_cl_rebase_for_current_report`](https://github.com/lidofinance/lido-oracle/blob/e50088b0cc51d3ae8954f5651348fb1405bdf61f/src/services/bunker.py#L89) in the file [`bunker.py`](https://github.com/lidofinance/lido-oracle/blob/e50088b0cc51d3ae8954f5651348fb1405bdf61f/src/services/bunker.py) calculates conditions of the bunker mode activation based on the change in total pooled ether value. In case of skipped report frame bunker mode condition in the function [`is_bunker_mode`](https://github.com/lidofinance/lido-oracle/blob/e50088b0cc51d3ae8954f5651348fb1405bdf61f/src/services/bunker.py#L68) is calculated in the function [`get_cl_rebase_for_current_report`](https://github.com/lidofinance/lido-oracle/blob/e50088b0cc51d3ae8954f5651348fb1405bdf61f/src/services/bunker.py#L97) based on the change in total pooled ether between the last report slot and current ref slot. Large positive rebase value in the skipped report frame may offset the negative rebase in the next frame, bunker mode won't get activated. Skipped frames during bunker condition may be critical for the protocol and affect the bunker mode activation. ``` python def get_cl_rebase_for_current_report(self, blockstamp: BlockStamp, simulated_cl_rebase: LidoReportRebase) -> Gwei: """ Get simulated Cl rebase and subtract total supply before report """ logger.info({"msg": "Getting CL rebase for frame"}) before_report_total_pooled_ether = self._get_total_supply(blockstamp) # Can't use from_wei - because rebase can be negative frame_cl_rebase = (simulated_cl_rebase.post_total_pooled_ether - before_report_total_pooled_ether) // GWEI_TO_WEI logger.info({"msg": f"Simulated CL rebase for frame: {frame_cl_rebase} Gwei"}) return Gwei(frame_cl_rebase) def _get_total_supply(self, blockstamp: BlockStamp) -> Gwei: return self.w3.lido_contracts.lido.functions.totalSupply().call(block_identifier=blockstamp.block_hash) ``` ##### Recommendation We recommend accounting for the skipped frames when calculating the bunker mode condition. Another mitigation strategy can be the sequential oracle report delivery, such that skipped frames won't be possible. ### WARNING #### [ACKNOWLEDGED] Withdrawal queue finalization sequence can be manipulated in `withdrawal.py` ##### Description The function [`get_finalization_batches`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/withdrawal.py#L55) in the file [`withdrawal.py`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/services/withdrawal.py) uses the `WithdrawalVault` and `LidoExecutionLayerRewardsVault` balances to calculate the last finalizable requests through the `WithdrawalQueue` contract. The function [`simulate_full_rebase`](https://github.com/lidofinance/lido-oracle/blob/9fd3a5dd8404a8c84d5664d059396f5649635208/src/modules/accounting/accounting.py#L251) obtains the value of EL rewards vault balance ```python def simulate_full_rebase(self, blockstamp: ReferenceBlockStamp) -> LidoReportRebase: el_rewards = self.w3.lido_contracts.get_el_vault_balance(blockstamp) return self.simulate_rebase_after_report(blockstamp, el_rewards=el_rewards) ``` which then gets passed to the `get_finalization_batches` function and then is used to calculate ETH available to finalize withdrawal requests. ```python def get_finalization_batches( self, is_bunker_mode: bool, share_rate: int, withdrawal_vault_balance: Wei, el_rewards_vault_balance: Wei ) -> list[int]: #<..> available_eth = self._get_available_eth(withdrawal_vault_balance, el_rewards_vault_balance) return self._calculate_finalization_batches(share_rate, available_eth, withdrawable_until_timestamp) ``` This allows the attacker to advance finalization queue by sending funds directly to the `LidoExecutionLayerRewardsVault` contract. The same scheme applies to the `WithdrawalVault` contract. The scenario may be as follows: 1. All the available funds (Withdrawal vault, EL rewards vault, buffered ether) allow to process $100M withdrawals in the withdrawal queue. 2. Queue contains withdrawals requests for the amount of $50M for the next frame. 3. Malicious actor may want to withdraw $50,000,001 within the next frame, which is more than the total available balance in the protocol. In the best case, they will have to wait 24 hours more for another oracle report. 4. By sending $1 to the `WithdrawalVault` or `LidoExecutionLayerRewardsVault` they can make their request be processed within the next frame. ##### Recommendation We recommend taking into account the `WithdrawalVault` and the `LidoExecutionLayerRewardsVault` balance changes accurately, so that an unexpected increase in balance can be matched against a calculated invariant. ##### Update ###### LIDO's response An instant withdrawal request finalization is prevented by requiring a withdrawal request placed not before the report’s collecting timeframe with an additional time margin (also known as a safe border across the oracle codebase). Apart from this, the proposed top-up scenario follows the intented design because, for example, these extra funds have no significant differences from MEV which is collected by the Lido-participating validators. To finalize withdrawals, the protocol is allowed to use all available funds from vaults and buffers unless bunker mode has been activated. Moreover, the withdrawing staker can do the new `submit` request to enrich the buffer with native ether while getting minted `stETH` in return. By doing this, one can achieve an effect similar to a partial withdrawal request fulfillment which doesn’t pose risks to the protocol or its stakers. #### [NEW] `latest_to_exit_validators_count` takes a non-zero value when no validators to exit in `ejector.py` ##### Description In the function [`_get_predicted_withdrawable_epoch`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/ejector/ejector.py#L243) in the file [`ejector.py`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/ejector/ejector.py) the maximum value of `exit_epoch` among validators(`max_exit_epoch_number`) can be less then `blockstamp.ref_epoch + 1 + MAX_SEED_LOOKAHEAD`(according to [`compute_activation_exit_epoch`](https://github.com/ethereum/consensus-specs/blob/dev/specs/phase0/beacon-chain.md#compute_activation_exit_epoch) function). This can happen when all validators have reached their `exit_epochs` and none of the validators have initiated an exit yet. In this case `max_exit_epoch_number` sets to `blockstamp.ref_epoch + 1 + MAX_SEED_LOOKAHEAD`, but at the same time `latest_to_exit_validators_count` leaves untouched with non-zero value: ```python max_exit_epoch_number, latest_to_exit_validators_count = self._get_latest_exit_epoch(blockstamp) max_exit_epoch_number = max( max_exit_epoch_number, self.compute_activation_exit_epoch(blockstamp), ) churn_limit = self._get_churn_limit(blockstamp) remain_exits_capacity_for_epoch = churn_limit - latest_to_exit_validators_count ``` It leads to the reduction of the capacity of the exit queue. ##### Recommendation We recommend setting to zero the `latest_to_exit_validators_count` when there are no validators in the exit queue: ```python activation_exit_epoch = self.compute_activation_exit_epoch(blockstamp) if max_exit_epoch_number < activation_exit_epoch: max_exit_epoch_number = activation_exit_epoch latest_to_exit_validators_count = 0 ``` ### INFO #### [ACKNOWLEDGED] Repetitive deposits and withdrawals are able to exhaust available validators' keys. ##### Description Each deposit of 32 ETH requires an activation of a new validator and each withdrawal of 32 ETH requires an exit of a validator unless there is enough ETH in the buffer and EL contracts that collect the rewards. Repetitive deposits of large amount of ETH will steadily exhaust the validator sets managed by node operators, requiring them to add more validators with new keys to the protocol. If all the set of available validators will be exhausted, deposits to the protocol will become impossible until new set of public keys will be added. Possible scenario: 1. Lido has a single node operator with 10K vaidators and 5K of them are active. 2. The attacker deposits 32K ETH activating 1000 validators then immediately sends a withdraw request. 3. Repeating the step 2 five times will exhaust the whole validator set and will require adding more new validator public keys to the protocol. ##### Recommendation We recommend introducing a delay between deposit and withdrawal in proportion to the deposit amount. ##### Update ###### LIDO's response The overall scenario is coherent with Ethereum staking approaches in general. Nevetheless, the following considerations make the attack surface tolerable: * The Lido protocol has a daily stake limit (it’s 150k ETH (4687 keys) per day) and there is an alerting system for large stake and available keys amount; moreover, the Node Operator Management team keeps track of keys with a reasonable reserve (40k keys at the time of response) for several days of max stake in advance * The attacker have to block their ETH for an undetermenistic time period since the buffer is one of the sources of finalizing requests, hence the attacker must wait further till real deposit of the keys and only then issue a withdrawal request * Waiting for funds isn’t free for the attacker (opportunity costs due to discontinued token rebases for them, exposure to additional risks) #### [ACKNOWLEDGED] Malicious activity of a node operator may affect the normal workflow of the protocol or result in a financial loss ##### Description Node operators have total control of the validators infrastructure, being incentivized for a good behavior only by the reward they get from the protocol. As node operators do not take any risks related to any malicious activity performed by their validators on the consensus layer, they may use their validators to create adverse conditions on the consensus layer, for example leading to a mass slashing. This will allow node operator to activate the bunker mode, blocking the withdrawals from the protocol and leading to a financial loss. While Lido protocol introduces decentralization mechanics such that no node operator concentrates more than 1% of the staking power of the protocol, even in that case node operator operates a significant amount of financial power which is under a risk of being slashed with a node operator not taking any material responsibility for the malicious activity it may perform, other than the loss of the node operator reward received from the protocol. There is a significant degree of trust to the node operator legitimacy and the process of adding a node operator to the protocol hardly guarantees the node operator long-term trustworthiness. ##### Recommendation We recommend introducing a stake of a node operator, analogous to the stake on the consensus layer, so that any malicious activity of the node operator will result in financial loss that raises the cost of the attack. ##### Update ###### LIDO's response Node operators are trusted entities inisde the Lido ecosystem. They are responsible for correctly running the validators as well as following the established Lido DAO policies (e.g., accruing MEV rewards to the dedicated contract using mev-boost infrastructure). The risk of node operator misbehavior is mitigated by maintaining diversified validators set and incentives structure (protocol fee) with monitoring and DAO governance processes. There is a set of policies and management actions: * onboarding new NOs to decentralize further; * limiting the stake amount per a single NO, especially during the onboarding period; * maintaining dashboards and tools to monitor network participation performance, MEV and priority fee distribution (open-sourced https://github.com/lidofinance/ethereum-validators-monitoring); * protocol fee is proved to be a good-behavior incentive in a long-run; * there is a set of on-chain levers can be used by the Lido governance to penalize the misbehaving entities: excluding new stake, disabling fee distribution, prioritized exiting of validators. The later option will be reinforced once withdrawal credentials-initiated exits would be implemented in Ethereum. #### [ACKNOWLEDGED] A protocol user and a node operator can be a single entity performing coordinated actions in the protocol ##### Description A node operator that also has a large deposit in the protocol can coordinate its actions between the consensus and execution layer leading to a complex scenarios that do not follow the regular workflow of the protocol. For example, a node operator can exit a large number of validators prior to the oracle report, so that it will allow to withdraw large amount of funds within a short time frame, which otherwise would require waiting for at least one oracle frame while oracle will process Lido validators exits. ##### Recommendation We recommend to introduce a mode of the protocol that blocks deposits and withdrawals in case of an unusual activity of node operators on the consensus layer. ##### Update ###### LIDO's response It’s practically possible that a node operator on the one hand can be a withdrawing whale staker on another. There are two considerations in general that discourage this approach of self-initiated exits: * an instant withdrawal request finalization is prevented by requiring a withdrawal request placed not before the report’s collecting timeframe with an additional time margin (also known as a safe border across the oracle codebase). * the exit bus oracle reports several times a day while it takes 1-7 days for a validator to be fully withdrawn depending [on the sweep](https://notes.ethereum.org/@launchpad/withdrawals-faq#Q-How-fast-will-I-be-able-to-make-a-partial-withdrawal-Or-when-will-I-get-access-to-the-excess-rewards-that-are-on-my-validator) in the chain. Hence, having a few oracle frames early or later makes lesser impact than the Ethereum sweep current queue state (validator index cursor) and its length. Nevertheless, the incentive to cover up the withdrawing amount with their own exited validators is not straightforward since the overall intent to withdraw a significant amount of funds is legit by design and can be performed using exits of other node operators as well. #### [NEW] Prediction of `withdrawal_epoch` without taking `activation_epoch` into account in `ejector.py` ##### Description In the [`_get_predicted_withdrawable_epoch`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/ejector/ejector.py#L243) function of the [`ejector.py`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/ejector/ejector.py) file among the `validators_to_eject_count` number of validators there may be those which have the `activation_epoch` set very far in the future, but the function tries to predict the withdrawal epoch without taking into account of the `activation_epoch`. Consider an example. We want to predict the withdrawal epoch when ejecting 5 validators. Their `activation_epoch` are `ref_epoch+50`. Suppose the other variables are: ```python max_exit_epoch_number = ref_epoch+10 latest_to_exit_validators_count = 1 churn_limit = 8 ``` The function will return the epoch when all validators will supposedly be withdrawable: ```python (ref_epoch+10) + MIN_VALIDATOR_WITHDRAWABILITY_DELAY ``` while validators can actually be withdrawable no sooner than: ```python (ref_epoch+50 + SHARD_COMMITTEE_PERIOD + 1 + MAX_SEED_LOOKAHEAD) + MIN_VALIDATOR_WITHDRAWABILITY_DELAY ``` because we need to wait at least until the validators "are eligible" to initiate the exit. It leads to creation of more requests to exit than needed, because with a smaller `withdrawal_epoch` value fewer [`future_withdrawals`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/ejector/ejector.py#L149) will be calculated and more validators will need to be ejected to cover amount of withdrawal requests. ```python withdrawal_epoch = self._get_predicted_withdrawable_epoch(blockstamp, len(validators_to_eject) + len(validators_going_to_exit) + 1) # ... future_withdrawals = self._get_withdrawable_lido_validators_balance(blockstamp, withdrawal_epoch) expected_balance = ( future_withdrawals + # Validators that have withdrawal_epoch # ... ) if expected_balance >= to_withdraw_amount: return validators_to_eject ``` ##### Recommendation We recommend revising the calculation of the `withdrawal_epoch` in the presence of "is not eligible" validators (which have `activation_epoch + SHARD_COMMITTEE_PERIOD > ref_epoch + 1 + MAX_SEED_LOOKAHEAD`) in order to prevent unnecessary ejections. ## Incidents report #### All oracles at different points in time must build the same report for a particular epoch ##### Description In the function [`submitReportData`](https://github.com/lidofinance/lido-dao/blob/v2.0.0-rc.2/contracts/0.8.9/oracle/AccountingOracle.sol#L386) of the `AccountingOracle` contract, oracles must submit the same report data at different points in time on all servers in order to keep track of historical share rates to values of a certain epoch. It may affect the calculation of average values in the bunker mode. ##### Solution In the function [`_calculate_report`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/accounting/accounting.py#LL145C9-L145C26) of the `Accounting` class, during reports calculation, oracles rely on the same slot - `ref_slot`, so their report data is the same. If the data of the reports still differ, then the final report is decided by consensus. As a result, the data of only one report is processed on-chain. The `ref_slot` itself is actually calculated from the current oracle time in the [`_getCurrentFrame`](https://github.com/lidofinance/lido-dao/blob/v2.0.0-rc.2/contracts/0.8.9/oracle/HashConsensus.sol#L644) function. The `ref_slot` is determined using the [`get_blockstamp_for_report`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/modules/submodules/consensus.py#L191) function. In it, the oracle checks that: - `CONTRACT_VERSION` and `CONSENSUS_VERSION` match on-chain versions; - report not created yet; - have the right to submit reports; - current `ref_slot` finalized; - deadline not reached. If one of the checks fails, then the oracle misses creating a report for this `ref_slot`. #### Validator did not exit by Ejector's request to exit ##### Description In the function [`_processExitRequestsList`](https://github.com/lidofinance/lido-dao/blob/v2.0.0-rc.2/contracts/0.8.9/oracle/ValidatorsExitBusOracle.sol#L364) of the `ValidatorsExitBusOracle` contract, a `ValidatorExitRequest` event is emitted to request an exit for the next validator, but at the consensus level, the validator may not exit during the frame. ##### Solution If the Ejector made a request to exit for validator, but the node operator did not initiate exit on that validator until the Ejector's next report, then the Ejector considers the validator as "delayed" when calls the function [`count_operator_delayed_validators`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/services/exit_order_iterator_state.py#L74) of the `ExitOrderIteratorStateService` class. As a consequence, node operators with the least number of "delayed" validators will be selected first in the function [`_predicates`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/services/exit_order_iterator.py#L127) to eject validators. If the node operator did not initiate the exit of the validator before the `ref_slot` of the next Accounting frame, then such validators will be included in the Accounting report as "stuck" by the function [`get_lido_newly_stuck_validators`](https://github.com/lidofinance/lido-oracle/blob/3.0.0-rc.2/src/services/validator_state.py#L42) of the `LidoValidatorStateService` class. Data about stuck validators is sent by the Accounting module as extra data. In this case, the node operator will receive a [penalty](https://github.com/lidofinance/lido-dao/blob/v2.0.0-rc.2/contracts/0.4.24/nos/NodeOperatorsRegistry.sol#L1327) until it makes a "refund" for these validators by the function [`updateRefundedValidatorsCount`](https://github.com/lidofinance/lido-dao/blob/v2.0.0-rc.2/contracts/0.8.9/StakingRouter.sol#L282) of the `StakingRouter` contract.
Last changed by  
 
George Avsetsin
183

Read more

Sandbox module

Деплой модуля

11/29/2023
Untitled

Это первый документ из предстоящей серии документов по спецификации Staking Router. Объём видится огромным и я бы хотел подойти к этом процессу итеративно. Документ представляет собой заголовки и размышления о тех частях, который на мой взгляд должны быть включены в грядущую спецификацию и следующий апгрейд роутера. На этой стадии хотелось бы определиться со скоупом того, что мы хотим проработать детальнее, а что отбросить.

10/9/2023
Manual Exit Bus Monitoring

The manual how to use the Lido CLI to track the status of validator exits. Install CLI Use the steps from readme in a repository to install the CLI: https://github.com/lidofinance/lido-cli#installation Unsettled requests Takes data from the Consensus Layer on a head block and matches it with the requested validators for each operator based on lastRequestedIndex. All validators whose index <= lastRequestedIndex are considered requested. ## Returns a list of operators and validators that were requested but not exited

5/12/2023
Draft monitoring app after shappella upgrade.

Description Необходимо задеприкеить текущий операторc виджет, поскольку он использует устаревшие контракты, а текущий вариант нужно переписать. В связи с этим, все метрики, которые есть на странице, должны быть перенесены на другой дашборд. Также нужно переписать валидацию, поиск и сабмит ключей, используя функционал KAPI, чтобы сабмит ключей был возможен под любым модулем. В связи с апгрейдом Lido до V2, необходима новая система, включающая несколько воркеров и дашборд. Задача дашборда - отображать все текущие характеристики с операторс виджета, а также смотреть данные в разбивке по модулям, операторам, такие как капасити по эфиру, ключам (вышедшим, застейканым), таргет лимит. Также нужна визуализация данных: как долго нод операторы выводят ключи, как много ключей потребуется вывести в ближайшее время (сколько ожидаем в ближайшем репорте). Визуализация данных должна поддерживать историчность, то есть дашборд должен уметь показывать данные по репортам за прошлый период. Кроме того, задача дашборда - отображать финализацию заявки: сколько сейчас нефинализированного steth в очереди, сколько во всех буферах: буфер, EL vault, WV vault. Задача демона - наполнять хранилище данными, чтобы дашборд мог отображать метрики с указанными фильтрами в удобном виде. Например, одна из детальных задач демона - хранить exit_epoch ключа. Другая задача системы - рассылать уведомления о предстоящем выводе ключей за день, 6 часов, 1 час. Вторая задача - рассылать алерты, если нод оператор задерживается с выводом ключа из сети. Предполагаем, весь этот функционал, может поставляться в docker-compose файле с подключенной и преднастроенной графаной для визуализации метрик, и рассылок нотофикаций Ask:

5/11/2023
Read more from George Avsetsin
Published on HackMD