Updated CSM Performance Oracle metric

This doc is closed!

Pls comment on and refer to https://hackmd.io/@lido/csm-v2-internal

Currently, CSM Performance Oracle uses the included attestations (inclusion delay and correctness are not considered) rate as a performance proxy for the CSM validators:

P_{v a l i d a t o r} = \frac{A_{i n c l u d e d}}{A_{a s s i g n e d}}

This metric is an excellent proxy for the validator's liveliness since each Ethereum validator should submit one attestation each epoch. However, two more validator duties are not accounted for in the current approach: block proposals and sync committee participation. These duties are much less frequent than attestations but are vital for the network. As CSM evolves and gains a stake share in the Lido on Ethereum protocol, a more robust and accurate performance proxy is required.

Unified performance rating

Performance rate representation in the form of the relation between fulfilled and assigned duties has proven to be robust and easy to understand. It is proposed that this approach be kept while enriching it with the block proposal and sync committee duties. The resulting formula will take the form:

P_{v a l i d a t o r} = C_{a} \times A_{e f f} + C_{b} \times B_{e f f} + C_{s} \times S_{e f f}

where

C_{a}, C_{b}, C_{s}

are weight coefficients and

A_{e f f}, B_{e f f}, S_{e f f}

are effectiveness ratings for attestations, block proposals and sync committee respectively.

A similar approach is used by beaconcha.in to calculate validator effectiveness.

Weight coefficients

The weight coefficients are defined as follows (according to eth2book):

C_{a} = \frac{54}{64}, C_{b} = \frac{8}{64}, C_{s} = \frac{2}{64}

However, this coefficients assume validator rewards over the long period of time during which all 3 duties were assigned. Since CSM Performance Oracle has a relatively short frame it is required to account for the scenarios when not all duties where assigned within the frame.

All three duties

C_{a} = \frac{54}{64}, C_{b} = \frac{8}{64}, C_{s} = \frac{2}{64}

Attestations and block proposals

C_{a} = \frac{54}{62}, C_{b} = \frac{8}{62}, C_{s} = 0

Attestations and sync committee

C_{a} = \frac{54}{56}, C_{b} = 0, C_{s} = \frac{2}{56}

Attestations only

C_{a} = 1, C_{b} = 0, C_{s} = 0

Effectiveness ratings

Attestations

The most general approach to attestation effectiveness calculation is:

A_{e f f} = \frac{A_{a c t u a l R e w a r d}}{A_{m a x i m a l R e w a r d}}

Actual attestation reward is calculated using a sophisticated algorithm described here. One can see that actual reward depends on the multiple factors like vote correctness and inclusion delay. These factors are not fully under the Node Operators control given that they validate from home and not top-tier data centers. Choice of the Ethereum clients might also affect attestation reward performance.

Since CSM is primarily focused on operators validating from home it is worth calculating attestation submission rate as attestation effectiveness.

A_{e f f} = \frac{A_{i n c l u d e d}}{A_{a s s i g n e d}}

This approach is vulnerable to the edge case of significant percentage of the incorrect attestation votes alongside high submission rate. However, this case is only reachable with the explicit malicious intent since all of the current Ethereum clients are designed for the maximal performance and would not demonstrate such behaviour specific code or configuration modifications. Any malicious case can still be detected by the tools like beaconcha.in or rated.network and counteracted with the Node Operator penalisation and ejection from the validator set.

Block proposals

With the block proposals the situation is simple and straightforward. Node Operators should maintain their setup in a way that allows them to submit valid block in time even if they are operating from the distant locations. This requirement comes from the fact that block production is vital duty of any validator. Unlike attestations where individual invalid or delayed votes can not disturb the network, block proposal is a binary duty. The block is either proposed or not. Hence, the block proposal effectiveness can be calculated as follows:

B_{e f f} = \frac{B_{p r o p o s e d}}{B_{a s s i g n e d}}

Sync committee

Sync committee is a very rare duty of the validator. The effectiveness of this duty can be calculated as:

S_{e f f} = \frac{S_{i n c l u d e d}}{S_{a s s i g n e d} - B_{m i s s e d}}

It is crucial to subtract missing blocks from the denominator since sync committee votes can only be included into the proposed blocks and block proposers are not correlated with the sync committee participants.

Average network performance

Using the formula for the individual validator performance the average network performance can be calculated as average of all individual validators performances.

P_{n e t w o r k} = \frac{\sum_{i = 0}^{N_{v a l i d a t o r s}} P_{v a l i d a t o r_{i}}}{N_{v a l i d a t o r s}}

However, it might be resource consuming to calculate individual performance for each validator. Also, we need to account for the change in the active validators number. To simplify the calculations an alternative approximate formula can be used:

P_{n e t w o r k} = C_{a} \times A_{e f f_{n e t w o r k}} + C_{b} \times B_{e f f_{n e t w o r k}} + C_{s} \times S_{e f f_{n e t w o r k}}

where:

A_{e f f_{n e t w o r k}} = \frac{A_{i n c l u d e d_{n e t w o r k}}}{A_{a s s i g n e d_{n e t w o r k}}}

B_{e f f_{n e t w o r k}} = \frac{B_{p r o p o s e d_{n e t w o r k}}}{S l o t s_{f r a m e}}

S_{e f f_{n e t w o r k}} = \frac{S_{i n c l u d e d_{n e t w o r k}}}{(S l o t s_{f r a m e} - S l o t s_{f r a m e_{m i s s e d}}) \times 512}

Summary

Given all of the calculations above, the final algorithm will look as follows:

CSM Performance Oracle collects data about attestations, block proposals, and sync committee participation for all validators participating in the network;
CSM Performance Oracle calculates the network's average "Unified performance rating" using the formula:
$P_{n e t w o r k} = C_{a} \times \frac{A_{i n c l u d e d_{n e t w o r k}}}{A_{a s s i g n e d_{n e t w o r k}}} + C_{b} \times \frac{B_{p r o p o s e d_{n e t w o r k}}}{S l o t s_{f r a m e}} + C_{s} \times \frac{S_{i n c l u d e d_{n e t w o r k}}}{(S l o t s_{f r a m e} - B_{f r a m e_{m i s s e d}}) \times 512}$
CSM Performance Oracle calculates the "Unified performance rating" for each CSM validator using the formula:
$P_{v a l i d a t o r_{C S M}} = C_{a} \times \frac{A_{i n c l u d e d}}{A_{a s s i g n e d}} + C_{b} \times \frac{B_{p r o p o s e d}}{B_{a s s i g n e d}} + C_{s} \times \frac{S_{i n c l u d e d}}{S_{a s s i g n e d} - B_{m i s s e d}}$
CSM Performance Oracle uses the following values:
- If all three duties were assigned:
  $C_{a} = \frac{54}{64}, C_{b} = \frac{8}{64}, C_{s} = \frac{2}{64}$
- If attestations and block proposals were assigned:
  $C_{a} = \frac{54}{62}, C_{b} = \frac{8}{62}, C_{s} = 0$
- If attestations and sync committee were assigned:
  $C_{a} = \frac{54}{56}, C_{b} = 0, C_{s} = \frac{2}{56}$
- If only attestations were assigned:
  $C_{a} = 1, C_{b} = 0, C_{s} = 0$
CSM Performance Oracle compares
$P_{v a l i d a t o r_{c s m}}$ with
$P_{n e t w o r k} - P_{t h r e s h o l d}$ to make a decision regarding reward distribution;

Appendix 1. Future evolution

CSM v2 will introduce custom performance thresholds for the identified solo stakers. This feature will allow to make performance rating calculation more precise (include vote accuracy and inclusion delay) while keeping the performance threshold low enough for solo stakers exclusively. Without a separate threshold accurate performance metric with the low threshold will allow for a significant downtime of the validators operating from DCs which is net negative for the protocol and the overall network. On the other hand accurate metric with the hight performance threshold will not allow home stakers with the limitations described above to get rewards.

Despite the fact that the update of the CSM Performance Oracle will be delivered within CSM v2 it is proposed to follow a "staged" approach, and first implement a simplified attestation effectiveness accounting. It will allow for the shorter development and release time. Also, it will help to gather more real-world data about CSM validators performance and make an informed decision on the value of the lowered performance threshold for the identified solo stakers.