# Safe Head: happy case ## Basic definitions and notation We will use $S$ for slots, $B$ for blocks, $E$ for epochs. Often times we will use $S$ for the block at slot $S$ abusing notation when no confusion could arise. For example, a forkchoice diagram of the form ```graphviz digraph{ rankdir=RL 10 -> 9; 11 -> 10; 12 -> 11; 13 -> 11; 13 -> 12 [style=invis]; 14 -> 13 } ``` denotes a reorg where the block in slot 13 has the block in slot 11 as parent. We will say things like *The parent of $S$ is $T$* to mean the parent of the block in slot $S$ is the block in slot $T$. All our forkchoice diagrams will have time going from left to right, that is a block $T$ to the right of a block $S$ will correspond to a slot $T>S$. For a slot $S$ we will denote by $H_S$ the canonical head during $S$. When this depends on the time withing the slot, eg. if the block has arrived or not, we will explicitly specify it. **LMD Weights** For a block $B$ at slot $S$, that is a leaf of the forkchoice tree, we will write $LMD_B = LMD_S$ to mean the LMD GHOST weight of the branch supporting this head. That is, in terms of the forkchoice implementation the *node's weight** For example, assuming that each node in the diagram above was voted by a single validator with 10ETH of effective balance, we would have $LMD_{14} = 50$ and $LMD_{12} = 40$. For two different tips at $S$ and $S'$ with equal FFG information (that is equal justification and finalization checkpoints) we have $LMD_S > LMD_{S'}$ implies that $S$ is preferred over $S'$ as the current canonical head. In the example above, the node at $14$ would be preferred over that at $12$. We will abuse notation and also use $LMD_S$ to denote the LMD weight of a node even when it is not a tip of forkchoice. With the same assumptions as above we would have $LMD_9 = 10$ and $LMD_{13} = 40$. Recall that when a validator cast a vote, its previous vote is removed from the corresponding forkchoice node's balance and weight. Consider a chain without any forks, there are exactly 32 validators $v_i$, $i=0,\ldots,31$ with 10 ETH effective balance and each has voted sequentially, that is validator $i$ voted at slot $i \,(32)$ for each one of the blocks. ```graphviz digraph{ rankdir=RL 1->0; 2->1; 3[label="..."]; 3->2; 97->3; } ``` In this case, the weight of the vote at slot $1$ was removed from this slot and assigned to slot $33$ when validator $v_1$ voted during epoch $1$. Then it was removed from block $33$ and assigned to $65$ during Epoch $2$ and finally removed from $65$ and assigned to $97$ during epoch $3$. We see that $LMD_{97} = 320$ (the total effective balance) and $LMD_{1}=0$. For a block $B$ with parent $P$, we will define $BAL_B = LMD_B - LMD_P$. That is the total stake that voted directly for $B$ and we will refer to it as the *balance of $B$*. In the first example above, the balance of every node is $10$. However, in the last example we have $BAL_1 = BAL_{33} = BAL_{65} = 0$ while $BAL_{97} = 10$. More generally for a slot/block $S$ and its ancestor $T$ we will write $LMD_{T,S}:= LMD_S-LMD_T$, that is, the relative weight of votes cast for blocks supporting $S$ after $T$. In particular we have $BAL_S = LMD_{P,S}$. In the last example above we have $LMD_{1,97} = 320$ and $LMD_{95,97} = 20$. **Safety Assumptions** Let $Tot$ be the total effective balance of validators divided by 32, this is roughly the total stake attesting per slot. We use PB to denote the proposer boost weight, currently set on mainnet to $0.4$. Let $\beta$ be percentage staked by malicious validators. We assume that $$\beta < \frac{32 - PB}{64} \simeq 0.49.$$ We assume that validators are online and attesting, this is measured below by the amount of stake of missing attestations that we will keep at a minimum (eg below 5% of the total stake). We assume that validators are equidistributed, each committee has the same proportion of honest/dishonest validators. ## Safe Head definition **Setup** Let $S_0\geq 32$ be the current slot, $E_0 := \lfloor \frac{S_0}{32}\rfloor$ be the current epoch. The current canonical head is $H_0 = H(S_0)$ at slot $S_0$, and its parent block is $B$ at slot $S_0 -1$. We assume that $E_0-1$ is justified, ie. the current justified checkpoint is of the form $(\text{Root:} R,\, \text{Epoch:} E_0-1)$. Let $\gamma$ be the total stake of attestations that could have been cast in the last 32 slots and have not been acounted for, that is, those attestations have not been seen yet either over gossip nor included in any block that is currently in forkchoice. Let $\eta$ be the total stake of attestations (only the last possible vote of each validator) that have been cast for non-canonical blocks. **Lemma 1.** Let $T$ be any ancestor of $B$, we have that $$ LMD_{T, B} \geq \max(S_0 - 1 - T,32)\cdot Tot - \gamma - \eta $$ **Proof.** This is simply the fact that the maximum possible number of votes is given by $\max(S_0 - 1 - T, 32)-\eta$ and from these votes the worst that can happen is that the attestations that weren't accounted for (ie $\gamma$) are given to non-canonical blocks. <div style="text-align: right">QED</div> A simple variation of this Lemma is **Lemma 2.** Let $S$ be any contending block of $B$ with common ancestor $T$, then $$LMD_{B} - LMD_S \geq \max(S_0 - 1 - T,32)\cdot Tot - \gamma - 2 \eta.$$ **Proof.** The proof is the same as Lemma 1, the only difference is the extra $\eta$ votes that not only do not count for $B$ but may count for the contending branch of $S$. <div style="text-align: right">QED</div> **Theorem 2.** Under the above conditions, let $\kappa$ be the highest LMD weight of any head contending with $B$ or any ancestor of $B$ (excluding $B$ itself) $$\gamma + \kappa + (2\beta - 1)\cdot Tot + PB \cdot Tot \leq LMD_B.$$ the block $B$ is final. **Proof.** We want to prove the following. For any slot $S$, let $H(S)$ be the canonical head (from the point of view of an honest validator) at $S$. Then $H(S)$ is a descendant of $B$ for $S \geq S_0$. If $S = S_0$, there is nothing to prove. Let $S > S_0$ be a slot that does not satisfy the hipothesis, that is $H(S)$ does not descend from $B$. Let $C$ be the slot of their common ancestor, let $H_B$ be the best head descending from $B$ at the time $S$, and let $S'$ be the best head on the chain of $S$ at slot $S_0-1$, so that we have ```graphviz digraph{ rankdir=RL C; dots1[label="..."]; dots2[label="..."]; dots3[label="..."]; dots4[label="..."]; S1[label="S'"]; S0[label=<S<sub>0</sub>>]; HB[label=<H<sub>B</sub>>]; S[label="S=H(S)"]; S->dots2 -> S1 -> dots1 ->C; dots2->HB[style=invis]; HB->dots4->S0->B->dots3->C; B->S1[style=invis]; dots3->dots1[style=invis]; } ``` Here notice that $H_B$ may not to descend from $S_0$ and the slot of $H_B$ is less than $S$. Without loss of generality we may assume that $S$ is the minimal slot with this property. This implies that $S_0+i$ either was in the chain of $B$ or was not the canonical head during $S_0+i$. for all $i \geq 0$ such that $S_0 +i \leq S$. For simplicity of exposition, let us first make the following assumptions: - both $S$ and $H_B$ have the same justified checkpoint $(R, E-1)$. In this case we must have $LMD(S) > LMD(H_B)$. - All of $C$, $S$ and $H_B$ are slots in the same epoch $E$. - There are only two viable heads at all times (that is, we have a simple fork) For each $i = 0, ..., S-S_0-1$, honest validators voted for the block $H_{S_0 + i}$ during $S_0 + i$, and this block is a descendant of $B$ by hypothesis. Some of these votes however may have been supporting the $S$ branch before, that is, any vote from an honest validator that was before $C$ would have supported the branch $S$ and no longer does. Honest validators votes on slots between $C+1$ and $B$ however, were not supporting the chain of $S$ before, and will therefore not change the difference in the LMD weight of the chain of $B$ with respect to that of $S$. On the other hand, all validators that vote between slots $S_0$ and $S-1$ are votes during epoch $E$. By our assumptions, their previous votes were cast during epoch $E-1$, therefore before slot $C$. The votes that are added to the chain of $B$ (not changing its weight) and subtracted from the chain of $S$ are then $$ (1-\beta) \cdot Tot \cdot (S-S_0) $$ On the other hand, malicious validators would be voting for the contending branch of $S$, and for each vote they cast that was previously voting for the B branch, it would now count towards $S$. In the worst case scenario, every single malicious vote, shifts from the B branch to the S branch. These votes, with total weight $$ \beta \cdot Tot \cdot (S-S_0), $$ will be removed from the $B$ branch and added to the LMD weight of the $S$ branch (without changing its weight since they were already counted towards $S$). We can assume also that all of the witheld attestations were malicious, were valid, and could be added to the $S$ branch, and that also the block at $S$ was timely, thus proposer boosted. This accounts for a total LMD weight for the branch of $S$ at the start of its slot of at most $$LMD_{H_S} \leq LMD_{S´} - (1-\beta)\cdot Tot \cdot (S-S_0) + \gamma + PB \cdot Tot.$$ On the other hand the branch supporting $B$ will have weight at least $$LMD_{H_B} \geq LMD_B - \beta \cdot Tot \cdot (S-S_0).$$ From $LMD_{H_S} > LMD_{H_B}$ it follows that $$ \gamma + LMD_{S'} + (2\beta - 1)\cdot Tot \cdot (S-S_0) + PB \cdot Tot > LMD_{B}$$ Since $2\beta < 1$ we obtain $$\gamma + LMD_{S'} + (2\beta - 1)\cdot Tot + PB \cdot Tot > LMD_B.$$ We now have reduced the problem to prove that our assumptions above were not necessary. Let us first deal with the second assumption, that is that $C$, $S$ and $H_B$ are all in the same epoch $E$. Let us suppose first that $C$ and $S_0$ are on the same epoch $E$, but not necessarily is the case for $H_B$ and $S$, that is, the fork is long enough to cross epoch boundaries but the common ancestor is close to the block $B$ which we want to prove is final. Notice that for each slot during epoch $E$ and after slot $S_0$, the difference of LMD weight between the tip of the chain supporting $B$ and the chain supporting $S$ increases by an amount of $(1-2\beta)\cdot Tot > 0$. Since $E-1$ was justified, and all votes during $E$ are for the same target, when crossing the epoch boundary to $E+1$ we have that the justification of the $B$ chain is at least the justification of the $S$ chain, that is $$H_B.\text{justified_checkpoint.Epoch} \geq H_{S}.\text{justified_checkpoint.Epoch}$$ This is because $\beta < 0.5$ and honest validators are attesting on the $B$ chain. It follows that honest validators will continue voting during $E+1$ on the $B$ chain, while malicious validators can continue voting on the $S$ chain. At the start of epoch $E+1$, the difference on LMD weights of both chains has increased by $(1-2\beta)\cdot Tot \cdot (32 (E + 1)- S_0)$ (accounting for the slots between the $S_0$ and the start of $E+1$). During $E+1$, validators that had voted up to slot $C$ during $E$ will account for a difference in LMD weights of $(1-2\beta)\cdot Tot \cdot (C + 1 - 32E)$. Malicious validators that had voted during $E$ on the chain supporting $B$, that is between the slots $C+1$ and $S_0-1$, can shift their vote during $E+1$ to the chain of $S$, these will increase the weight on $S$ and decrease it from $B$, the maximum difference caused by these is $2 \beta \cdot T\cdot (S_0 - 1 - C)$. It follows that during a slot $S$ in epoch $E+1$ we have $LMD_{H_B} - LMD_{S}$ is at least $$LMD_B - LMD_{S'} + (1-2 \beta) \cdot Tot \cdot (33 +C - S_0) - 2 \beta \cdot Tot \cdot(S_0 -1 - C)$$ By Lemma 2 we have $$LMD_B - LMD_{S'} \geq (S_0 - 1 - C) \cdot Tot - \gamma - 2\eta.$$ From where we obtain that during epoch $E+1$ we have $$ LMD_{H_B} - LMD_{S} \geq 32 (1 - 2 \beta) \cdot Tot - PB \cdot Tot - 2 \gamma - 2 \eta$$ where we have added another $\gamma$ terms of withdeld votes that counted for the $S$ branch and proposer boost to the last block produced by the malicious branch. We see that the RHS is positive if $$ 2(\gamma + \eta) < (32 (1 - 2 \beta) - PB) \cdot Tot, $$ which is guaranteed by our assumptions on $\beta$. If the slot $S$ is in an epoch higher than $E+1$, the situation does not change, since malicious validators have already been accounted for the branch $S$ and honest validators for the branch $B$, thus the LMD difference will continue to increase, and the same argument can be applied for justification.
Last changed by  
 
potuz
189

Read more

Validator performance tracking

We explain in detail the different components that make up an Ethereum validator's reward. We describe common situations that make stakers miss some of these rewards and show how to diagnose the -- typically uncommon-- scenarios in which there are actual problems with the software and action is required. 1. Introduction A common concern of node operators is --Is my node performing well?--, or -- Is my node performing at least as the average?--. The essence of the question is in fact --Should I be changing anything in my setup or not?--. It turns out that these questions do not really have an easy answer. There have been many attempts to trying to create a single score that would measure a validator's performance, but they all fail because of different reasons to answer the very basic questions mentioned above. Instead of trying to track a single indicator (or a ton of useless ones and be worried about some small variance on any of those), validators should analyze why their performance took a hit: did I miss an attestation due to a late block, a reorg, something else out of my control?, did I miss a proposal due to my EL being stressed or was my block reorged because it was received late by the network?. In this document, we will try to help node operators to identify warnings, their cause, and make informed decisions on when to tweak their setup. In order to do this we first start by describing, in some level of technical detail, the different components of a validator's reward. We then move to analyze each one of them separately as they are susceptible to different external factors. We finally describe how to exploit existing monitoring tools (grafana, logs, beacon explorers, etc) to diagnose causes for drops in rewards. The organization of this post is as follows. In section 2 we describe the statistical expected variance in income amongst perfectly working validators. In section 3 we describe all the components that make attestation rewards, how much to expect in different scenarios of attestation inclusion and how to diagnose a missing attestation. In section 4 we describe the analogous situation for proposals. In section [5](# 5-Sync-committee-messages) we describe quickly sync committee rewards. In section 6 we cover transaction tips. In section 7 we mention some tools to monitor performance that are available in Prysm, and finally in section 8 we mention how to get support from the Prysm team. 2. Reward variance is evil

4/4/2023
Validators vs Builders, trust assumptions

Often times we want to make statements of the form, if x% of validators are honest, property P holds, or equivalently, under the supposition that an attacker holds less than y% of the total stake, statement Q is true. Examples of these are of the form: "With a proposer boost of 40% then an attacker with more than 60% of the stake can reorg arbitrarily the block right before he's assigned to propose". "If 90% of the validators are honest, and we have seen all attestations and all blocks in the last two epochs, then the head of the chain cannot be reorged", etc. These statements become much harder to make in the context of PBS, where trust assumptions on validators may become trust assumptions on builders, which is a much more reduced set. CL data vs EL data In a context where most validators defer their execution payload, data blobs, or any future non-pure consensus part of the block to an external builder. The builder may have a final decision on certain aspects of blocks, thus whatever trust assumptions we are making on honesty of validators may cease to be true. In other cases, the builder has zero or very little impact. For example, a trust assumption of the form: "x% of validators are not willing to arbitrarily fork blocks". The validity of this statement is the same under PBS or not, builders have very little bearing on the LMD weight of a given chain, validators alone decide their attestation targets.

3/4/2023
Report on another bad reorg on Goerli

The following reorg was seen on Goerli Unfortunately this graph iss ordered by slot number (left to right) and not by timestamps. The lightblue epoch on the left is 155221, the other lightblue epoch on the right is 155223, the white one in the middle is (you guessed it) 155222. When epoch 155221 ends, the last block that the chain has seen is 4967101 (slot 29 in that epoch, the top branch is based on that block). This block has not enough votes to justify epoch 155221, so the chain advances keeping the justified epoch to be 155220 The chain advances on the top branch, everything is normal, all clients participating, until right after slot 4967121 After that block is imported, block 4967102 appears (4 minutes late). This is the last block in epoch 155221, in slot 30, since the block in the next slot was never proposed (or seen). The block 4967102 has enough votes to justify 155221, this splits the view of the chain: those that have deployed the witholding fix (only prysm) will not see this block as head and continue to build on top of the canonical chain. Those that didn't deploy it yet will immediately reorg to this block. Lodestar proposes block 155224 on top of that block (the first block in the bottom branch) and from there on, Lighthouse, Teku and Nimbus follow.

2/11/2023
Capella interop, Prysm - Nethermind

Build Nethermind git clone --recursive -b feature/shanghai-eip-4895-withdrawals https://github.com/NethermindEth/nethermind.git cd nethermind/src/Nethermind dotnet build Nethermind.sln -c release export NETHERMIND=$(pwd)/Nethermind.Runner/bin/Release/net6.0 Build prysm git clone --recursive -b capella https://github.com/prysmaticlabs/prym.git cd prysm bazel build //cmd/beacon-chain

11/10/2022
Read more from potuz
Published on HackMD