# Report on another bad reorg on Goerli The following reorg was seen on Goerli ![](https://i.imgur.com/Ej6Nkqq.jpg) Unfortunately this graph iss ordered by slot number (left to right) and not by timestamps. The lightblue epoch on the left is 155221, the other lightblue epoch on the right is 155223, the white one in the middle is (you guessed it) 155222. - When epoch 155221 ends, the last block that the chain has seen is 4967101 (slot 29 in that epoch, the top branch is based on that block). - This block has not enough votes to justify epoch 155221, so the chain advances keeping the justified epoch to be 155220 - The chain advances on the top branch, everything is normal, all clients participating, until right after slot 4967121 - After that block is imported, block 4967102 appears (4 minutes late). This is the last block in epoch 155221, in slot 30, since the block in the next slot was never proposed (or seen). - The block 4967102 has enough votes to justify 155221, this splits the view of the chain: those that have deployed the witholding fix (only prysm) will not see this block as head and continue to build on top of the canonical chain. Those that didn't deploy it yet will immediately reorg to this block. - Lodestar proposes block 155224 on top of that block (the first block in the bottom branch) and from there on, Lighthouse, Teku and Nimbus follow. - Prysm does not see this chain as head since the top branch is competing in its view and has a heavier weight with 15 blocks worth of votes. So when prysm proposes block 4967132 it does so on the top branch. - This also explains the higher vote count on 4967121 as every prysm node continued voting on it as head, despite the chain advancing on the bottom. - The other clients continue seeing the bottom branch with higher justification, so they continue building on top of it. - When the epoch changes to 155223, then every justification is realized, and now every client sees the two branches with the same realized justified epoch: 155221. Thus everyone reorgs back to the top chain. - Had prysm not deployed the witholding fix, the reorg back would have been to block 4967121 instead of 4967132
Last changed by  
 
potuz
198

Read more

Validator performance tracking

We explain in detail the different components that make up an Ethereum validator's reward. We describe common situations that make stakers miss some of these rewards and show how to diagnose the -- typically uncommon-- scenarios in which there are actual problems with the software and action is required. 1. Introduction A common concern of node operators is --Is my node performing well?--, or -- Is my node performing at least as the average?--. The essence of the question is in fact --Should I be changing anything in my setup or not?--. It turns out that these questions do not really have an easy answer. There have been many attempts to trying to create a single score that would measure a validator's performance, but they all fail because of different reasons to answer the very basic questions mentioned above. Instead of trying to track a single indicator (or a ton of useless ones and be worried about some small variance on any of those), validators should analyze why their performance took a hit: did I miss an attestation due to a late block, a reorg, something else out of my control?, did I miss a proposal due to my EL being stressed or was my block reorged because it was received late by the network?. In this document, we will try to help node operators to identify warnings, their cause, and make informed decisions on when to tweak their setup. In order to do this we first start by describing, in some level of technical detail, the different components of a validator's reward. We then move to analyze each one of them separately as they are susceptible to different external factors. We finally describe how to exploit existing monitoring tools (grafana, logs, beacon explorers, etc) to diagnose causes for drops in rewards. The organization of this post is as follows. In section 2 we describe the statistical expected variance in income amongst perfectly working validators. In section 3 we describe all the components that make attestation rewards, how much to expect in different scenarios of attestation inclusion and how to diagnose a missing attestation. In section 4 we describe the analogous situation for proposals. In section [5](# 5-Sync-committee-messages) we describe quickly sync committee rewards. In section 6 we cover transaction tips. In section 7 we mention some tools to monitor performance that are available in Prysm, and finally in section 8 we mention how to get support from the Prysm team. 2. Reward variance is evil

4/4/2023
Safe Head: happy case

Basic definitions and notation We will use $S$ for slots, $B$ for blocks, $E$ for epochs. Often times we will use $S$ for the block at slot $S$ abusing notation when no confusion could arise. For example, a forkchoice diagram of the form digraph{ rankdir=RL 10 -> 9; 11 -> 10; 12 -> 11; 13 -> 11; 13 -> 12 [style=invis];

3/23/2023
Validators vs Builders, trust assumptions

Often times we want to make statements of the form, if x% of validators are honest, property P holds, or equivalently, under the supposition that an attacker holds less than y% of the total stake, statement Q is true. Examples of these are of the form: "With a proposer boost of 40% then an attacker with more than 60% of the stake can reorg arbitrarily the block right before he's assigned to propose". "If 90% of the validators are honest, and we have seen all attestations and all blocks in the last two epochs, then the head of the chain cannot be reorged", etc. These statements become much harder to make in the context of PBS, where trust assumptions on validators may become trust assumptions on builders, which is a much more reduced set. CL data vs EL data In a context where most validators defer their execution payload, data blobs, or any future non-pure consensus part of the block to an external builder. The builder may have a final decision on certain aspects of blocks, thus whatever trust assumptions we are making on honesty of validators may cease to be true. In other cases, the builder has zero or very little impact. For example, a trust assumption of the form: "x% of validators are not willing to arbitrarily fork blocks". The validity of this statement is the same under PBS or not, builders have very little bearing on the LMD weight of a given chain, validators alone decide their attestation targets.

3/4/2023
Capella interop, Prysm - Nethermind

Build Nethermind git clone --recursive -b feature/shanghai-eip-4895-withdrawals https://github.com/NethermindEth/nethermind.git cd nethermind/src/Nethermind dotnet build Nethermind.sln -c release export NETHERMIND=$(pwd)/Nethermind.Runner/bin/Release/net6.0 Build prysm git clone --recursive -b capella https://github.com/prysmaticlabs/prym.git cd prysm bazel build //cmd/beacon-chain

11/10/2022
Read more from potuz
Published on HackMD