---
tags: eth2devs
description: Notes from the regular proof of stake [Eth2] implementers call
image: https://benjaminion.xyz/f/favicon-96x96.png
---

# PoS Implementers’ Call #91 - 2022-07-14
[Quick contemporaneous notes by Ben Edgington; fka "Eth2 Implementers' Call"]

Agenda: https://github.com/ethereum/pm/issues/566
Livestream: https://youtu.be/yBEPzzeo1a4

## Merge

### Mainnet Shadow Fork 9

MSF9 happened earlier today - not sure why TTD estimation was off. Network is finalising, but barely.

- Lighthouse nodes were falling out of sync and not catching up until the next epoch. A fix was already in progress and is now deployed.
- Besu: 4 of 5 nodes hit an invalid block - looks like previous issue.
- Nethermind is offline (deliberately). On MSF8 they ran into an issue they had seen on Ropsten. Therefore on MSF9 they ran lots of varied configurations to try to reproduce. Succeeded in doing so (see below).
- Erigon nodes did not sync in time. Will use a snapshot in future tests.
- Nimbus nodes: Pari had messed up the configuration.

Lessons learned [Pari]: Sync Erigon differently in future; use better client config management.

[Marek] Nethermind has a rare issue that was thought fixed. But after 12 days, saw it again on MSF8. Reducing the pruning cache seems to trigger the issue faster. May have a fix but still investigating. Not specifically Merge related, but triggered more easily by shadow forks. [Lukasz] Related to having quite a random distribution of payloads at different heights - out of order payloads seems to trigger it more easily.

[Pari] The network was not finalising for about 60 epochs. Recovery from non-finalisation once the Nimbus config was fixed seemed to be smooth.

Since we are looking to merge Goerli soon, we should do a Goerli shadow fork next. It's much quicker to sync than mainnet. Target Wednesday next week.

Need to work out why the TTD estimation was off.

MSF8 - Nethermind would like to keep it around for a couple of days before tearing it down.

### Prater vs Goerli naming

[Background](https://github.com/ethereum/pm/issues/566#issuecomment-1184148151). Consensus layer clients could introduce `goerli` as an alias for `prater`. Could deprecate or not in future. No particular reason we need to coordinate on this - just a client UX issue. Need to be clear that it's just a name change, no need to resync.

### 1 vs 2 phase merge deployment

Want to avoid TTD being hit before Bellatrix is live on the beacon chain.

Three ways to approach this, see [discussion document](https://notes.ethereum.org/@timbeiko/merge-release-strategy) for the details:

1. Single release with both Bellatrix and the TTD baked in
- May be some infinitesimal risk that a hash rate increase hits TTD too early: this would require equivalent hash power to a 51% attack and would be very easy to detect.
- Offers simplest user experience.
2. Dual releases: first for Bellatrix, second to specify the TTD
- Super safe, but more work for users.
- Potentially extends time scales for the Merge.
3. Single release plus TTD override (as we have done for some testnets)
- Again safe wrt to TTD, but complex for users.

General view seems to be that (a) simplifying user experience is very important, and (b) that the risk of hitting the TTD too early does not change our security model under PoW (that an attacker has <50% of hash power). So the first option looks good, a one-phase approach.

We will trial this approach with the Goerli Merge.

[Tim] For mainnet, would be good if, within 24 hours of choosing the TTD and the Bellatrix height, clients could make releases.

Geth is making changes to start the EngineAPI early so that consensus clients on Bellatrix will not complain about it being unavailable. [Paul H] It would be good if all execution clients could do the same to allow users to set up their dev environments asap. To be raised on ACD as well.

### Goerli TTD

See [here](https://ttd.goerli.net/) for timing predictions. Goerli is very predictable.

Nethermind would be happy to plan the Goerli TTD now. 4 weeks from now would be reasonable; 3 weeks is tight.

[Adrian S] Goerli and Prater are the main testnets for community testing - we need to allow users enough time from Goerli Merge to mainnet Merge to check their infrastructure. Would prefer an earlier Goerli to allow more time for stakers/community testing, even if clients are not 100% stable.

Get some input on tomorrow's Merge community call. Suspect people will ask for more testing time.

[Tim] Suggested time-line for discussion:

- Goerli/Prater client releases 27th or 28th of July.
- Announce 28th/29th.
- Prater Bellatrix on the 8th of August
- Goerli Merge on the 11th.
- ACD 18th August plan mainnet Merge:
- Bellatrix early september;
- Merge two weeks later (week of Sept 19th).

Could be more aggressive in places, e.g. could have off-schedule ACD to set mainnet TTD earlier. All would be subject to Goerli Merge not blowing up!

**Action**: Tim to make TTD proposal for Goerli on Monday. Will create a PR for comments/objections and merge it if no push-back.

## MEV-Boost

### quick update from @metachris

Prysm has finished the Builder Specs implementation, so Teku and Prysm are now ready. Lighthouse almost there.

MEV-Boost had a security audit. Some minor issues, and [one medium issue](https://github.com/flashbots/mev-boost/issues/161) that relay operators need to pay attention to.

Rolling out updates to relays on testnets, and appreciate feedback from devs (e.g. latency measurements). Request from Nimbus to open up access for all validators on one testnet: plan to do this on Sepolia.

Working on proposer [config file format](https://github.com/flashbots/mev-boost/issues/154). There is a push to get this finalised.

Appreciate all the support from the EF security team!

The audit covered mostly the MEV-Boost implementation, not so much the specification.

[Pari] Would it be possible to have a relayer for one of the shadow forks? Yep - this is possible. MSF10 would be in about 2 weeks.

### merge transition delay on builder network

Does everyone think this is a good way to implement the initial MEV-Boost embargo?

[Enrico] The [current spec](https://github.com/ethereum/builder-specs/pull/38) introduces complexity for clients as we don't currently have a historic record of when checkpoints were _first_ finalised. Need to know when we are at +16 epochs from the first finalised Merge epoch. It's an issue if the node is restarted soon after the Merge as the info will no longer be in the database.

[Danny] Probably fine to use some heuristics to work around the issue.

[Paul H] Do we need +16 epochs? Once we're finalised, everything is likely implicitly ok. This is super easy to implement.

The goal of the MEV-Boost delay is to simplify the engineering over the merge, so we should do the simplest thing here. Maybe just rely on relayers not coming online for some epochs post-finalisation.

Starting MEV-Boost at finalisation seems to be the most straight-forward approach. Relays can also be asked to wait longer, but we don't need to rely on that.

Fin.

* * *

# Chat highlights

From Łukasz Rozmej to Everyone 03:05 PM
: its a bit different issue than
same symptoms, different

From Tim Beiko to Everyone 03:16 PM
: How easy is it to estimate a TTD on Goerli?
Do we know the % of blocks with diff 1 vs. 2?

From terence(prysmaticlabs) to Everyone 03:17 PM
: Börli

From stokes to Everyone 03:17 PM
: boerli nee prater

From Parithosh Jayanthi to Everyone 03:18 PM
: goerli has an average ttd of 1.375
its been super predictable till now

From danny to Everyone 03:18 PM
: or just never remove the prater flag

From Parithosh Jayanthi to Everyone 03:19 PM
: Unsure if Afri is on the call, but he's got this up and running for TTD estimation on Goerli: https://ttd.goerli.net/

From Tim Beiko to Everyone 03:20 PM
: Is this a “shadow deprecation” 😛?

From Paul Hauner to Everyone 03:21 PM
: I wrote some dank compile-time macros to generate those flags. An alias will be a challenge 😅

From Tim Beiko to Everyone 03:21 PM
: https://notes.ethereum.org/@timbeiko/merge-release-strategy

From Preston Van Loon to Everyone 03:22 PM
: Sounds like Tim borrowed Vitalik's mic

From Potuz to Everyone 03:22 PM
: lol

From Micah Zoltu to Everyone 03:22 PM
: MicDAO

From Adrian Sutton to Everyone 03:22 PM
: The take over is complete...

From danny to Everyone 03:22 PM
: I sent him my mic from last week

From Micah Zoltu to Everyone 03:23 PM
: Pod Cast Pass?
(I regret my extremely obscure pun already)

From lightclient to Everyone 03:24 PM
: how would things like dappnode deal with ttd override?

From danny to Everyone 03:25 PM
: ask for a new release

From Mikhail Kalinin to Everyone 03:30 PM
: Strictly speaking, we may be in the position where 80% hashrate increase would be enough to make TTD hit before Bellatrix. But it is still unrealistic that we will miss this increase and not react accordingly

From Tim Beiko to Everyone 03:31 PM
: I think a reasonable deployment for main net would be 10 days before bellatrix and 10 days before TTD after that

From Mikhail Kalinin to Everyone 03:32 PM
: When it becomes publicly known?
Like 12-14 days before bellatrix?

From Tim Beiko to Everyone 03:32 PM
: Sorry, I meant:
1. Announce Bellatrix
2. Wait 10 days
3. Bellatrix hits
4. Wait 10 days
5. TTD hit

From Mikhail Kalinin to Everyone 03:34 PM
: But before announcement we will be discussing TTD publicly, right? So, it's known a bit in advance of announcement

From Micah Zoltu to Everyone 03:35 PM
: An attacker wouldn't want to start ramping up hashpower until the official announcement.

From Tim Beiko to Everyone 03:35 PM
: They want to start as soon as a value is chosen
so if, say, the value is in a client PR before blog.ethereum.org, they want to start at the former point

From Micah Zoltu to Everyone 03:36 PM
: I would prefer more than 1x TTD_pick:bellatrix vs bellatrix:TTD.

From Zuerlein to Everyone 03:37 PM
: Goerli TTD?

From Parithosh Jayanthi to Everyone 03:37 PM
: https://ttd.goerli.net/

From Mikhail Kalinin to Everyone 03:39 PM
: So, if there are 2+20 days to TTD hit from when it becomes known. Then we tolerate 83% of hashrate increase right from the beginning which is quite good

From MariusVanDerWijden to Everyone 03:40 PM
: Goerli 4 weeks from now?

From Mikhail Kalinin to Everyone 03:40 PM
: Assuming Bellatrix is 2+10 days after

From danny to Everyone 03:41 PM
: I’d prefer 8/3 if we are looking for a compromise

From Potuz to Everyone 03:42 PM
: I agree with Adrian fwiw...

From Gajinder to Everyone 03:42 PM
: should we have one goerli shadow fork before this

From Parithosh Jayanthi to Everyone 03:42 PM
: gsf next week wednesday

From Gajinder to Everyone 03:42 PM
: awesome

From Phil Ngo to Everyone 03:42 PM
: If we don't have consensus today on this, we have the opportunity to hear from the community itself tomorrow

From Tim Beiko to Everyone 03:43 PM
: That’s a good point, actually

From Zuerlein to Everyone 03:43 PM
: So what does this schedule look like as proposed?
Definitely shouldn't be announce and fork a 4 day period...

From danny to Everyone 03:46 PM
: 9 days would be better. a monday for a wednesday fork

From MariusVanDerWijden to Everyone 03:51 PM
: ITSSS GOINNGG DOWNNN LFGGG :rocket:

From Parithosh Jayanthi to Everyone 03:51 PM
: week of 12th -> ethBerlin live merge viewing party 😛
be there or be square

From lightclient to Everyone 03:52 PM
: 👀

From Chris Hager to Everyone 03:55 PM
: https://github.com/flashbots/mev-boost/issues/161

From Chris Hager to Everyone 03:59 PM
: https://github.com/flashbots/mev-boost/issues/154
https://github.com/flashbots/go-boost-utils

From Micah Zoltu to Everyone 04:03 PM
: What is the issue this discussion occurred on?

From stokes to Everyone 04:04 PM
: https://github.com/ethereum/builder-specs/pull/38

From Potuz to Everyone 04:05 PM
: we don't track, but we keep the payloads, so if they are non-null they are execution

From terence(prysmaticlabs) to Everyone 04:07 PM
: ^ yeah, it’s a fast look up for us

From Chris Hager to Everyone 04:16 PM
: Flashbots is committed to not delivering any blocks from our relay until an agreed time / trigger