---
tags: eth2devs
description: Notes from the regular proof of stake [Eth2] implementers call
image: https://benjaminion.xyz/f/favicon-96x96.png
---

# PoS Implementers’ Call #93 - 2022-08-11
[Quick contemporaneous notes by Ben Edgington; fka "Eth2 Implementers' Call"]

Agenda: https://github.com/ethereum/pm/issues/594
Livestream: https://youtu.be/CIAGQMUKEZ4

## Merge

### Goerli Merge review

[Issue tracker](https://notes.ethereum.org/@parithosh/goerli-merge).

Goerli Merge happened approx. 12 hours ago. Currently healthy, though has been wobbly at times.

[Pari] We had two terminal blocks this time, and some of the chaos is attributable to that. Dropped from ~90% to ~70% then fell below 66%. A few nodes ended up on their own forks. See the [issue tracker](https://notes.ethereum.org/@parithosh/goerli-merge).

Some Nimbus team and Lodestar team nodes had wrong configs. Fixing those has increased participation to ~81%. If those hadn't happened, we'd have been all good.

Nethermind had an issue handling the multiple terminal blocks. There is a fix in testing that will be in the next release. There is a test for this on Hive.

Erigon stalled before self-healing. Got stuck on a side fork. After 128 slots it switched to optimistic sync and recovered. [AndrewA] Still not clear what occurred here; investigating. Btw, why 128 blocks before optimistic sync? [Mikhail] It's for protection against fork choice poisoning. It allows enough time for the honest chain to progress. [Danny] If Erigon had executed the side chains during the transition, this issue would not have occurred. It's worth the additional load to do this during the transition. [Mikhail] Once the transition block is imported the 128 slot timeout no longer applies.

A Lighthouse--Erigon RocketPool node had issues. Investigating.

An instance of Besu incorrectly replied to Teku about the canonical chain.

[Marek] Nethermind--Teku clarification. Teku requests old blocks, how to reply if you don't have them? [PaulH] Lighthouse used to track "isSyncing" on the execution client, but this proved to be unreliable. [Marius] If you don't have all the history you should not return "SYNC". [Lukasz] That's not how Nethermind currently works - in sync is only about whether it has the state need to validate the chain head. May need to change that. [Marius] Does Teku need the historic receipts for the deposit contract? [AdrianS] Teku's search for the minimum genesis block is what is causing the issue. Now we are well post-genesis, we could remove this. Nevertheless, we should find some way to distinguish between the two meanings of being "in sync". [Lukasz] What do other exec clients do? Should Nethermind change this? [AndrewA] If Erigon has anything missing it returns "syncing". [Fabio] Besu will download everything via backward sync before reporting "in sync". [Marius] Same for Geth. [Marek] Unlike other exec clients, Nethermind always syncs state faster than history.

[PaulH] Took a good look through the fork choice data before the Merge transition was finalised. There were a few forks around, but nothing to cause concern. It was able to sort itself out.

[Marius] Some nodes lost a lot of peers on the consensus layer. Is the peer scoring too aggressive? [Pari] Some Nethermind were restarted and had to re-peer. [Marius] It was more of a gradual decline - 90% drop over time. [Danny] This might account for why some of the forks were 3-4 blocks rather than shorter; due to transient network partitioning.

### Terminal block hash update

Any progress updates on implementing TBH override?

[Marius] Geth still doesn't have this. Only on the network level to not peer with nodes on other forks. Ditto Nethermind, Erigon.

[Danny] It's probably sufficient only to have a plan to implement this in case of emergency and keep code bases clean for now.

### Spec for placeholder TTD in the Engine API

See [this issue](https://github.com/ethereum/execution-apis/pull/285).

[PaulH] Helps to check people's deployments pre-Merge. Consensus clients already implement this. Should be easy for execution clients to add.

[Danny] There may be no further client releases before we decide the TTD, so this may not be needed. [PaulH] But may also be useful for other networks like GBC.

[Marius] Think that Geth has this pending. [Danny] **Action: please comment on the PR if you have a view, by tomorrow.**

### Mainnet Bellatrix epoch and TTD

Reference: [Bellatrix options](https://hackmd.io/Xq0HavcIRvufUVvA2ZjoUA), [TTD Calculations](https://notes.ethereum.org/@MarioHavel/merge-ttd).

DAG size issue (exceeding 5GB). Seems like there are workarounds to continue to use 5GB mining rigs for a while. Odds of a big drop in hash rate seem low. We have not seen it in the past. We can confirm next week, but unlikely to be a massive drop between now and then.

Does the following work?

- Tuesday 23rd for EF blog post
- Client releases with TTD and Bellatrix config ahead of that

Prysm :+1:

Erigon :-1:, would like more time to work on issues mentioned above. [Danny] We could signal that execution client releases might follow after Bellatrix for some clients. Clients released by 23rd will be "mainnet ready", but more hardened releases might follow.

[MattN] Can we prioritise getting the Hive test suite fixed up? [Mario] Things are in fairly good shape. The latest run should reflect the current status. [Mikhail] Including the "terminal block from sidefork" test? [Mario] Yes, that's fixed. [Lukasz] Nethermind is passing most of the Hive tests. Would be good if the state roots of the tests were to vary - they currently all have the same state root.

Bellatrix epoch setting. Review of Adrians [doc](https://hackmd.io/Xq0HavcIRvufUVvA2ZjoUA). [Jacek] Are any proposed epochs on a 8192 slot boundary? This would be beneficial for the historical roots accumulators in state. [PaulH] Nonetheless, we should prioritise users being available.

**Proposal: Bellatrix upgrade at epoch 144896.**

[MarioHavel] Review of [TTD options](https://notes.ethereum.org/@MarioHavel/merge-ttd). [Ansgar] Any attack on Bellatrix via hashrate acceleration would be visible well in advance, and we'd have plenty of time to react. A small interval between Bellatrix and TTD should be fine on that basis. [Danny] A buffer would help in case of any issues in Bellatrix. Uncomfortable targeting less than a week. [Marius] Prefers at least 10 days. Need to give people sufficient time to update their infrastructure. Bellatrix might be the kick they need to do so. [Jacek] Do we expect clients to have the TTD built in? [Tim] Yes.

[Micah] Do we need extra time for mainnet shadow forks after Bellatrix? [Danny] We will continue to do MSFs independently. [Pari] Planned to be weekly.

**Proposal: TTD 58750000000000000000000**

This targets ~September 15. To be confirmed on ACD next week.

## MEV-Boost

No significant updates. Testing, and working on the relay monitor.

### Clarification of builder specs

[AlexS] See [here](https://github.com/ethereum/builder-specs/pull/46), "clarify local building should happen in parallel". Seems like a no-brainer.

### Circuit breaker design

[AlexS] (Recap - see [last time's notes](https://hackmd.io/@benjaminion/BJjamMl69#Liveness-discussion-circuit-breaker-proposal) for the detail). Automatically fall back to local block production if lots of blocks are missing. **Action: Please review [the PR](https://github.com/ethereum/builder-specs/pull/47)**.

Lighthouse has implemented a rolling window, with 8 skips per epoch - can be user configured. Prysm also does rolling, with number of skips TBD. Can discuss randomising this. Good for different clients to do different things. Users being able to tune it is also good.

[Alex] Would be good to have this rolled out pre-Merge. Hopefully fairly straightforward.

## Research and Spec updates

Skip discussion of EIP-4844 - out of time.

* * *

# Chat highlights

From danny to Everyone 03:04 PM
: https://notes.ethereum.org/@parithosh/goerli-merge

From Łukasz Rozmej to Everyone 03:07 PM
: hive test is currently insufficient, but Mario is working on enhancing it
we already gave feedback. hive need not to be empty and actually change state

From Paul Hauner to Everyone 03:08 PM
: LH+Erigon bad block fix is to update Erigon, right?
https://github.com/ethereum/consensus-specs/blob/dev/sync/optimistic.md
See “Why SAFE_SLOTS_TO_IMPORT_OPTIMISTICALLY?”

From Paul Hauner to Everyone 03:17 PM
: We should never have added that logic!

From danny to Everyone 03:17 PM
: heh
years later...

From Paul Hauner to Everyone 03:21 PM
: Is a blockchain node ever actually “synced?”

From danny to Everyone 03:21 PM
: ask jacek

From Paul Hauner to Everyone 03:21 PM
: It’s a simple question that’s very difficult to answer

From danny to Everyone 03:22 PM
: e.g. a node could be doing Beam sync (remember that??) and VALID will be on head but old eth queries for blocks will never be met
this is degenerate wrt old queries and honest expectation but unrelated to engine

From Parithosh Jayanthi to Everyone 03:27 PM
: some of the nethermind was restarted
soooo they'd have to re-peer

From danny to Everyone 03:31 PM
: https://github.com/ethereum/execution-apis/pull/285

From Marek Moraczyński to Everyone 03:32 PM
: we're returning placeholder but we haven't released the version yet

From Jamie Lokier to Everyone 03:32 PM
: Fwiw, beam-like sync isn't as forgotten as might appear. Execution state for Beam is available over the snap/1 protocol (though 128 block depth is limiting compared with before), and some Portal work seems to be heading in a similar direction.

From Tim Beiko to Everyone 03:34 PM
: I can add the placeholder TTD in the spec - happy to just take what Geth has already if it works for all other clients

From Paul Hauner to Everyone 03:34 PM
: I have a PR open already thanks Tim. I’ll fetch the link.

From Tim Beiko to Everyone 03:34 PM
: ahh yes I just realized!
:-)
I can give an update on the DAG stuff if that's helpful

From Mario Havel to Everyone 03:35 PM
: https://notes.ethereum.org/@MarioHavel/merge-ttd

From Paul Hauner to Everyone 03:35 PM
: https://github.com/ethereum/execution-apis/pull/285

From Adrian Sutton to Everyone 03:36 PM
: And the client releases would be *very* shortly after we confirm as in within a day or two. So likely you'd start prepping releases pretty much now to be ready to ship them once we confirm the config.

From Paul Hauner to Everyone 03:37 PM
: Monday is the earliest we can reasonably get a release out

From danny to Everyone 03:37 PM
: right, we’d expect a blog post by ~tuesday with releases

From Zuerlein to Everyone 03:38 PM
: Placeholder TTD seems fine even if there is no hashrate drop?

From Marius to Everyone 03:38 PM
: you can also release on Saturday or Sunday if you want to

From Ansgar Dietrichs to Everyone 03:39 PM
: any reason we can't have releases by end of next week?

From Zuerlein to Everyone 03:40 PM
: That's a good idea.

From Tim Beiko to Everyone 03:41 PM
: I think we can do 10ish days

From Adrian Sutton to Everyone 03:41 PM
: So to confirm, client releases would be out by 22 August at the latest and blog post on 23 August?

From Tim Beiko to Everyone 03:42 PM
: yes, and we could have new EL releases after Bellatrix, ideally >10 days before TTD

From Zuerlein to Everyone 03:42 PM
: This suggestion of additional releases sounds good.

From Parithosh Jayanthi to Everyone 03:42 PM
: Agreed, its a bit easier to communicate with essential release and recommended release

From Tim Beiko to Everyone 03:42 PM
: If Bellatrix is between Sept 1-6, then you can imagine EL clients have their second release right around then and you have TTD around Sept 15-20ish

From Parithosh Jayanthi to Everyone 03:43 PM
: i think its unlikely that in any scenario we’d go 1mo without any client releases, there's always going to be things to release

From Zuerlein to Everyone 03:43 PM
: Mainnet validators should be very attuned to latest releases.

From Tim Beiko to Everyone 03:44 PM
: Yes, and esp. if we communicate 1 month in advance that they should pay attention :-)
"announcement of an announcement” type of thing

From Adrian Sutton to Everyone 03:45 PM
: It was very effective leading up to the beacon chain launch to say you definitely need *this* release and you should continue to monitor for further releases, potentially last minute and be ready to upgrade.

From Ansgar Dietrichs to Everyone 03:45 PM
: Bellatrix Sep 5, mainnet ttd aimed for Sep 13 seems doable to me

From Adrian Sutton to Everyone 03:45 PM
: We'll need to communicate that because we may need to tweak the TTD lst minute anyway

From Tim Beiko to Everyone 03:45 PM
: that is a good point actually
we want people to pay attention in case of a TTD override already

From Zuerlein to Everyone 03:45 PM
: Seems to fit everyone's needs?

From Paul Hauner to Everyone 03:46 PM
: We’ve always made a point of saying there *might* be more releases, but there’s no guarantee so update now and don’t wait for another.

From stokes to Everyone 03:46 PM
: we want ppl to pay attention in all cases, so regardless it should be clear in any communication to stay tuned in

From Tim Beiko to Everyone 03:46 PM
: https://hackmd.io/Xq0HavcIRvufUVvA2ZjoUA
Adrian's doc

From Marius to Everyone 03:46 PM
: geth is good to final release next week

From Tim Beiko to Everyone 03:48 PM
: 6th has the nicest number, too
145000 looks very clean

From Ansgar Dietrichs to Everyone 03:48 PM
: each day delay comes with a $25M price tag though, just as a reminder :-)

From lightclient to Everyone 03:48 PM
: 👀

From Micah Zoltu to Everyone 03:48 PM
: For TTD, not Bellatrix.

From danny to Everyone 03:50 PM
: 147456

From Ansgar Dietrichs to Everyone 03:51 PM
: on sep 5 - wouldn't bellatrix on a holiday actually be a good thing for hobbyist stakers?

From Micah Zoltu to Everyone 03:52 PM
: I prefer a technical reason over an aesthetic reason.

From Tim Beiko to Everyone 03:52 PM
: yes, same
hobbyists can also upgrade before the fork, but I'm ambivalent between 5 & 6 - we usually try and have hard forks more mid week so it;s not the first thing during the week
4:30 is better than 4
especially if you know it happens then

From Potuz to Everyone 03:55 PM
: perfect in Brazil :)

From Zuerlein to Everyone 03:55 PM
: The precedent for delaying updates on holiday grounds was set during Sepolia

From Paul Hauner to Everyone 03:55 PM
: SGTM

From Micah Zoltu to Everyone 03:55 PM
: August 22: all clients have a release with TTD baked in.
August 23: blog posts/announcements.
September 6: Bellatrix
September 7: Any last minute releases for clients ready.
September 8: Blog Post/Announcements for final releases.
September Middle: Merge

From danny to Everyone 03:55 PM
: https://notes.ethereum.org/@MarioHavel/merge-ttd

From Tim Beiko to Everyone 03:55 PM
: Can someone paste the epoch here for reference?
or if you can add the full row to your doc, Adrian, that'd be great :-)

From Ansgar Dietrichs to Everyone 03:56 PM
: I really think the 2 week delay between Bellatrix and TTD is unnecessary. 1 week is plenty.

From Marius to Everyone 03:56 PM
: no two weeks is good

From Ansgar Dietrichs to Everyone 03:57 PM
: but good as in $200M

From Marius to Everyone 03:57 PM
: yes

From arnetheduck to Everyone 03:57 PM
: +1

From Ansgar Dietrichs to Everyone 03:57 PM
: why?

From Parithosh Jayanthi to Everyone 03:57 PM
: Any reason why? Are you guessing hashrate would pick up so much that we'd hit ttd before bellatrix?

From Zuerlein to Everyone 03:57 PM
: 10 day delay?

From Adrian Sutton to Everyone 03:57 PM
: Yep just added it to the doc but we're saying epoch 144896, slot 4636672

From Marius to Everyone 03:57 PM
: we took so long, this one week for security is a small price to pay

From Tim Beiko to Everyone 03:58 PM
: I think 10 days is the min I'd be comfortable between bellatrix and TTD

From Nishant to Everyone 03:58 PM
: a longer period for users to update and configure properly their setups will always be better.

From danny to Everyone 03:58 PM
: I think 10 is good. actually will be a week to two in practice

From Ansgar Dietrichs to Everyone 03:58 PM
: even if hashrate were to pick up dramatically, we would see that way before bellatrix, so that would still give us enough time to have new releases with a later TTD out

From Tim Beiko to Everyone 03:58 PM
: we can fix "anything” in ~10 days

From Mikhail Kalinin to Everyone 03:58 PM
: fwiw, 144896 is the first epoch of the new Eth1Data voting period

From Zuerlein to Everyone 03:58 PM
: 10 day target would look at ~16th TTD

From Marius to Everyone 03:59 PM
: i like 16th ttd

From danny to Everyone 03:59 PM
: +1

From Ansgar Dietrichs to Everyone 03:59 PM
: but we don't need to be able to react in that time. if the hashrate goes up, that would happen way before Bellatrix
so even 10 days seem unnecessary

From Tim Beiko to Everyone 03:59 PM
: if ELs put out recommended releases, around bellatrix, the 10 days is useful though

From Ansgar Dietrichs to Everyone 04:00 PM
: but why can't they do that a few days before bellatrix?
say have recommended releases by friday before bellatrix
then 10 days from then ttd
which would be sep 13
also Tuesday as a ttd target seems useful anyway, because it's more likely hashrate will drop and the date will slip towards a bit later.

From Micah Zoltu to Everyone 04:01 PM
: Only a 50% increase in hashrate required to brick the network makes me pretty uncomfortable.
(from the chart)

From Tim Beiko to Everyone 04:01 PM
: right, but that's also above ATH hashrate

From Adrian Sutton to Everyone 04:01 PM
: Over a long period when we can easily adjust to avoid it though.
That's an expensive attack that you know we can mitigate before it has any effect.

From Ansgar Dietrichs to Everyone 04:02 PM
: @Micah it won't brick the network though, just requires new EL releases once we see the hashrate go up

From Tim Beiko to Everyone 04:02 PM
: I'm pretty comfortable with anything that requires >ATH hashrate

From Micah Zoltu to Everyone 04:02 PM
: I think with September 6th we just need ATH, not > ATH? (hard to tell from the chart).

From danny to Everyone 04:03 PM
: need ATH starting immediately at picking of TTD
not just reaching it again
which is observable, at lesat

From Tim Beiko to Everyone 04:03 PM
: will be in 30m :-)

From Parithosh Jayanthi to Everyone 04:03 PM
: if we notice ATH till the blogpost goes out, we can always re-evaluate

From Ansgar Dietrichs to Everyone 04:04 PM
: agree with Pari

From Tim Beiko to Everyone 04:04 PM
: I think I like 585
oops, misread, maybe more 586 or 587

From Ansgar Dietrichs to Everyone 04:05 PM
: I would prefer I actually do like 585

From Łukasz Rozmej to Everyone 04:07 PM
: so for security purposes its best to reveal TTD as late as possible (next week ACD or with releases)

From Tim Beiko to Everyone 04:08 PM
: Not really, though, because if you know our rough range, you'd want to start mining (or accumulating hashrate ASAP)

From Parithosh Jayanthi to Everyone 04:09 PM
: +1 for 10 days, I think its a good compromise between giving time for client releases & security/issuance budget

From Ansgar Dietrichs to Everyone 04:09 PM
: if they set it up at the last possible moment though - won't that be true no matter when we set ttd?

From Parithosh Jayanthi to Everyone 04:09 PM
: Assuming you forgot to setup a node by Bellatrix, 10 days is still enough to sync almost any EL

From Łukasz Rozmej to Everyone 04:09 PM
: +1 for 2 weeks

From Zuerlein to Everyone 04:11 PM
: +1 for the middle

From Micah Zoltu to Everyone 04:11 PM
: I'm a fan of keeping consensus client forks and execution client forks to be separated by a good amount of time (generally speaking, not just for The Merge). Changing both at the same time (or near each other) introduces unnecessary risk.

From danny to Everyone 04:11 PM
: i am for the compromise

From Ansgar Dietrichs to Everyone 04:11 PM
: sep 15 (the option from the doc) would be 9 days after bellatrix - seems like a good compromise

From Tim Beiko to Everyone 04:11 PM
: I'm happy with 58750000000000000000000
the Sep 15 value suggested by Mario

From terence to Everyone 04:12 PM
: +1 as many shadow forks as we can please

From Ansgar Dietrichs to Everyone 04:12 PM
: +1 to 58750000000000000000000

From Łukasz Rozmej to Everyone 04:12 PM
: +1 for shadow forks

From Marius to Everyone 04:13 PM
: 5875 is september 15?

From Tim Beiko to Everyone 04:13 PM
: yes

From Marius to Everyone 04:13 PM
: I'd be okay with that
I want to run the numbers myself on monday though

From Tim Beiko to Everyone 04:14 PM
: Yes, we can reconfirm this on ACD
but roughly align on this value, and ideally have draft PRs by then

From Micah Zoltu to Everyone 04:15 PM
: I would vote for 19th, since it is very likely TTD will be late, that puts it during the week rather than weeke
*weekend

From Marius to Everyone 04:15 PM
: LFG

From Parithosh Jayanthi to Everyone 04:15 PM
: Let's do thisss

From Marius to Everyone 04:15 PM
: Hype

From Tim Beiko to Everyone 04:15 PM
: the merge

From Phil Ngo to Everyone 04:15 PM
: 🚀

From Gajinder to Everyone 04:16 PM
: 👏

From stokes to Everyone 04:17 PM
: https://github.com/ethereum/builder-specs/pull/46

From terence to Everyone 04:20 PM
: https://github.com/ethereum/builder-specs/pull/47

From carlbeek to Everyone 04:25 PM
: has there been any discussion on using hysteresis on this?

From stokes to Everyone 04:26 PM
: more complex