---
tags: eth2devs
description: Notes from the regular proof of stake [Eth2] implementers call
image: https://benjaminion.xyz/f/favicon-96x96.png
---

# Consensus Implementers’ Call #102 - 2023-02-09
[Quick contemporaneous notes by Ben Edgington; fka "Eth2 Implementers' Call"]

Agenda: https://github.com/ethereum/pm/issues/711
Livestream: https://youtube.com/live/YMu50yNUz5Y

## Capella

### devnet updates

Zhejiang fork, the Shanghai/Capella public testnet, was two days ago - great success. No issues.

[Marius] Some Geth nodes encountered a bad block while syncing. Looks like some client may not be RLP-ing blocks correctly. **Please check your code, execution clients**. Only happens on blocks with empty withdrawals. Difficult to reproduce from the Geth side alone. [Barnabas] Can set up a very small testnet without any BLS changes to reproduce.

[Danny] Do we need a devnet 7 for further BLS change testing? Sense that we are ok as-is.

### relayer/builder testing

Relays and builders have not yet been tested on any of the Shapella testnets. [Barnabas] Reached out to Flashbots, but no response yet. [Terence] There is a MEV Boost implementation ready for testing.

[Potuz] We shouldn't consider forking Sepolia before testing the builder. [Marius + Alex] Disagree. [Danny] If not tested by before Goerli we should certainly discuss timelines. [Dustin] We have only two chances to test the fork transition before mainnet - would like not to waste these opportunities.

Need people from Flashbots in the room to make any realistic determination of readiness.

[Tim] In general we have not waited on infrastructure providers. We move fairly slowly at the protocol level. There is a risk of being blocked if we end up waiting for everybody in the ecosystem to be ready.

[Potuz] We have a couple of safety mechanisms for MEV-Boost: (1) fallback to local block production for one-off errors, (2) circuit breaker - neither have these have been realistically tested yet. This is a bad situation to be in. Suggest that all clients do not support MEV-Boost at the fork, until it is proven stable. Force local block production instead. [Sean] Note that MEV-Boost is off by default.

[See chat for more comments.]

[Tim] There is time before upgrading Goerli for MEV-Boost to be updated and tested. We should not let external parties determine our timelines.

[Alex] Around the Merge there were some Kurtosis tests to test fallbacks etc.

[Danny] So, we can test fallbacks in Kurtosis and in Hive. [Alex] As long as the fallbacks work, the chain will be fine - this should be our focus. [Dustin] Concerned about the middle-ground where MEV Boost is just "flaky" - not binary works or does not work.

[Mario] Hive does not currently support MEV-Boost stuff. Scope of work is unknown. [Alex] Will work on Kurtosis and discuss how to get tests into Hive. **For further work after the call**.

### Sepolia fork date

Around 28th February seems like a sweet spot. Requires a blog post a week ahead, meaning client releases by Friday 17th or Monday 20th latest.

[Danny] Actually, Monday upgrades are fine now. Targeting Tuesdays is a hangover from PoW when fork times were uncertain. [Tim] Slight preference for Tuesdays so people have Mondays for last minute issue handling.

**No objections to Tuesday 28th**. Will do a final check among core devs before announcing in 24 hours.

[Potuz] Would be good to have a date for Goerli to save doing another release before mainnet. [\*] Wasn't much appetite for picking a Goerli fork date on this call.

## 4844

### block/blob decoupling

PR is done pending Danny's review. No big surprises.

### Simulation results

[Anton] Decoupling looks pretty good. [See data and charts](https://docs.google.com/spreadsheets/d/11ihZPkYwozbhxglQW7duQ3Hcnk8nLCkB4ZpXdAEW3OI/edit#gid=204007125). Better results with flood publishing off, rather than on (the default).

Summary: 40-50% reduction in message receipt time with decoupling; need to revisit our flood publishing default.

### Cryptography

A verification strategy for isolated blobs has been put in place. Verification is slightly slower than with the old aggregated proof method, but is not too bad. Much less impact than network latency. This is just initial results.

### Other

[Lightclient] Can we merge [this PR](https://github.com/ethereum/EIPs/pull/6385)? [Etan] There can be collisions with other types - **discuss further on the PR**.

[Sean] APIs for [signing blobs](https://github.com/ethereum/beacon-APIs/issues/300) - **looking for feedback**.

## Research, spec, etc

### Warning about merge of [eip4844 -> deneb](https://github.com/ethereum/consensus-specs/pull/3215) PR

The Deneb rename is going to affect other open PRs - Hsiao-Wei will tray to take care of rebasing them, unless authors get there first. Planning to merge this PR asap, today. Will be in the next release.

## Open Discussion/Closing Remark

### [CL EIP and editors](https://github.com/ethereum/pm/issues/711#issuecomment-1423020244)

[Tim] Execution and consensus sides have very different processes right now. Python spec on execution side is helping to align things. But EIPs remain valuable. Would be good to see increased use of EIPs on consensus side. The EIP process is (rightly) perceived as high-friction by consensus devs.

Adding some EIP editors from the consensus side could help. Increases bandwidth, and also gives some override privileges to smooth the process. Scope is "core" EIPs only.

**Call for volunteers!** Contact Tim.

### Engine API deprecation PR

[Deprecating](https://github.com/ethereum/execution-apis/pull/375) exchange transition configuration. Post-Merge, this call is useless. There are other ways to check that execution nodes are alive.

Two options:
1. Use a hard fork as a coordination point to remove the call from both sides. Simple, but needs everyone to do it for it to be clean.
2. Just hide any error messages if the call fails. Then removal does not require any coordination.

Which approach is preferred, and can we do this before Capella? We have 8-9 days before releases.

[Danny] Default should be to "no" so close to the upgrade. So postponing for now in the absence of strong opinions. **Decision: postpone until the next upgrade**

### [SSZ Transactions Breakout Room](https://github.com/ethereum/pm/issues/721)

It's happening - SSZ transactions in the execution layer. Wednesday 15th [breakout call](https://twitter.com/lightclients/status/1623382870612971520). Come and relive the serialisation wars.

* * *

# Chat highlights

pari 14:05
: https://zhejiang.ethpandaops.io/
https://zhejiang.beaconcha.in/

stokes 14:11
: Yeah mev-boost w/o builder doesn’t do much

Mikhail Kalinin 14:13
: It is important to add withdrawals_root check to this pipeline for all CL clients

Dustin Brody 14:14
: there is literally no capella mev-boost testing yet in a realistic setting

Gajinder 14:15
: Lodestar checks withdrawals root post getheader before signing as part of generating post state to sign

Barnabas Busa 14:17
: Its easy, don’t use MEV boost :D

Dustin Brody to Everyone 14:17
: yes, get FB or one of the other builder API people on the devnets and testnets

Gajinder to Everyone 14:18
: Lodestar fallback flows are tested and we have seen in in mainnet as well where sometime builder api fails

stokes 14:19
: The warning could mean “suddenly the chain halts"

terence 14:19
: We already have that warning ha

Ansgar Dietrichs 14:19
: This would set a dangerous precedent by making stakers choose between max profit and the official client releases

lightclient to Everyone 14:19
: +1

Barnabas Busa to Everyone 14:19
: We need to decouple mev boost from hardforks
Let them do development when they want

Tim Beiko to Everyone 14:19
: +1 Barnabas :-)

Barnabas Busa 14:19
: and let clients update their clients when they can

Tim Beiko 14:20
: If it's off by default, and there are already warnings in client teams' docs, it does seem like a separate concern

Dustin Brody to Everyone 14:20
: it is unfortunate FB has this leverage, but they do
empirically

Tim Beiko 14:20
: Do they?

Dustin Brody 14:20
: 70+% MEV usage. If not them, people will look for others

Tim Beiko 14:20
: If their product isn't ready, that's on them, and node operators can chose if/what to choose to capture MEV
*what to run

Gajinder to Everyone 14:21
: Use lodestar 😂

Dustin Brody to Everyone 14:21
: the danger is that their product is ready sort of just-in-time

MariusVanDerWijden to Everyone 14:21
: Do we have a local builder that goes over the mev boost interface?

Dustin Brody 14:21
: but not enough in time for CLs to test

stokes 14:21
: Im working on one right now marius

Dustin Brody 14:21
: or to have been tested in sepolia & goerli thoroughly
works for them, no one else

MariusVanDerWijden 14:21
: How long till ship @stokes?

stokes 14:22
: Later today or tomorrow
Its mostly done
And then I was gonna deploy to zhejiang

Gajinder to Everyone 14:22
: I think builders are available for sepolia as well

Dustin Brody to Everyone 14:22
: then they look fine because they are supposedly ready and they advertise that loudly

stokes to Everyone 14:22
: The thing is we also want to test the dominant client impl on mainnet — i.e. the mev-boost code
/flashbots code

MariusVanDerWijden to Everyone 14:22
: I don't think _we_ should care about this

Barnabas Busa 14:23
: +1

Potuz 14:24
: it's not only other's code: it's our own code that hasn't been tested, which is why _we_ should care too

Gajinder 14:24
: Agree just test the fallback paths

terence 14:24
: How to test the fallback paths without builders though

Potuz to Everyone 14:24
: not only fallback, even if everything is fine, the builder code is not tested on CL clients today

Barnabas Busa to Everyone 14:24
: I feel like client teams should focus on delivering shapella asap, like we discussed. Then start testing mev+shapella if they want to.

stokes to Everyone 14:24
: Yeah I agree potuz, but will have to handle rollout when we get there

Barnabas Busa to Everyone 14:24
: and if those two don’t align then they don’t.

Gajinder to Everyone 14:25
: Just keep using current builders
They should fail on capella

Barnabas Busa 14:25
: Current builders will not include withdrawals.
People will very quickly disable mev boost and start producing local blocks.

Gajinder 14:26
: Lodestar will automatically do it for them and i guess other clients will too

Dustin Brody 14:26
: that creates unnecessary config drama

zahary 14:27
: FWIW, the fallback path is actively tested in our CI simulations

pari 14:27
: Barnabas and I can work on a transient testnet. We can setup a mock-relay and then mev-boost on enough nodes. Then we can disable the mock-relay to test the fallback.

Barnabas Busa 14:27
: Shouldn’t it be the responsibility of the mev boost guys to do testing for this?

Dustin Brody 14:28
: Barnabas: agree, ideally

MariusVanDerWijden 14:28
: Hive support shouldn't be too hard

Mikhail Kalinin 14:29
: If we say that we need to test CL clients then it is probably good to have this setup on our side instead of depending on FB

Gajinder 14:29
: Fallback testing is client responsibility

Barnabas Busa 14:31
: I would vote 27th 14:00 UTC epoch 56700

Dustin Brody 14:31
: Yes, fallback testing. But happy-path testing depends on a truly working builder API

MariusVanDerWijden 14:32
: Geth: "we don't object"

Barnabas Busa to Everyone 14:32
: sepolia is pretty much closed validator set tho

Tim Beiko to Everyone 14:32
: there are non-validator node operators too :-)
and it's actually well supported now in wallets, block explorers, etc.

Tim Beiko 14:34
: Does anyone have a candidate epoch for Feb 28?
Can't find Adrian's tool

danny to Everyone 14:34
: feb 28

pari 14:34
: https://slots.symphonious.net/

Dustin Brody 14:35
: it has been kind of nice to consider alignment with natural boundaries such as sync committee epochs -- it should be intentional whether to try to trigger issues by doing this in the middle of those, or make it less risky by putting it at such boundaries

Tim Beiko 14:36
: Is there an easy way to know a sync boundary epoch?

Barnabas Busa to Everyone 14:36
: 28/02/2023 14:00 UTC 56925 epoch time: 1677592800?

Dustin Brody 14:36
: they are epoch mod 256 == 0

danny 14:39
: https://docs.google.com/spreadsheets/d/11ihZPkYwozbhxglQW7duQ3Hcnk8nLCkB4ZpXdAEW3OI/edit#gid=204007125

Tim Beiko 14:40
: epoch 56832 is a sync committee boundary - but not the best time
2/28/2023, 4:04:48 AM UTC

Potuz 14:40
: is there a reason why you want it to be in a sync boundary?
Barnabas Busa to Everyone 14:40
4 am utc is pretty off time for most people tho ?

Tim Beiko to Everyone 14:41
: Next one is 57088 - 3/1/2023, 7:23:12 AM UTC

Barnabas Busa to Everyone 14:41
: we haven’t had any issue with sync boundaries

danny to Everyone 14:41
: we care about the 8192 slot boundary for history

Potuz 14:43
: We've been forking at epoch 20 on devnets.... not that I envision a problem, but it'd be better to be consistent in the test environment

lightclient 14:49
: can we merge this change to 4844? https://github.com/ethereum/EIPs/pull/6385

sean 14:50
: https://github.com/ethereum/beacon-APIs/issues/300

danny 14:51
: https://github.com/ethereum/consensus-specs/pull/3215
from tekupaul

MariusVanDerWijden 14:52
: This is how last names where created

Trent to Everyone 14:52
: SPEC ELF

lightclient 14:58
: the twitter poll was quite clear

MariusVanDerWijden to Everyone 14:58
: We all want this

Trent 14:58
: seems pretty clear it should be done

lightclient 14:59
: https://twitter.com/lightclients/status/1623372648783953920

Tim Beiko 14:59
: https://ethereum-magicians.org/t/proposal-forking-ercs-from-eips-repository/12804/22

danny 15:01
: https://github.com/ethereum/execution-apis/pull/375

MariusVanDerWijden 15:04
: I'm pro this change, not need it necessarily in shanghai though

sean to Everyone 15:04
: I think I have to check on the team’s capacity
But personally support getting rid of it

Potuz 15:06
: I second Sean's

MariusVanDerWijden 15:06
: nah lets decide now

Potuz to Everyone 15:06
: We even disabled the call so I am 95% confident that it's fine for us

MariusVanDerWijden 15:06
: not do it

Etan (Nimbus) 15:07
: https://github.com/ethereum/pm/issues/721

lightclient 15:08
: https://twitter.com/lightclients/status/1623382870612971520