Consensus Implementers’ Call #97 - 2022-11-03

[Quick contemporaneous notes by Ben Edgington; fka "Eth2 Implementers' Call"]

Agenda: https://github.com/ethereum/pm/issues/630
Livestream: https://youtu.be/GWkhFCvwOT4

Introductory

Tim Beiko in the chair this week. Lots on the agenda!

Spec, research, etc.

Withdrawals

A couple of recent discussions: moving withdrawals into a single pipeline (PR 3042), and then queue-less withdrawals (PR 3068).

The single pipeline has the side-effect of not prioritising full withdrawals over (automatic) partial withdrawals. If we are OK with that, then queue-less simplifies the process considerably. Some concern about having to loop through the whole validator set if there is an inactivity leak of over 5 days or so, but generally seems acceptable and client teams have mostly endorsed the queue-less approach.

Note that the withdrawal process in either case is fully deterministic.

Stakers do not request to withdraw, they request to exit, and then become eligible for automatic withdrawal (of the whole stake) during the normal withdrawal sweep.

[Micah] Does this change preclude implementing suspension of staking/putting validators to sleep? [Potuz] It doesn't make it harder or easier. Validator suspension remains a research project.

[Alex] Should we bound how much of the validator set is scanned for withdrawals each slot? [Potuz] Bounding is ok, but we should make the bound large so that holes in the validator set (exited validators) do not become an issue. E.g.

$2^{17}$.

[Lion] has an approach that allows full withdrawals to be processed faster independently of partial withdrawals. This would entail a full scan of the validator set for every block (or maybe not). It adds four bytes to the state.

[Alex] Strong preference for simplicity. [Lion] This could be added later once withdrawal logic a la 3068 is in place if we see issues with that.

Decision: continue to polish #3068 as the candidate approach.

[Alex] Just need to make sure that there is no undue extra load on clients due to reorgs. [Potuz] Running the loop when we update head in forkchoice does not add extra load.

[Jacek] How much does effective balance change? That is more expensive to handle. [Potuz] On exited validators, the EB becomes zero, on partial withdrawals the EB does not change - remains at MAX_EFFECTIVE_BALANCE.

[Potuz] Minimum skim amount is more interesting. Very small withdrawals would become disproportionately expensive to construct proofs for. [Alex] These can be batched on the execution side to amortise the cost.

Extensions to EngineAPI

…for withdrawals.

[Gakinder] It's working fine with Geth.

Decision: this is good to go.

Adding block value to EngineAPI

So that local blocks can be compared with those from a builder: PR.

[Terence] Good baseline for defence against censorship.

[Lukasz] Need to decide which number to return. "Balance after block" is the easiest. [Alex] But the balance diff is probably more useful for comparison. [Mikhail] Either is fine; anything else just adds complexity without gain. [Gajinder] Another vote for the balance difference.

[Chris - Flashbots] The diff, or the proposer payment is the most useful value. Further discussion is here. Still some discussion ongoing about how blocks are scored. [Lukasz] That's a slightly different topic to our current discussion about the EngineAPI. [Chris] It is connected to this as you need to compare your local number with the block score from MEV-Boost.

[Marius] Doing the diff increases the likelihood I include transactions that benefit myself. Conversely, my own outgoing transactions reduce my overall score (irrespective of how valuable the block actually is).

[Further detailed discussion ensues… see the recording if you want the details. Basically an EL takeover of the CL call

]

Geth and Prysm have protoypes. Geth returns just the normal coinbase fees total; Prysm compares that number with the MEV-Boost blocks. Action: Conversation to the PR. Timeline: ASAP!. Aim to approve to merge by next CL call.

[Sean] Since this number is coming from a trusted execution client, it doesn't really matter how it's computed, as long as the consensus client understands this. But there is some value in standardisation.

Rebase of 4844 on top of Capella

PR.

[Terence] Removing the withdrawals queue from the beacon state makes this much easier. Happy to approve.

[Sean] Lighthouse has been pushing for this, but have made withdrawals separate, so are now happy to support whatever the consensus is. Can stub out withdrawals to make progress on 4844. Are teams ok with this? [Terence] Prysm has no strong opinion. [Sean] Ditto.

Decision: General agreement to work on withdrawals first and build 4844 on top of that.

Historical batches

[Jacek] PR to clean up one of the data structures in the state. Get root for the block root accumulator separately from the root for the state root accumulator. This allows for efficient validation of historic blocks. PR came late in the Bellatrix development, so we postponed it. Now would be a good time to include it.

No objections or concerns. Action: Please comment on the PR.

Goerli ETH supply, WITHDRAWAL_BOOST_FACTOR

PR.

Goerli ETH is very hard to come by. Goerli community call: client teams don't want to put too much effort specifically into testnets; application teams need somewhere to test. Deprecating Goerli is not really feasible right now, would be better to provide more Goerli ETH.

Withdrawals give us an opportunity to inflate the supply. A multiplier on withdrawal amounts.

[Tim] This seems much easier to implement on the consensus side rather than the execution side. Need to take care of overflows in total supply. An individual could amplify their own Goerli ETH by repeatedly staking and withdrawing.

[PaulH] Client performance suffers for every new validator registered. If people repeatedly enter and exit it could dramatically increase that size. [Jacek] Double the current size is probably ok.

[Jacek] Testnets have two purposes: staker testing (needs 32 ETH per stake), and application testing (needs less on the whole). Can we separate these? [Potuz] EthStaker has a proxy contract that deposits on behalf of stakers.

In general, fixing Goerli will just discourage people from moving to Sepolia. But there is less infrastructure support for Sepolia in the short term. Might take 6-12 months for Sepolia to become as useful.

[Pari] Could reduce max effective balance and ejection balance for Goerli to free up a lot of staked ETH.

[Oleg] We need to have long-term networks for some application testing. [Marius] Ephemeral testnets are good for stakers, but not so much for applications.

Sepolia will not suffer these supply issues by design.

General view seems to be against modifying Goerli, and push people to Sepolia.

Action: Discussion to #Testnets channel on EF Discord.

MEV-Boost updates

See here.

AOB

[Alex] See the Withdrawals Progress doc for latest status.

---

Chat highlights

From Tim Beiko to Everyone 02:02 PM

https://github.com/ethereum/pm/issues/630

From Micah Zoltu to Everyone 02:03 PM

Danny has been "out" for a long time. I'm going to start assuming he is dead and Tim is just covering it up for some reason.

From Potuz to Everyone 02:11 PM

sweep circularly all validators

From stokes to Everyone 02:11 PM

There is no explicit queue anymore

From Mikhail Kalinin to Everyone 02:11 PM

But logically there is still the queue

From Potuz to Everyone 02:12 PM

validator index

From Paul Hauner to Everyone 02:12 PM

The current approach is more functional in that in just stores enough information in the state to generate the list of withdrawal objects each block, rather than generating the list each epoch and storing it in the state.

From terence to Everyone 02:13 PM

The queue is not part of the consensus but the order is. Sorting is completed “outside” of the consensus

From stokes to Everyone 02:15 PM

This is kind of the bucket of “we have bigger problems” if we are leaking for 5+ days

From terence to Everyone 02:17 PM

Clients are free to optimize and cache under the worst-case scenario as well

From Ansgar Dietrichs to Everyone 02:17 PM

Stupid question: withdraws for exits from before the fork would automatically start immediately after the fork, right?

From stokes to Everyone 02:18 PM

You start processing at cappella, yes

From Micah Zoltu to Everyone 02:20 PM

I strongly prefer simplicity.

From stokes to Everyone 02:20 PM

(same)

From Micah Zoltu to Everyone 02:20 PM

Even just a little simpler.

From Potuz to Everyone 02:22 PM

Also testing becomes much harder with different logics

From Ansgar Dietrichs to Everyone 02:22 PM

Does this mean that during mass slashing, it can take longer until the slashed validators are exited?

From stokes to Everyone 02:23 PM

no
The exiting is happening separately

From Ansgar Dietrichs to Everyone 02:24 PM

Ah okay, makes sense

From Paul Hauner to Everyone 02:30 PM

Jacek, Potuz’s PR makes life easier for hash tree root I believe. No more list of withdrawal objects in the state.

From stokes to Everyone 02:30 PM

And effective balance updates are still batched at epoch bddy
(And they would happen in any case)

From Potuz to Everyone 02:32 PM

I'll start fixing the tests now on my PR, unless Hsiao-Wei wants to take care of that!
:)

From stokes to Everyone 02:32 PM

Lol you should make the first pass
I can help/review

From Hsiao-Wei Wang to Everyone 02:33 PM

@potuz I'm free today to work on the tests 🔥

From Mikhail Kalinin to Everyone 02:33 PM

@potuz assuming you keep all balance in memory, is this right?

From stokes to Everyone 02:33 PM

Ah ok, I can also help w/ tests so feel free to ping if you need help

From Potuz to Everyone 02:33 PM

you guys are the best

From Micah Zoltu to Everyone 02:33 PM

You can't DoS us if we are already DoSsing ourselves!

From Tim Beiko to Everyone 02:33 PM

https://github.com/ethereum/execution-apis/pull/195

From lightclient to Everyone 02:34 PM

lgtmt

From Mikhail Kalinin to Everyone 02:34 PM

@stokes I will take another look tomorrow

From Tim Beiko to Everyone 02:34 PM

https://github.com/ethereum/execution-apis/issues/307

From Mikhail Kalinin to Everyone 02:35 PM

you can merge the PR — I will take a look anyway

From stokes to Everyone 02:35 PM

Haha ok, we can race ;)

From Lion dapplion to Everyone 02:36 PM

As of now, validators earn an avg of 0.00317 ETH per day. With current parameters partial withdrawals will loop all validators (if everyone had 0x01 credentials) every 3 days. So that's a target withdrawal amount of 0.00951 ETH
Has someone calculated the share of validators with 0x01 withdrawal credentials currently?

From stokes to Everyone 02:37 PM

Its very very small atm
Mostly staking pools

From Chris Hager to Everyone 02:37 PM

Flashbots would prefer only the fees, as it more accurately reflects the "additional value provided" (i.e. excludes mempool transactions to the proposer feeRecipient)

From terence to Everyone 02:37 PM

+1 diff

From Chris Hager to Everyone 02:37 PM

lots of discussion here: https://collective.flashbots.net/t/block-scoring-for-mev-boost-relays/202

From Enrico Del Fante to Everyone 02:37 PM

Diff, cos otherwise we need to change builder apis too

From stokes to Everyone 02:37 PM

As simple as possible as long as it is useful for comparing against remote bids

From Micah Zoltu to Everyone 02:38 PM

Wouldn't this code go in MEV Boost?

From stokes to Everyone 02:38 PM

The EL needs to compute it

From Potuz to Everyone 02:38 PM

Second this

From stokes to Everyone 02:38 PM

(locally)

From Micah Zoltu to Everyone 02:38 PM

The EL needs to compute a block, but I thought that block went to MEV Boost for processing (like all other blocks proposed by builders)?

From stokes to Everyone 02:38 PM

Yeah, I think that is a bit upstream of just this feature

From Łukasz Rozmej to Everyone 02:39 PM

It should be Ethereum dictating how builders (edxtensions) work than the other way around :)
but diff is fine

From stokes to Everyone 02:39 PM

The beacon node could compare (1) local vs (1) remote payload
Or we hook the EL back around into mev-boost and then it compares the local payload as “just another bid"
I would prefer to keep the communication across nodes simpler
So prefer the first message in this block

From Chris Hager to Everyone 02:40 PM

https://collective.flashbots.net/t/block-scoring-for-mev-boost-relays/202

From Micah Zoltu to Everyone 02:40 PM

Is there another place that same discussion is happening? I remember participating, but not in that thread…

From stokes to Everyone 02:40 PM

There are 1 or two GitHub issues around payment proofs

From Chris Hager to Everyone 02:41 PM

https://github.com/ethereum/builder-specs/pull/51
yeah, for EL block value it'd be fine, and it's tangentially related. although would be nice to use the same scoring

From Mikhail Kalinin to Everyone 02:44 PM

diff handles pretty much all that we need, including COINBASE payments, priority fees and transactions to the feeRecipient address. It doesn’t account for outgoing value transfers sent from feeRecipient address — this is the only con that I can see and it seems like negligible one

From Łukasz Rozmej to Everyone 02:44 PM

we could always do both diff and fees, but its probably overkill

From Tim Beiko to Everyone 02:44 PM

is that much more computation work?

From Łukasz Rozmej to Everyone 02:44 PM

not really

From Chris Hager to Everyone 02:45 PM

maybe the relays should do balance diff!

From Micah Zoltu to Everyone 02:45 PM

That is my vote.

From Chris Hager to Everyone 02:45 PM

the discussion includes good points on both sides

From stokes to Everyone 02:46 PM

The intent is that they are the same
If they aren’t it is a “bad” bid

From lightclient to Everyone 02:46 PM

right

From Chris Hager to Everyone 02:46 PM

please take a look at the convo at https://collective.flashbots.net/t/block-scoring-for-mev-boost-relays/202 – it brings up good points on both sides.
one important argument would be if one kind of proofs would be easier to validate inside mev-boost (without EL access)

From lightclient to Everyone 02:48 PM

if i’m building locally i don’t see why you can’t just calculate the sum of all fees in the block, regardless of your txs sent

From stokes to Everyone 02:48 PM

Yes, this is the “balance diff"

From Mikhail Kalinin to Everyone 02:48 PM

really?

From lightclient to Everyone 02:48 PM

not a balance diff though

From stokes to Everyone 02:48 PM

The concern is I also send a txn that decreases my balance

From MariusVanDerWijden to Everyone 02:48 PM

No this is the fee one

From Tim Beiko to Everyone 02:49 PM

is there a way to do balance diff but ignore transactions sent by the fee recipient?

From Micah Zoltu to Everyone 02:49 PM

Yes, but not easily.

From stokes to Everyone 02:49 PM

Thats the more precise way but it is more involved

From Micah Zoltu to Everyone 02:49 PM

And code complexity is bad.

From stokes to Everyone 02:50 PM

^

From lightclient to Everyone 02:50 PM

why do we need to think about balance diff vs. sum of fees received?
for local blocks

From Micah Zoltu to Everyone 02:50 PM

So we can decide whether we should use local block vs MEV Boost block.

From MariusVanDerWijden to Everyone 02:52 PM

Also fees are easier to compute than diff

From stokes to Everyone 02:53 PM

Can I game this by just sending myself a txn?
e.g. just sending myself some eth

From Tim Beiko to Everyone 02:53 PM

What's your incentive to game this?

From stokes to Everyone 02:53 PM

Angry client devs are trying to censor remote builder network

From Tim Beiko to Everyone 02:54 PM

right, but the validator is the one who needs to send themselves ETH (to the same address)

From stokes to Everyone 02:55 PM

Next step is a PR to execution APIs
To spec out the feature
Value comparison is a separate convo

From Mikhail Kalinin to Everyone 02:56 PM

agreed with @stokes

From terence to Everyone 02:56 PM

https://github.com/ethereum/execution-apis/pull/314

From Micah Zoltu to Everyone 02:58 PM

I like sum of transaction fees for the EL "bid".

From Mikhail Kalinin to Everyone 02:59 PM

Me too, after the conversation today

From Micah Zoltu to Everyone 02:59 PM

We could have that "fudge" on either EL or CL, but should standardize on one or the other.

From lightclient to Everyone 03:00 PM

yeah just need to pass any value to the CL right now

From Paul Hauner to Everyone 03:00 PM

I would be keen to put fudge as a flag on CL

From stokes to Everyone 03:00 PM

That makes the most sense to me

From Micah Zoltu to Everyone 03:00 PM

Why CL instead of EL (curiosity, I don't care much)?

From stokes to Everyone 03:00 PM

Have beacon node builder code decide local vs remote payload
CL is going to have access to bids
Want to avoid EL talking to mev-boost at most costs

From Micah Zoltu to Everyone 03:01 PM

EL could have a flag to fudge all of the local bids by x%.

From stokes to Everyone 03:01 PM

I’d rather have a tolerance diff on the CL, rather than inflating bids at the source

From Tim Beiko to Everyone 03:01 PM

https://github.com/ethereum/consensus-specs/pull/3052

From Micah Zoltu to Everyone 03:01 PM

Then when CL compares the number they get from EL to the number they get from Boost, the EL number would be x% higher.

From Paul Hauner to Everyone 03:01 PM

CL can log saying “I chose local but because of fudge”

From stokes to Everyone 03:01 PM

Yup, and I think its better as a “primitive” to not distort the truth
Instead, make this a policy inside the beacon node

From Paul Hauner to Everyone 03:02 PM

If EL does fudge CL doesn’t know
EL can’t know if it was chosen due to fudge or not

From Gajinder to Everyone 03:02 PM

https://github.com/ethereum/builder-specs/pull/51

From stokes to Everyone 03:02 PM

You can imagine other things building on top of this engine API and not needing to have to think about a distortion

From Micah Zoltu to Everyone 03:03 PM

I think "not lying to APIs" is a good argument to favor putting fudge in CL.
I'm convinced. Thanks!

From stokes to Everyone 03:03 PM

🙂

From Micah Zoltu to Everyone 03:04 PM

Does the CL do anything to ensure that the MEV Boost block is built on top of what it thinks is the current head?

From stokes to Everyone 03:04 PM

It only looks at bids which are rooted at the head

From Micah Zoltu to Everyone 03:05 PM

Is this trusted or proven currently?

From stokes to Everyone 03:05 PM

The node has a local fork

From Micah Zoltu to Everyone 03:05 PM

If the relay claims a block is built on top of block a, could it then publish a block built on top of b instead?

From Mikhail Kalinin to Everyone 03:05 PM

There is one thing with paying transactions and COINBASE payments that has been risen in one of the threads. We can’t count them correctly if the feeRecipient is a contract forwarding value to some other address. This is a small but one more argument in favour of counting transaction tips only — they are in-protocol payments with no forwarding possible

From stokes to Everyone 03:06 PM

There are various checks in the full code path that prevents this from happening @micah

From Micah Zoltu to Everyone 03:06 PM

👍

From stokes to Everyone 03:06 PM

You could have a bad impl of mev-boost

From Micah Zoltu to Everyone 03:06 PM

I'll ask more questions in Discord. :)

From stokes to Everyone 03:06 PM

So it is something worth tracking

From Roberto B to Everyone 03:07 PM

Agree!

From MariusVanDerWijden to Everyone 03:07 PM

Prio for withdrawal > 4844 imo

From terence to Everyone 03:07 PM

Yeah def

From MariusVanDerWijden to Everyone 03:07 PM

Lets get it out and tested asap

From Potuz to Everyone 03:07 PM

yeap

From Tim Beiko to Everyone 03:07 PM

https://github.com/ethereum/consensus-specs/pull/2649

From Gajinder to Everyone 03:08 PM

lodestar is almost ready with withdrawals testing

From stokes to Everyone 03:09 PM

Btw tracking withdrawals progress here:
https://notes.ethereum.org/LqHDfmQdS96KWgs3UzGeyA
Missing teku and nimbus PRs for the feature set

From MariusVanDerWijden to Everyone 03:10 PM

I added them
and added an amphora like matrix
Only erigon missing, but "they wait until geth ships withdrawals"

From Paul Hauner to Everyone 03:11 PM

FWIW it’s been a while since I thought about this historical block roots PR, but I recall it being a good idea and something worth doing.

From stokes to Everyone 03:12 PM

@marius this is AWESOME ty

From Tim Beiko to Everyone 03:13 PM

M4: Stretch goal tbd 😎

From stokes to Everyone 03:13 PM

We will come up with something fun

From Tim Beiko to Everyone 03:14 PM

https://github.com/ethereum/consensus-specs/pull/3073

From Micah Zoltu to Everyone 03:14 PM

Can someone remind me why we care that people are paying for Goerli?

From lightclient to Everyone 03:14 PM

i mean there is, we just don’t want to xD

From stokes to Everyone 03:15 PM

@Micah it would be nice to have a “staking testnet”
But currently hard to get goETH to actually do that

From lightclient to Everyone 03:15 PM

^ mario is going to work on this

From Ansgar Dietrichs to Everyone 03:18 PM

Can’t we just massively increase validator rewards?

From stokes to Everyone 03:18 PM

That doesn’t shift the allocation of the pie tho

From Ansgar Dietrichs to Everyone 03:18 PM

Right, but it means it is easy to create new ETH and give it out

From Pari to Everyone 03:18 PM

we could still solve the allocation problem with faucets. increasing rewards is definitely something we should do as well.

From stokes to Everyone 03:18 PM

“… and give it out” — this doesn’t follow

From Tim Beiko to Everyone 03:19 PM

Are rewards "as easy" to change than something like this?

From stokes to Everyone 03:19 PM

no

From stokes to Everyone 03:20 PM

no
Could instead lower MAX_EFF_BAL on goerli

From Potuz to Everyone 03:20 PM

I agree with lightclient, Sepolia is doomed to die if we fix Goerli

From stokes to Everyone 03:20 PM

And not touch withdrawals
Then we should let sepolia die

From Tim Beiko to Everyone 03:20 PM

I don't know that Goerli will "die' if we do nothing - it will just become a "paid testnet”
I personally think that's worse

From Micah Zoltu to Everyone 03:20 PM

Why is that worse?

From stokes to Everyone 03:20 PM

But then there will be natural incentive to use sepolia

From Tim Beiko to Everyone 03:21 PM

And if it "died" because of validator ETH minting abuse, that's probably less worse

From Pari to Everyone 03:21 PM

oh! we can reduce effective balance, reduce ejection balance and that would also mean partial withdrawals will unblock a ton of eth, right?

From Tim Beiko to Everyone 03:21 PM

@Micah, I'm not a fan of the idea of GoETH ending up being valuable, becaues it means people can just pay to get the GoETH

From stokes to Everyone 03:21 PM

Ppl can always pay to get goETH tho (?)

From Micah Zoltu to Everyone 03:21 PM

Why is it a problem that people can pay to get goETH?

From Ziad Ghali to Everyone 03:21 PM

i prefer testnet

From Tim Beiko to Everyone 03:22 PM

Sure, but there's a difference between "people can always pay" and GoETH being paid by default

From stokes to Everyone 03:22 PM

sure

From lightclient to Everyone 03:22 PM

i think if the clients turn all their validators off for goerli ppl will probably move

From Micah Zoltu to Everyone 03:22 PM

I still don't understand why we want to avoid having a "paid testnet"? What problem is create and for who if Goerli is "paid"?

From Jimmy Chen to Everyone 03:22 PM

+1 to pari’s idea - any concerns with it?

From stokes to Everyone 03:23 PM

I don’t think giving existing validators more eth really does anything to fix supply issues
We are just changing liquidity of the supply, not who holds what

From Micah Zoltu to Everyone 03:24 PM

Maybe the lack of Sepolia infrastructure support + the lack of access to goETH will finally push people to run their own infrastructure!

From stokes to Everyone 03:24 PM

I would not hold your breath micah

From Afri to Everyone 03:24 PM

micah, the liability for the people who launched it

From Chris Hager to Everyone 03:24 PM

better documentation and tooling for spinning up a local PoS testnet would go a long way 🙂

From Micah Zoltu to Everyone 03:24 PM

I will accept key handover from the current Goerli owners @Afri. 😁

From Afri to Everyone 03:24 PM

alex, the increased supply who drive the price towards zero

From stokes to Everyone 03:25 PM

Assuming it enters into circulation
Step 1. I now have more liquid eth

From Jimmy Chen to Everyone 03:25 PM

Ejection balance could help coz lots of stakers just shutdown their node not long after it’s activated

From stokes to Everyone 03:25 PM

Step 2. Why do I now spend it

From Potuz to Everyone 03:27 PM

Goerli is also very useful for us as a testground of CL in general, more than just users staking.

From Paul Hauner to Everyone 03:27 PM

A super dirty method would be to add a few lines to CL says “if the_genesis validator_root is goerli, then put 1 trillion ETH into Pari’s validator”
Make it happen when Capella rolls out there.

From Afri to Everyone 03:28 PM

related https://ethereum-magicians.org/t/proposal-predictable-ethereum-testnet-lifecycle/11575

From Chris Hager to Everyone 03:29 PM

doesn't look like there's time left for the mev-boost status update, so just posting it:
put together some notable things happening in and around mev-boost: https://hackmd.io/BjlZYFCNSz2lwjv2Yjes9w?view

From stokes to Everyone 03:29 PM

thank you Chris!

From Mikhail Kalinin to Everyone 03:29 PM

We will be testing 4444 at some point in time. Long tail of chain history might also be useful for this case

From stokes to Everyone 03:33 PM

https://notes.ethereum.org/LqHDfmQdS96KWgs3UzGeyA