Interop Testing Call #35 – May 05, 2025

Notes

Interop Testing Call #34 – April 28, 2025

Moderators: Mario Vega & Tim Beiko
Structure: First 15 minutes for non-EOF topics, followed by EOF-related discussion.

Summary

• Mainnet Shadow Fork in progress; EF Testing prepping a new BLS-heavy test release.
• PeerDAS Devnet 6 deprecated; Devnet 7 coming this week.
• Consensus spec update PR (#4291) proposed to align EL and CL minimal specs.
• EOF Discussion: After strong debate, EOF removed from Fusaka upgrade.
• Option D explored (limited introspection), but community support was split.
• Focus shifted back to PeerDAS + scaling improvements for Fusaka.
• App dev feedback emphasized — stronger app-dev engagement required for future EVM upgrades.
• Future EOF considerations deferred to Glamsterdam cycle.

Key Documents Shared

• Besu EOF HackMD
• EOF impact on Solidity Libraries
• Option D Summary
• Ethereum Magicians Post - Banning Code Introspection
• Lido Research Group's Position on EOF

Testing & DevOps Updates

Mario Vega (Testing)

• Announced a new release: Execution Spec Tests v4.3.0.
• Main breaking change: introduction of Deposit Contract Testing.
  • These scenarios are considered impossible on mainnet but were requested for testing.
• Next steps:
  • Contact each client team for a stance on whether to fix or ignore the impossible test cases.
  • All other test cases have passed in Hive testing.
  • A new EF Testing release is planned this week, which will include additional BLS testing, but all scenarios have already been validated against all clients and no issues were found.

Parithosh Jayanthi (DevOps)

• Mainnet Shadow Fork:

  • Work is ongoing and the network should be up within the next hour or two.

• Nightly Test Update:

  • One nightly test failed with an assertion error during consolidation.
  • Investigation is ongoing; if it's a real issue (not a false positive), it will be reported.

• Additional Point by Barnabas:

  • Focused on minimal spec testing.
  • Discussion postponed to later in the call.

Additional Discussion

• Pawel: Will the EEST release include static tests?
  • Mario: Yes, all necessary static tests are ready and will be included.

Client Updates

Besu

Justin

• Update: "Failing fast" if Pectra-enabled genesis files are missing system contract addresses.
• Raised concern (via Barnabas and Pari) regarding Devnet 7 genesis file needing an update.
• Justin F. has opened a PR for the Mainnet Shadow Fork.

Barnabas

• Commented that for the Pectra Shadow Fork, they likely won't create a new genesis file—focus is on observing the fork transition.

Justin F.

• Clarified that it’s not about a new genesis specifically for Pectra; it's about ensuring Pectra is enabled somewhere.
• Barnabas: Acknowledged and agreed.

Luca (Serenita)

• Reported that the Electra attestation packing bug in Nimbus (which involved not including recent attestations) has been fixed.
• Shared updated analysis data: gist link.

Nimbus

Barnabas

• Mentioned that Nimbus made a new release last week, and people who previously updated may need to update again.

Prysm

James

• Informed that Prysm is preparing a v6.0.1 release.

PeerDAS

Parithosh Jayanthi (DevOps)

• Devnet 6 is now deprecated.
• Aiming for Devnet 7 launch within this week.
• Most of the discussion will happen tomorrow during the PEERDAS call and in local Kurtosis testing sessions are ongoing.

Consensus Specs Change

Barnabas

• Shared an overview of the new proposal:
  • Consensus Spec PR #4291

Summary (Parithosh Jayanthi)

• Currently, CL (Consensus Layer) has different values for mainnet and minimal specs, while EL (Execution Layer) does not differentiate by spec.
• Instead of introducing minimal specs to ELs, the proposal sets the same value for both.

Next Steps

• CL client teams are requested to review the PR.
• Minimal preset testing can begin soon.
• Parithosh: It's fine to wait a week or two before full adoption.
• Barnabas: If running the minimal preset with execution requests now, it will fail until updates are applied.

Extended EOF Discussion

Tim Beiko shared the overview of the current discussion point by suggesting that people are generally supporting of making a decision in the best interest of Ethereum.
Invited Ipsilon team to share the analysis - Identified key issues with current EOF impact on popular projects.

Concerns Raised

• Developer friction: Difficulty with importing libraries post-EOF.
• New vulnerabilities: Introspection risks, harder audits, and reentrancy vectors.
• Tooling gaps: Need compiler and node support for robust EOF compatibility.
• Significant bytecode changes will require re-audits even for minor source code changes.

Client Sentiments

• Geth: Supports Option D, but concerns remain about split development paths.
• Besu: Prefer Option D over no EOF, but openness to flexibility. Supports Option A as well. Details on Fusaka Position added here
• Erigon: Supports EOF broadly.
• Nethermind: Pro-EOF but understanding of broader ecosystem hesitations.

Developer Opinions

• Foundry, OpenZeppelin, and app devs raised alarms (on chat):
  • Stack-too-deep issues solvable without EOF.
  • Gas and code introspection necessary for many use cases.
  • Risk of multiple VM standards (legacy vs. EOF forks).
• Ansgar emphasized on process improvement and ecosystem stability. Also, mentioned that the decision of not including EOF may not be because of technical issues but the the active change of what the application developers are looking for.

Technical Notes

• Option D:

  • Remove EIP-7069 (CREATE2 issues).
  • Allow selective code/gas introspection (with caveats).
  • Marked as a permanent change, not a stepping stone.

• Debates:

  • If introspection is kept, contracts must be written defensively.
  • If removed, upgrading contracts in future becomes harder.

Decisions and Next Steps

• Remove EOF from Fusaka. Tim Beiko will create the PR.
• Refocus Fusaka solely on PeerDAS and scalability.
• Future potential reconsideration of EOF during Glamsterdam or another cycle.
• EOF must rebuild strong app developer support if it’s to be reconsidered.

Other remarks

• Tim acknowledged that the process for large EVM changes needs significant reform.
• Focus on developer-first prioritization going forward.
• Avoid rushing controversial changes without strong consensus.

Zoom chat summary

This captures the key points, opinions, and documents shared during the discussion on the status of EOF and its potential inclusion in the Fusaka upgrade in zoom chat.

Tim Beiko

• Asked if Besu had shared their PoV.
• Shared that the discussion should aim at making the best decision for Ethereum.
• Emphasized: even though developers choose what to build, users and ecosystem adoption ultimately decide what survives.
• Stated if decision isn't clear, recommend removing EOF from Fusaka.
• Summarized final decision: Remove EOF from Fusaka, revisit under Glamsterdam upgrade with better process.

pcaversaccio

• Shared earlier Besu feedback document: Besu EOF Position HackMD

Justin Florentine (Besu)

• Besu initially favored Option A, but is open to Option D.
• Expressed frustration at the "decaying signal to noise" on EOF discussions.
• Noted: "Twitter is not market analysis."
• Supported decoupling PeerDAS from EOF.

Ansgar Dietrichs

• Emphasized that disagreements are technical and motivated by wanting the best for Ethereum.
• Agreed that if EOF still has confusion/tension, it should be pulled.
• Supported Tim's plan but noted it felt like an "executive call".

lightclient (Matt)

• Asked whether current discussion was engagement farming.
• Emphasized if we're unsure, remove EOF and prioritize fixes like stack-too-deep.

Hadrien Croubois (OpenZeppelin)

• OpenZeppelin wants to be EOF-compatible, but needs compiler and node support.
• Highlighted testing challenges.

Danno Ferrin (Ipsilon)

• Shared critical analysis documents:
  • Impact of EOF on Solidity libraries
  • Option D Overview
• Suggested going with Option D.

Vectorized (Ben Adams)

• Noted many practical concerns:
  • 1167 bytecode can't be deployed in EOF.
  • Prefers allowing gas and code introspection.
• Warned removing introspection would hurt patterns like secure proxy detection.
• Advocated for Option D if it restores critical functionality.

fiddy (Lido Research Contributor)

• Mentioned Uniswap v4 is not the best forkability example.
• Lido group consensus: Against EOF for added complexity without clear benefit.
• Shared link: Lido Research comments

frangio

• Pointed out audits generally rely on source code correctness.
• Preferred Option C.

Felix (Geth)

• Geth team internally split but spokesperson supports Option D.
• Warned full EOF could cause more disruption.

Ayman

• Strongly opposed introspection in EOF.
• Highlighted unpredictability for formal verification.

moody

• Linked rationale behind banning introspection:
  • Magicians Post on banning code introspection
• Warned without restrictions, EOF versions will fragment into v1, v2, v3…

gakonst

• Stated clearly: Option D or no EOF.
• Emphasized gas/code introspection are necessary for many existing patterns.
• Warned: "Ban introspection = seppuku" for major dev tooling like Foundry.

Marius

• If we aren't confident, EOF should not be included.
• Noted removing EOF helps avoid potential future upgrade chaos.

Fede (Lambda)

• Not a fan of EOF.
• Believes users don’t benefit enough, and it's not a priority compared to other needs.

Ahmad Bitar (Nethermind)

• Emphasized that teams had opportunities to engage earlier in the spec development.

Som (Erigon)

• Acknowledged technical soundness, but emphasized lack of strong app developer demand.
• Supported decoupling PeerDAS and EOF.

Charles C (Vyper)

• Noted that Option D needs independent evaluation, not rushed over a weekend.

Duncan Townsend (0x)

• As an application dev, expressed uncertainty about where active input was welcomed.

Dan Cline

• Warned about dual EVM evolution: canonical vs non-canonical EVM paths if EOF bifurcates standards.

Final Decision

• EOF will be removed from the Fusaka upgrade.
• Focus: shipping PeerDAS, scaling gas limits, minimal EL changes.
• Future Plan: Reevaluate EOF independently for Glamsterdam fork under new ACD engagement process.

Zoom chat (Raw)

Note:

This is a direct copy-paste of the chat from the call. I may have missed a few messages, and there’s a possibility of some duplicates during the copying process.

Tim Beiko (Apr 28, 2025, 10:10 AM)
re: EOF, did Besu share their PoV yet or not? Sorry if I missed it. Think all other EL teams have.

pcaversaccio (Apr 28, 2025, 10:11 AM)
I just remember this one https://hackmd.io/@RoboCopsGoneMad/H1z1lky6Je

Justin Florentine (Besu) (Apr 28, 2025, 10:13 AM)
i'm still Option A, as scheduled and already decided. I could entertain the new Option D that ipsilon is building.

Justin Florentine (Besu) (Apr 28, 2025, 10:14 AM)
i would love to have a live, meta-conversation about the point that tim just made, in another forum.

Ansgar Dietrichs (Apr 28, 2025, 10:15 AM)
thanks Tim for that context / reminder, especially with divisive topics like EOF always good to remember that these are just technical disagreements among people that all passionately want the best for Ethereum

lightclient (Apr 28, 2025, 10:16 AM)
is this engagement farming?

Ipsilon document
https://notes.ethereum.org/@ipsilon/eof_solc_libs_impact

Hadrien Croubois (OpenZeppelin) (Apr 28, 2025, 10:16 AM)
Note: As OZ, we want to write code that works with EOF. We expect some of it won't, but we really want to do as compatible as possible. Unfortunatelly, we need compilers and node to tests all that.

lightclient (Apr 28, 2025, 10:18 AM)
one important thing to note is that even if the changes in source relatively light, there will almost certainly need to be new audits since much of the bytecode will change
frangio (Apr 28, 2025, 10:20 AM)
audits are generally done on source code and assume compiler correctness, or rely on tests (incl fuzzing) or verification to detect compiler issues

Vectorized (Ben) (Apr 28, 2025, 10:19 AM)
Note that 1167 is legacy bytecode and cannot be deployed in EOF.

fiddy (Apr 28, 2025, 10:19 AM)
btw the uniswap v4 example is not necessarily the best one since uniswap probably has no intention to change an immutable piece of code.
Tim Beiko (Apr 28, 2025, 10:20 AM)
Do you know how often Uniswap forks change the existing code vs. just extend it?
The original rationale for analyzing Uniswap was that it’s the #1 most forked contract IIRC

fiddy (Apr 28, 2025, 10:21 AM)
Uniswap code is forked quite often. I however doubt that Uniswap v4 can be successfully forked (from a business context).

frangio (Apr 28, 2025, 10:22 AM)
v4 is not a good example of this because hooks build extensibility into the core. forks shouldn't be necessary

Ben Adams (Apr 28, 2025, 10:22 AM)
Uniswap v4 can't be forked until 2027-06-15 based on their licence

Ansgar Dietrichs (Apr 28, 2025, 10:15 AM)
thanks Tim for that context / reminder, especially with divisive topics like EOF always good to remember that these are just technical disagreements among people that all passionately want the best for Ethereum

Justin Florentine (Besu) (Apr 28, 2025, 10:19 AM)
i agree, but this endless cycle of discussing EOF has a decaying signal to noise ratio and i'm going to start being more assertive about calling that out.

frangio (Apr 28, 2025, 10:23 AM)
is option C also on the table? personally think that's a better choice

Danno Ferrin | Ipsilon (Apr 28, 2025, 10:23 AM)
https://notes.ethereum.org/@ipsilon/eof-option-d-summary

stokes (Apr 28, 2025, 10:24 AM)
Is Option D significantly different enough from legacy (perhaps with the bundle of other EIPs suggested for ‘stack too deep’ and code limit) to justify essentially two different VM types for all time?

Tim Beiko (Apr 28, 2025, 10:24 AM)
This is the most important question about (D) IMO. Solidity’s comment on the agenda says that, for them, it is

Łukasz Rozmej (Apr 28, 2025, 10:24 AM)
I think there are 2 questions:
Should we keep EOF?
Should we switch to flavor D?

frangio (Apr 28, 2025, 10:23 AM)
is option C also on the table? personally think that's a better choice

Duncan Townsend (0x) (Apr 28, 2025, 10:26 AM)
Ahh, I missed that nuance

stokes (Apr 28, 2025, 10:24 AM)
Is Option D significantly different enough from legacy (perhaps with the bundle of other EIPs suggested for ‘stack too deep’ and code limit) to justify essentially two different VM types for all time?

Justin Florentine (Besu) (Apr 28, 2025, 10:26 AM)
i understand that, but i'm not sure i buy that. twitter is not market analysis, and the main point of "having to update dependencies" is again, not news.

stokes (Apr 28, 2025, 10:24 AM)
Is Option D significantly different enough from legacy (perhaps with the bundle of other EIPs suggested for ‘stack too deep’ and code limit) to justify essentially two different VM types for all time?

Justin Florentine (Besu) (Apr 28, 2025, 10:27 AM)
i.e. option d allows gas introspection, and so we're now having the "multiple eternal eofs" just like we had it before

fiddy (Apr 28, 2025, 10:19 AM)
btw the uniswap v4 example is not necessarily the best one since uniswap probably has no intention to change an immutable piece of code.

Vectorized (Ben) (Apr 28, 2025, 10:30 AM)
i like D. But i don’t think there should be a compiler warning lol.

Luis Pinto | Besu (Apr 28, 2025, 10:25 AM)
Also do we think code and gas introspection is a good practice?

Marius (Apr 28, 2025, 10:30 AM)
They can't be dropped ever

Felix (geth) (Apr 28, 2025, 10:30 AM)
Would there be any merit to option D-prime with gas introspection enabled and code introspection disabled?
Danno: if we just pull out 7069…

Ansgar Dietrichs (Apr 28, 2025, 10:31 AM)
can we at least please be very explicit: if we ship EOF option D:

USE GAS AND CODE INTROSPECTION AT YOUR OWN RISK

Future Ethereum changes will never outright break contracts using these completely, but they might break specific use cases. It is your job to write code robust towards that possibility.

Tim Beiko (Apr 28, 2025, 10:31 AM)
Yes, we can add an Info EIP like we did for SELFDESTRUCT before deprecating it

Vectorized (Ben) (Apr 28, 2025, 10:32 AM)
i think banning gas introspection just because some people used gas-limited calls poorly is not a good reason to remove it.

Full EOF vs Option D

• why Ban gas introspection - to change gas schedule easier

Hadrian

• tooling open zappelin be compatible with EOF

Ayman (Apr 28, 2025, 10:34 AM)
code introspection and gas introspection introduce a lot of unpredictability to contract execution, makes it unpredictable and un-analysable (or at least extremely hard to analyse), JIT or AOT approaches become too restrei and even FV done on bytecode levelbecomes unreasonable

moody (Apr 28, 2025, 10:34 AM)
Wasn't code introspection banned so that EOF contracts could be automatically upgraded? https://ethereum-magicians.org/t/eof-proposal-ban-code-introspection-of-eof-accounts/12113

If you give that up, we're going to end up with EOFv1, v2, v3… with each breaking change. Seriously damages the ability to share code.

Ben Adams (Apr 28, 2025, 10:34 AM)
Still can't introspect EOF contracts, just legacy

Vectorized (Ben) (Apr 28, 2025, 10:34 AM)
if we remove gas introspection, i’ll just abuse escape hatch more hardcorely

Giulio (Apr 28, 2025, 10:34 AM)
Can we go through with option D and re-discuss removing gas introspection in the future?

Luis Pinto | Besu (Apr 28, 2025, 10:34 AM)
EOF v2 in the plans

Fede| Lambda: Don't dislike EOF for technical reason but due to users experience. Don't think it is useful for users and there are many other that needs priority.

Marius (Apr 28, 2025, 10:35 AM)
Thats a good point, whats going to happen with legacy contracts that have eof dependencies that have legacy dependencies

Tim Beiko (Apr 28, 2025, 10:35 AM)
After Fede, I think we should hear from Reth (most against EOF), then Nethermind (most pro?), Geth (most split?), Besu and Erigon

moody (Apr 28, 2025, 10:34 AM)
Wasn't code introspection banned so that EOF contracts could be automatically upgraded? https://ethereum-magicians.org/t/eof-proposal-ban-code-introspection-of-eof-accounts/12113

If you give that up, we're going to end up with EOFv1, v2, v3… with each breaking change. Seriously damages the ability to share code.

Justin Florentine (Besu) (Apr 28, 2025, 10:36 AM)
i agree, but its definitely not possible without eof

df (Apr 28, 2025, 10:37 AM)
Fully agree with Fede's point of view. EOF is not a good use of resources. There are a lot of important things to do and it does very little

moody (Apr 28, 2025, 10:38 AM)
I also strongly agree with Fede's point–this isn't working backwards from users at all. As a long time developer on EVM, who wrote a lot of the code in the referenced examples Uni v3 and v4, shouldn't I be excited for this change if it's so good for EVM? I've done my fair share of wrestling with stack too deep and bytecode size limits

fiddy (Apr 28, 2025, 10:38 AM)
This is Fiddy from Lido's Research contributors group. Within the team there is a rough consensus with Fede from Lambda. We also do not think the added complexity brings sufficient improvements that Ethereum needs at the moment. For more info: https://github.com/ethereum/pm/issues/1499#issuecomment-2834828019

fiddy (Apr 28, 2025, 10:39 AM)
Also to note: EOF has no significant positive or negative impact to Lido operations

Vectorized (Ben) (Apr 28, 2025, 10:39 AM)
generally, imo the VM should not babyshit devs. EOF’s removal of gas and code introspection falls along these lines imo. Babysitting is best done on the library layer.

Dan Cline: having introspection will be a fine tradeoff.

Tim Beiko (Apr 28, 2025, 10:43 AM)
If we do ship Option D, and we want to focus on scaling the gas limit (which we said we did), I think this means we realistically do nothing else in Fusaka beyond trivial changes. Are teams OK with that?

fiddy (Apr 28, 2025, 10:43 AM)
BTW as I understand Fusaka already is going to have increased codesize limits. The Spurious Dragon will finally be deprecated. This is quite big. I remember spending months trying to find even 1 single byte of space for business logic, only to later find that I needed 200 more.

fiddy (Apr 28, 2025, 10:43 AM)
So, EOF or no EOF, Fusaka is delivering good things :)

SanLeo (Apr 28, 2025, 10:44 AM)
Hopefully Fusaka ships PAY as well, would be very nice to have.

Sophia Gold (Apr 28, 2025, 10:44 AM)
You can never migrate some legacy contracts and we probably won't migrate any of them given what that would take so I think this is irrelevant. It would always be two EVM versions that would all need to be compiled down to a zk friendly ISA to get proving advantages

Guillaume (Apr 28, 2025, 10:44 AM)
split between Felix and the rest of the team

Som - Erigon (Apr 28, 2025, 10:44 AM)
With the possibility of specifying EIPs or EVM bytecode target platforms in the EOF header, maybe we are looking at a future where non-introspection code is used by ZK provers, and the others are used by legacy contract users (even if we deprecate legacy EVM)?

That kinda extensibility alone should be enough to consider EOF

Vectorized (Ben) (Apr 28, 2025, 10:45 AM)
allowing read code introspection on legacy is like reading immutable storage. imo, code introspection is not an anti-pattern. it is the only way to have a proxy that can be trusted to return the correct implementation onchain without spoofing.

Vectorized (Ben) (Apr 28, 2025, 10:47 AM)
it’s to be able to know that the proxy does not return a fake implementation. so that it can be safely used in some contexts.

frangio (Apr 28, 2025, 10:47 AM)
can you share a link to this

Vectorized (Ben) (Apr 28, 2025, 10:47 AM)
it’s a different kind of function from CWIA

gakonst (Apr 28, 2025, 10:46 AM)
Hi guys just reiterating: code / gas introspection need to stay. They are useful patterns. Ship EOF without it, nobody will use it because it removes things people do today. I would be at peace if we do that. But I would rather burn a few opcodes to solve stack too deep simply for Solidity, just given Vyper has shown compelling results w/o EOF for zkvms, stack too deep and gas perf.

gakonst (Apr 28, 2025, 10:46 AM)
Option D or no EOF
Ban gas / code introspection = seppuku. We maintain foundry. I know why it matters.

gakonst (Apr 28, 2025, 10:47 AM)
Ban gas / code introspection = seppuku. We maintain foundry. I know why it matters.

gakonst (Apr 28, 2025, 10:49 AM)
That's all you need to see

Fede | Lambda (Apr 28, 2025, 10:52 AM)
Ethrex prefers no EOF

Storm Slivkoff (Apr 28, 2025, 10:52 AM)
The way this conversation is structured prioritizes the perspective of node client devs. This adds a natural bias toward EOF. But other demographics in the Ethereum ecosystem are generally against EOF

df (Apr 28, 2025, 10:52 AM)
At least 90% of the research team says no EOF (since beginning, I believe)

Danno Ferrin | Ipsilon (Apr 28, 2025, 10:52 AM)
I don’t want to do a breaking container change for at least 10 years, 5 if it’s urgent. I am OK with Option D for 5-10 years.

Charles C (vyper) (Apr 28, 2025, 10:52 AM)
option D needs its own evaluation, there are interacting consequences from removing part of the design that i don't think were fully evaluated over the weekend

Ben Adams (Apr 28, 2025, 10:52 AM)
Arguing about EOF takes away from other resources :P

Fede | Lambda

• option D is a compromise, but it is like would you rather have something or nothing.
• I don't see anybody asking this.

Fede | Lambda (Apr 28, 2025, 10:55 AM)
Which application asked for this?

Storm Slivkoff (Apr 28, 2025, 10:52 AM)
The way this conversation is structured prioritizes the perspective of node client devs. This adds a natural bias toward EOF. But other demographics in the Ethereum ecosystem are generally against EOF

Justin Florentine (Besu) (Apr 28, 2025, 10:55 AM)
I actually disagree with this. Our doors are open, and when it comes to paying off techdebt, those actors have no incentive to pay it off.

Ansgar Dietrichs (Apr 28, 2025, 10:55 AM)
agree - this doesn’t in itself mean EOF is bad. because acd is the way it is, the team never had an incentive to actively make the case to anyone else than core devs.

changing this for future EVM feature changes needs to be a priority, no matter which way we go for EOF today.

lightclient (Apr 28, 2025, 10:55 AM)
did i hear this right - option D only allows EOF to read legacy byte code?

fiddy (Apr 28, 2025, 10:55 AM)
Vyper does not support it.

gakonst (Apr 28, 2025, 10:56 AM)
Basically if Option D has high conviction on being solid and good lets do it. If not, just dont do EOF

Vectorized (Ben) (Apr 28, 2025, 10:56 AM)
Option D, or i’ll abuse escape hatch.

If escape hatch is somehow banned in the future, i’ll write an RIP to add back escape hatch functionality. ;)

Ansgar Dietrichs (Apr 28, 2025, 10:56 AM)
we are already changing the prioritization process for after Fusaka - so the point by Fede is valid, but more backwards facing

gakonst (Apr 28, 2025, 10:57 AM)
Option D to me = ALL introspection is back

gakonst (Apr 28, 2025, 10:57 AM)
Zero contract breakage in Solidity

Felix: EVM was not a good design from the begining. The motivation in having EOF is to improve EVM

Justin Florentine (Besu) (Apr 28, 2025, 10:58 AM)
nobody ever asks for platforms to pay off techdebt, their stewards are responsible for doing that BEFORE the users notice it has decayed

gakonst (Apr 28, 2025, 10:58 AM)
The goal of the conversation is: can you do EOF w/o breaking Solidity code. Whatever gets us that. If not, then EOF cannot get rollef out.

Hadrien Croubois (OpenZeppelin) (Apr 28, 2025, 10:58 AM)
That is not true. Application have asked for create2

Hadrien Croubois (OpenZeppelin) (Apr 28, 2025, 10:58 AM)
and tstore/tload

Carl Beekhuizen (Apr 28, 2025, 11:00 AM)
Just to reiterate, you still can't introspect EOF with option D

Fede | Lambda (Apr 28, 2025, 11:00 AM)
The EVM is an objectively bad VM. We fully agree in that. The problem is that you don’t make it better just because you can.

gakonst (Apr 28, 2025, 11:00 AM)
Guys Ethereum doesn't need a better VM to not lose at the moment, it needs this grp of people to do what APP DEVS want.

gakonst (Apr 28, 2025, 10:57 AM)
Zero contract breakage in Solidity

Ben Adams (Apr 28, 2025, 11:00 AM)
Can tell is a contract etc

Danno Ferrin | Ipsilon (Apr 28, 2025, 11:00 AM)
use of CREATE and CREATE2 in assembly blocks would need to change.

gakonst (Apr 28, 2025, 11:00 AM)
You cannot Break solidity.

Just non starter

df (Apr 28, 2025, 11:02 AM)
It is true that App devs do not request VM changes. But VM changes should trace back to something that app devs want.
Stack too deep is a one of those asks, but it can be completely done without EOF.
Everything else does not seem like it has any strong benefit for app devs.

Justin Florentine (Besu) (Apr 28, 2025, 11:02 AM)
does just rebranding this as a whole new VM help?

Carl Beekhuizen (Apr 28, 2025, 11:00 AM)
Just to reiterate, you still can't introspect EOF with option D

Dan Cline (Apr 28, 2025, 11:02 AM)
The pointed q is "what breaks with option D"?

iPhone (Apr 28, 2025, 11:02 AM)
With legacy

Vectorized (Ben) (Apr 28, 2025, 11:02 AM)
its ok for my use case, which is handcrafted legacy bytecode verification.

i understand that EOF bytecode might not be the same thing on different platforms.

jochem-brouwer (EthJS) (Apr 28, 2025, 11:04 AM)
I think there is a big gap between core devs and app devs. Where can we find app devs? What do they want?

Tim Beiko (Apr 28, 2025, 11:04 AM)
Many are in this chat, telling us 😄

Storm Slivkoff (Apr 28, 2025, 11:04 AM)
Are we 100% sure that these option-d-banned operations are truly rare corner cases? Has anyone checked?

gakonst (Apr 28, 2025, 11:00 AM)
Period. Do you guys write smart contracts?

Fede | Lambda (Apr 28, 2025, 11:04 AM)
It-s just researchers pushing something because they did so and they are about to break compatibility and put network effects at risk

Ben Adams (Apr 28, 2025, 11:04 AM)
Yes; why do you want to read container formatting?

Storm Slivkoff (Apr 28, 2025, 11:05 AM)
Can you repeat succinctly what types of introspection are banned in option D?

Vectorized (Ben) (Apr 28, 2025, 11:10 AM)
as a library dev, i think most of the compute is spent on massaging bits to pack into storage. and 256-bit defi maths.

and its possible to speed all these up with a simple clz opcode. so EVM is not stagnent, even if we don’t have EOF.

fiddy (Apr 28, 2025, 11:11 AM)
pls sqrt and cbrt as an opcode

jochem-brouwer (EthJS) (Apr 28, 2025, 11:11 AM)
We need a place where smart contract devs can communicate this. Is this place somewhere?

lightclient (Apr 28, 2025, 11:09 AM)
If we don’t know what the right decision is, then we know what the decision is. No need to keep going back and forth. Remove EOF from fusaka, add minimal changes to fix stack-too-deep.

Storm Slivkoff (Apr 28, 2025, 11:10 AM)
Such a large complexity change should require an enthusiastic yes from most/all relevant stakeholders

Dan Cline (Apr 28, 2025, 11:11 AM)
Yes, and we should not have decided on something without full agreement on an exact scope / version, with knowledge of the consequences

Vectorized (Ben) (Apr 28, 2025, 11:13 AM)
not a core dev… but i think adding EOF (even tho it’s additive) adds pema maintainence overhead and drag to all client teams.

stokes (Apr 28, 2025, 11:12 AM)
It does sound like we all agree we should have more input from apps to make a decision here, and that suggests we should wait for that to happen before making a decision

If that’s the case, we take EOF out for now and re-evaluate once we have the data we need

Justin Florentine (Besu) (Apr 28, 2025, 11:13 AM)
by that reasoning, we will never pay off techdebt ever, because the users won't care about it till it is too late.

frangio (Apr 28, 2025, 11:13 AM)
there needs to be a post mortem on how we got here

iPhone (Apr 28, 2025, 11:14 AM)
Before the call ends, can we make a decision or at the least commit to a deadline?

Tim Beiko (Apr 28, 2025, 11:14 AM)
I will also note that while Core Devs choose what code they write, they don’t decide if the code actually gets adopted

Ahmad Bitar | Nethermind (Apr 28, 2025, 11:15 AM)
You could have participated in the discussion when the EOF team was working on the spec. Every client team had a representative

Felix (geth) (Apr 28, 2025, 11:15 AM)
I can only reiterate that we are shipping EOF mostly for the benefit of the implementers, to enable a faster EVM. At the same time we are trying to ensure contract development is not affected too much. This tradeoff always applies to all EVM changes, it's just a bigger change this time.

Ben Adams (Apr 28, 2025, 11:15 AM)
If we pull EOF we will need tests to ensure no consenus failures from the code being removed

Tim Beiko (Apr 28, 2025, 11:14 AM)
I will also note that while Core Devs choose what code they write, they don’t decide if the code actually gets adopted. This is a pretty major consideration and we should weight it.

Similarly, users can “exit” to other chains, not just by forking.

fiddy (Apr 28, 2025, 11:15 AM)
Ideally we'd want users to not want to exit.

​​​​Barnabas (Apr 28, 2025, 11:17 AM)

I’d really prefer not to keep dragging this.

Ansgar Dietrichs (Apr 28, 2025, 11:17 AM)
it is just insane to ship a feature that still has so much confusion and tension around it.

And I do feel sorry for the EOF team, and I personally would have preferred to ship EOF in fusaka.

But it’s time to make the only possible decision and move on. focus on doing this better in the future.

Łukasz Rozmej (Apr 28, 2025, 11:15 AM)
I can understand with small course correction like going with option D.

To not include or not in Fusaka is 6 month late to discuss.

Łukasz Rozmej (Apr 28, 2025, 11:17 AM)
not really

Łukasz Rozmej (Apr 28, 2025, 11:17 AM)
solidity IS the user of EOF

Tim Beiko (Apr 28, 2025, 11:17 AM)
To Ansgar’s point re: “anti-EOF people did the worst job of the process”, I think it’s worth being way more charitable to people whose full time job is not to engage with ACD and to dismiss their concerns because of how they came in

Tim: Disgrees to the late removal from the upgrade. The process is for the outcome.
I am not comfortable keeping it in Fusaka, gave an example of Moody that him being the solidity developer is nt very excited for EOF getting in.
It feels like if people wants EOF, there has to be more onboarding of the app devs and team supporting. But we are not at the point yet.
My suggestion, remove EOF from Fusaka. From process, don't open the door for anything new.
If we want to improve the EOF we should
I don't see a world where we strongly push EOF. It is not a technical issue. If 4/5 client team wants to have it, we should probably go with that, but I don't have a say.

jochem-brouwer (EthJS) (Apr 28, 2025, 11:22 AM)
Cool :)

frangio (Apr 28, 2025, 11:23 AM)
if declined EOF should definitely make a comeback with strong fundamentals and prove its value

pcaversaccio (Apr 28, 2025, 11:23 AM)
IMHO, the feedback has been coming later in the process because the entire complexity of EOF and the constant changing is consuming too much time for any non-core-devs. The best way to not gather external feedback is to write complex and constantly shifting specs.

jochem-brouwer (EthJS) (Apr 28, 2025, 11:22 AM)
We have PAY CFId for EOF, I want PAY standalone if we remove EOF

SanLeo (Apr 28, 2025, 11:23 AM)
PAY is a great feature that app developers want

Vectorized (Ben) (Apr 28, 2025, 11:23 AM)
what was the rationale for not adding this to the legacy EVM?

Justin Florentine (Besu) (Apr 28, 2025, 11:23 AM)
this feels like a veto of SFI

jochem-brouwer (EthJS) (Apr 28, 2025, 11:23 AM)
Fusaka focus: PeerDAS. We have shifted EOF from Prague -> Osaka and now we can thus shift it to Amsterdam

Fede | Lambda (Apr 28, 2025, 11:26 AM)
I know we are the new guys here but please stop saying that all the client teams are in favor of EOF. The two new execution clients are against it.

Ansgar Dietrichs (Apr 28, 2025, 11:26 AM)
if we pull EOF, can we spend a very brief moment talking about next steps? when would we select the exact “thin replacement” (swapn etc)?

and can we agree that the main “replacement EL feature” would be focus on scaling and early glamsterdam prototyping?

Tim Beiko (Apr 28, 2025, 11:27 AM)
But IMO Scaling the Gas Limit is that feature

Vectorized (Ben) (Apr 28, 2025, 11:25 AM)
Say if we somehow have EOF… Are we open to add new opcodes to both legacy and EOFs going forward?

iPhone (Apr 28, 2025, 11:27 AM)
Harder in terms of adoption

Charles C (vyper) (Apr 28, 2025, 11:27 AM)
i think we should get signaling from ACDE that EVM changes will not be shut down going forward.

Tomasz Stańczak (Apr 28, 2025, 11:27 AM)
No standalone EIPs

fiddy (Apr 28, 2025, 11:28 AM)
Fund Venom so solidity can be compiled by an alternative backend. Venom is currently experimental but is extremely promising and has already integration with Vyper.

jochem-brouwer (EthJS) (Apr 28, 2025, 11:28 AM)
We need PAY. Also for 7702. CALL to 7702 just to dump eth will be expensive for CALL because it also has to warm this extra code address

Ansgar Dietrichs (Apr 28, 2025, 11:26 AM)
if we pull EOF, can we spend a very brief moment talking about next steps? when would we select the exact “thin replacement” (swapn etc)?

and can we agree that the main “replacement EL feature” would be focus on scaling and early glamsterdam prototyping?

Tim Beiko (Apr 28, 2025, 11:28 AM)
I don’t think so?

Gajinder (Apr 28, 2025, 11:28 AM)
Delayed execution

Carl Beekhuizen (Apr 28, 2025, 11:28 AM)
Agree with @Ansgar Dietrichs here, we should make some progress even if only symbolic

iPhone (Apr 28, 2025, 11:28 AM)
+1 on no replacement

Felix (geth) (Apr 28, 2025, 11:29 AM)
EL only fork!

Francesco (Apr 28, 2025, 11:30 AM)
We said on Thursday that the decision would be made now

Ahmad Bitar | Nethermind (Apr 28, 2025, 11:30 AM)
Agree with Justin

Marius (Apr 28, 2025, 11:31 AM)
Is it a vocal minority? I feel like the room is pretty split 50/50

Ansgar Dietrichs (Apr 28, 2025, 11:26 AM)
if we pull EOF, can we spend a very brief moment talking about next steps? when would we select the exact “thin replacement” (swapn etc)?

and can we agree that the main “replacement EL feature” would be focus on scaling and early glamsterdam prototyping?

Ben Adams (Apr 28, 2025, 11:31 AM)
not really; some are

Duncan Townsend (0x) (Apr 28, 2025, 11:31 AM)
Is it a minority, or does it only feel like it if nobody solicits opinions from app devs?

stokes (Apr 28, 2025, 11:34 AM)
+1 to Tim on handling in glamsterdam with better process

Marius (Apr 28, 2025, 11:34 AM)
I'm just tired, boss

iPhone - Radek (Apr 28, 2025, 11:34 AM)
Pawel wants to say something

Charles C (vyper) (Apr 28, 2025, 11:32 AM)
like EOF gives us technical debt and forced work

Charles C (vyper) (Apr 28, 2025, 11:34 AM)
can discuss more over DMs if you like

Hsiao-Wei Wang (Apr 28, 2025, 11:34 AM)
Thank you Tim

polar (Apr 28, 2025, 11:35 AM)
Sounds good, thx Tim.

Interop Testing Call #33 – April 14, 2025

Call Lead: Mario Vega

Pectra Updates

Barnabas

• Using MEV boost and have MEV builders.
• Will begin testing for flow.

Marius

• Re-reviewed Pectra tests due to concerns about insufficient coverage.
• Discovered 23 edge cases for EL and CL; addressing them.
• Confident in test coverage now.
• Developing scripts to test 7702 edge cases and validate pool health.
• Spencer and Justin working on new test cases.
• PR with more tests: #4265

Mario

• Tests once written should be executed.
• On EEST, working on postmortem report of testing.
• There should be a writeup and will include the edge cases so we don't miss them in future. Its WIP.

Enrico

• Raised issues with MEV-Boost v1.9-rc3.
• Working on SSZ compiled test plans.

Justin

• Will support debugging MEV-Boost issue.
• Tests will be available after nightly build merge.
• Asked if CL clients can bypass spec tests?

Terence / Parithosh / Pawan

• No bypass currently, but support is possible or being looked into.
• No clear answer on model-based testing yet.

Modified Withdrawal Contract Test Cases

Spencer

• Felix created a test modifying the withdrawal contract.
• Besu/Reth throw invalid block; Geth/Erigon accept it.
• Wants feedback from clients on whether this is a concern.

Daniel (Besu)

• Considered theoretical; low priority for now.

Som (Erigon)

• Erigon continues block producing silently.

Roman

• Issue with assumption about EIP constraints being handled by the system contract.

Spencer

• Will follow up only if other clients express interest.

Negative Testing in Execution Layer

Mario

• Found a few issues last week. Some exceptions were noticeable and not hitting the expected handler
• There was mismatch in receipt or gas use and they didn't hit the right handler
• Several exceptions weren’t caught as expected.
• Question - do we want to make exception mapping for clients as a mandatory check mandatory?
• Shared link: Invalid Blocks Notes

Lukasz

• Exception list changes infrequent at Nethermind.

Danno

• My concern is this brings exceptions into the protocol now. Before this what mattered was that it was rejected, not why it was rejected. If we are testing error strings we de facto make them part of the protocol
• Prefer flexibility in client behavior.

Jochem

• Warned against restricting implementation optimizations.

Mario / Lukasz

• We don't have to make it part of Protocol but just testing.
• Multiple exceptions can be accepted.
• Shared mapper examples

Attestation Metrics

Nflaig

• 7-day metrics improved but not perfect.
• Asked if another attestation analysis was needed for attestation performace or wait for the stable release?

Enrico (Besu) / Terence (Prysm)

• Current metrics are fine.

Bot Updates

Parithosh

• Introduced a new bot for building client images.
  Discord Link

PeerDAS Testing

Barnabas

• Devnet 5 ongoing; 75% participation.
• Issues with Reth noted.

Roman (Reth)

• Degraded block processing due to resource exhaustion.

Parithosh

• Machines had only 8GB RAM; resource usage expected to grow.

Rafael

• Asked if issues were on supernodes (confirmed yes).

Barnabas

• MEV workflow functional with Fulu.

Justin

• Proposed BPO fork.
• Shared thread

Marius / Justin

• I think they should just be normal hf on EL tbh. Compared to past forks (Glacier).
• EL side manageable, CL more complex. EL thread

Will / Manu

• PeerDAS discussion item for next day.

EOF

General

• Devnet 0 completed with clean fuzzing.
• Devnet 1 pending merge; only one PR left.
• Clients like Besu, EVM1 ready.

Marius

• Awaiting a stable devnet 1 spec.

Parithosh

• Shared devnet spec: EOF Devnet 1

Roman (Reth)

• Shared implementation branch: PR #2377

Barnabas

• Autobuilding the Reth branch moving forward.

Jochem (EthJS)

• Working on implementation; shared plans to publish the branch.

Som (Erigon)

• Implementation WIP; dev unavailable currently.

Interop Testing Call #32 – April 7, 2025

Facilitator: Parithosh Jayanthi

Next Steps

• Pari to follow up with Nimbus on attestation packing fix.
• Daniel to provide update on Besu teams validators running into hardware bottlenecks.
• DevOps to:
  • Prepare and execute a round of mainnet shadow fork testing.
• DevOps to update eth-clients metadata for the fork date.
• Testing teams to continue running tests and raise issues as needed.
• Problabs will publish Blob analysis results.
• Danno to submit PR for EIP editors' review by tomorrow.
  • Update and finalize test plan.
  • Add more clients to EOF Devnet 1.
  • Plan Devnet 2 (including opcodes).
• Parithosh and team will begin internal planning for Fusaka Devnet post-EOF.
• Parithosh will prepare for livestreaming of future testing calls.

Pectra – Hoodi

• Pari: Validator tests are complete; all intended features have been covered and the network looks stable.
• Pari: Noted an orphan block on Sepolia (by Teku); believed to be a one-off, and Teku is investigating.
• Sam: Continuing to analyze attestation performance.
• Terence (Prysm team):
  • Prysm updated attestation handling – changes merged.
  • No bugs reported; focus currently on proposer logic, not full network analysis.
• pk910: Cross-withdrawal address consolidation is still pending, but other teams have tested and confirmed it's feasible.
• Pari: Will wait to see if Nimbus corrects attestation packing.
• Nico: No notable changes in the last 7 days.
• Daniel:
  • Tested 25k validators on one Lighthouse node.

Specs & MEV Testing

• Mario Vega: Specs testing ongoing. Test suites being re-reviewed and executed against all clients. No major issues found yet.
• MEV testing is active – consistent stream of MEV blocks being observed.
• Partition tool has been used to gather client-side fixes, collected from clients' retrospective reviews.
• Pawan: Diff fuzzer has been running all EF test cases for a week – no crashes so far.
• Terence: Mainnet fork date update required in upstream specs.
• nflaig: Confirmed that update is likely only needed at:
  eth-clients/mainnet/config.yaml
• Pari: DevOps team will handle the update in the repo.

PeerDAS

• Pari shared link: PeerDAS Devnet 6
• Barnabas Busa:
  • Devnet 6 setup in progress with Kurtosis.
  • Lighthouse and Grandine clients integrated.
  • Geth team aware of issues; fix pending.
• Marius: Code is fixed but slow; better hardware may help.
• Pari: PR submitted by Barnabas to Kurtosis. All spam tools expected to work.
• Problabs is analyzing mainnet blob usage; report to be published in coming days.
• Marius: Geth does not yet support RPC v2 – patch expected tomorrow.
• Barnabas: Asked for clarification – whether the fix relates to a v1 wrapper. Parithosh confirmed it.

EOF (EVM Object Format)

• Danno:
  • Working to roll out specs for EOF.
  • Test plan (including EEST) will be updated.
  • EVM 1 is already running EEST.
  • Having 3 clients running will be nice to have – aiming for more.
• ETA for EOF Devnet 1 aligns with Pectra fork timeline.
• EOF Devnet 2 will include Opcode changes.
• Pari asked if PeerDAS EL version is easy to rebase.
  • Marius + Roman: Confirmed rebasing is straightforward.
• After EOF Devnet 1 is stable, Fusaka Devnet planning will start (Interop, June).
• Pari shared: EOF Devnet 1
• Livestreaming:
  • Danno: Shared concerns but agreed livestreaming will be needed eventually.
  • Async fallback: Use Party Lounge when needed.

Interop Testing Call #31 – March 31, 2025

Facilitated by: Parithosh Jayanthi

Next Steps

• Lucas is working on block reward and effectiveness analysis; results to be shared once available.
• Mario will prepare more EEST tests and a potential release.
• Tim Beiko & Stokes will collect async input and coordinate the decision on the Pectra mainnet fork block date.
• Parithosh will create a tracking page for EOF Devnet 1, potentially combining testing EOF with PeerDAS Devnet 6.

Hoodi Testnet

• Pectra fork occurred last Wednesday.
• Testing details:
  • PK910: EL-triggered exit seems fine.
  • Partial withdrawals yet to be tested.
  • Some pending in deposit queue – to be followed up.
  • Initial testing with 7702 completed.

Attestation Issues

• Pari: No major outages so far, but attestations less optimal than expected. Discord thread
• Enrico: Haven't looked very closely, but essentially clients lack optimal attestation packing algorithms. It was a big problem on testnet but not on mainnet.
  • Teku still has a naive version; updates planned.
  • A bug affecting attestation propagation fixed in latest Teku release.
• Pari: Most of the issues were on scaling up.
• Enrico: Teku running very big number of attestation and takes close to 3sec to publish. If running 5k validator, it should not be a problem.
• nflaig: Double checked the metrics. Only timely target attestations get rewarded. Many don't make it to the next block.
• Saulius: Some clients not packing blocks efficiently; could be identified by comparing client behavior across the same slot.
  • Currently no exact data; proposing a joint client comparison effort.
• Nico: Acknowledged the difficulty but noted that some patterns might be visible through existing data.
• Nico: Mentioned that Lucas is conducting an analysis on block rewards and other metrics, and more data should be available soon.
• Saulius: Proposed a joint effort to test and compare client performance for the same slot, which could yield clearer insights.
• Nico: Not entirely convinced it’s critical to compare within the same slot but agreed it might be ideal.
• Pari: Suggested checking with the Vouch team, who may already have relevant data.

Next step: Parithosh will check with Vouch team, they might have helpful data.

Effectiveness Concerns & Performance Metrics

• nflaig: If this goes live on mainnet as-is, users will likely complain about reduced effectiveness—even if it's not a critical network risk.
• Parithosh: Agreed, users are accustomed to high performance and will notice the drop.
• nflaig: Users already complain when effectiveness drops from 99.9% to 99.7%.
• Marius: Asked for current effectiveness metrics.
• Parithosh: Head rate is fluctuating between ~75% and 88%, while it should ideally be around 92%.
• Marius: That’s a significant drop.
• Parithosh: It definitely needs to be fixed before mainnet. It's an optimization issue, not a bug.
• Marius: A 17–4% drop is major—why wasn't this caught earlier? Could it be related to the change from 256 to 8 aggregates per block?
• Parithosh: it's an optimization issue, not a pure bug. Issues like this require a public network to surface; they may not appear in smaller-scale testing environments.
• nflaig: Lodestar runs 5k keys per node – may perform better on mainnet.

Client-Specific Updates

• Parithosh mentioned that Potuz shared a Fork choice issue that is fixed in Prysm.
• Pawan: The issue was related to epoch processing in Lighthouse. It performed balance updates twice during a single pass, which caused the problem. Most clients are now aware of the issue, and it has been fixed.Lighthouse fix PR
• Pawan: Working on improved fuzzing to detect similar issues earlier.
• Mario: Some RLP tests were ran locally, EIP-7702 tests passed across all clients; planning a test release next week.

Pectra Mainnet Block Discussion

• Tim Beiko: Asked how the group should approach selecting the Pectra fork block.
• Barnabas Busa: Suggested deferring the discussion to Thursday's ACD call.
• Parithosh: Proposed doing a temperature check in the meantime.
• Stokes: Supported a temp check, noting uncertainty among CL teams given recent updates.
• Pawan: Expressed low confidence at the moment.
• Nixo: Highlighted that confirming April 30 on April 3 would break the 30-day notice commitment.
• Pawan: Requested a few more days to complete fuzzer runs.
• Stokes: Asked if teams are comfortable with April 30 "vibes-wise."
• Stokes: Requested if specs could be released earlier than Wednesday.
• Justin Traglia: Confirmed specs could be released sooner.
• Som (Erigon): No objections to April 30 but suggested May 5–6 as a backup if the decision is made Thursday.

Next step: Tim and Alex will collect more decision data points from async chat.

PandaOps & Coordination Tools

• Hive dashboard for tracking.
• PandaOps bot available in interop Discord channel.
• Dedicated Discord channels for:
  • Pectra
  • EOF
  • PeerDAS
• Client teams feel free to tag PandaOps as needed.

PeerDAS Updates

• Will is helping with coordination and devnet spec updates.
• Current status:
  • Geth PeerDAS implementation still under test.
  • n/m implementation is there, yet to tested. Bug in Prysm has been fixed.
  • Waiting on clients to havegetblob v2 support.
• Will be discussed in upcoming PeerDAS testing call.

EOF (EVM Object Format)

• Danno:
  • Devnet 0 running.
  • Erigon added fuzzing.
  • EOF Devnet 1 will follow after Pectra mainnet release.
  • Working on devnet release and EIP updates.
  • Will ping Discord once specs are finalized.
• Pari:
  • Suggests combining EOF Devnet 1 and PeerDAS Devnet 6 in may.
    Next step: Parithosh Will create a devnet tracking page.

Interop Testing Call #30 (March 24, 2025)

Moderated by: Parithosh Jayanthi

Updates on Hoodi Testnet

• Network Status:

  • The network has been live for a week.
  • Testing teams have reached out to various entities interested in deposits; most deposits are already done.
  • If anyone else is interested, please reach out to Parithosh.
  • Expecting the Pectra fork within the next two days.
  • Validator Keys are already being handed over to the client team that reached out. Anyone missing should contact the testing team.
  • Almost all clients have the release, except for Geth.
  • Tim Beiko will update the blog post to include the client release details.

• Mikhail:

  • Noted a participation rate of 95%.
  • Would like to maintain a stable participation rate even if it stays at 95%.

• Pari:

  • Informed that most variability is due to the ongoing key handovers.
  • Will follow up with other teams to increase participation.
  • Suggested that perhaps key handovers should be paused before the fork to ensure stability.
  • Shared a validator activity link for reference: Validator Activity Dashboard
  • Mentioned that Xatu nodes are currently actively monitoring.

• Mikhail Kalinin:

  • Raised the topic of the parse_deposit_data PR (EIP PR #9460) for discussion if time permits.

Pectra Devnet 6

• Status:

  • The network is live and under testing.

• Barnabas:

  • Noted that participation is low, possibly due to issues between Geth and Lighthouse (LH).
  • Mentioned that other clients appear to be functioning well.
  • Advised that LH and Geth devs should look into the machines.
  • Will follow up with the Geth team later.

EIP-6110 Clarification

• Mikhail:

  • Discussed two approaches after a conversation with Jochem:
    • Full Dynamic Approach:
      • EL (Execution Layer) clients parse data based on the ABI.
      • Parsed data is passed directly to the CL (Consensus Layer) without converting it into a fixed type.
    • Fixed Layout Approach:
      • EL clients support only the mainnet layout with fixed offsets and sizes for data.
  • Believes the fixed layout works well from the CL perspective.
  • Is in favor of fixed specifications but would like feedback from the EL and Testing teams regarding complexities.
  • Noted that N/m and other clients might require a dynamic approach for adjustments.

• Mario:

  • Raised a concern regarding handling larger fields in the dynamic case.
  • Pointed out that testing is simpler with the fixed approach.

• Gajinder Singh:

  • Suggested including the actual bytecode for the deposit contract in the EIP (similar to system contracts) since it is now considered a system contract.
  • Proposed that an informational EIP for permissioned chains (e.g., Sepolia) should detail any extra handling required.

• Jochem Brouwer (EthJS):

  • Noted that Sepolia has a different bytecode but maintains the same event layout for the DepositEvent.
  • Stated that the current PR explicitly checks the DepositEvent layout and will fail the block if there is a mismatch.
  • Shared the PR link again: EIP PR #9460

• Decision:

  • The next step is to proceed with the fixed type approach as suggested by Barnabas.
  • Mikhail and Jochem will work on updating the EIP specifications.

Fail on System Contract

• Parithosh Jayanthi:

  • Shared the link to the related PR: EIP PR #9508

• Som:

  • Proposed that the system contract should not fail silently if:
    • No code is found at the address.
    • The contract fails but returns the header; in such cases, the block should be invalidated.
  • Highlighted that Erigon is currently skipping failures silently, possibly due to assumptions about other clients.
  • Noted that the PR suggests a 30-million-gas limit for the system contract call.
    • The expectation is that the call should not consume all 30 million; if it does, it should trigger an out-of-gas error.
    • Out-of-gas should be treated as a general failure.
  • Emphasized that transaction pools should be aware of recurring failures.
  • Acknowledged that while such behavior is not expected, there is no algorithm in place to alleviate it.

• Jochem Brouwer (EthJS):

  • Asked if the system call concept could be generalized into a broader “system call” EIP since similar issues apply to other system contracts.
  • Raised a point regarding generalization in terms of gas limits and other applicable variables.

• Mario Vega:

  • Expressed concern that if the contract is not present at the time of the fork (making the block invalid), it would be impossible to advance the chain.
  • Asked about the mechanism to unstuck the chain in such cases.

• Som:

  • Responded that pushing the Pectra timestamp might be the solution.
  • Emphasized that the system contract is not optional for Pectra.
  • A chain that is not Pectra-ready should not activate this feature.

• Gajinder Singh:

  • Expressed agreement with Som’s proposal.

• Enrico Del Fante (tbenr):

  • Mentioned that the proposal makes sense.

• Ben:

  • Asked whether this mechanism would have stopped issues prior to the Holesky update.
  • Som confirmed that it would have, if there would have been the similar issue.

• Additional Discussion:

  • Concerns were raised regarding recoverability and the behavior in withdrawal scenarios.
  • Mikhail suggested expanding the discussion to consider a broader generalization of the system call—potentially as a separate EIP.
  • Pari mentioned that while generalization could be discussed asynchronously, it is unlikely that any final decisions will be made today.
  • There was consensus on the need for community-wide feedback on defining a “system call” as perceptions vary between the community and EIP authors.
  • However, Som highlighted the example of Gnosis, of slightly different ways of handling parameters. Thus one definition of system may not be a good idea
  • Jochem agreed to the importance of defining the ambiguous “system call” concept due to its heavy implementation specificity.
  • Roman emphasized the importance of finalizing the EL spec changes as soon as possible.

• Next Step:

  • Continue the discussion on the thread at ACD.

Peer DAS

• Status Update:

  • Devnet 5 continues to run.
  • Prysm is experiencing an excessive number of nodes.
  • The team is exploring a new file approach.
  • Lighthouse (LH) is encountering an issue with consistent column counts, with Dapp Lion working on it.
  • An issue between LH and N/m has been reported and is being addressed.
  • Aggressive testing is scheduled to begin next week.
  • Devnets are currently running with over 10 blobs, and experiments with more client combinations are underway.
  • Most clients will be focusing on self-proof computation.
  • The next call is scheduled for tomorrow; EL team participation is encouraged.

• Parithosh Jayanthi:

  • Shared PR details from Sunny Side Labs for further reference: Peer DAS Discussion

EOF Updates

• Danno: No significant update for this week.

Next steps

• Tim Beiko will update the blog post to include the client release details.
• Mikhail and Jochem will work on updating the EIP-6110 specifications with the fixed type approach as suggested by Barnabas.
• Continue the discussion on defining “system call” on the thread at ACD.
• EL team participation in Peer DAS call is encouraged.

Interop Testing Call #29 (Mar 17, 2025)

Launch of Hoodi Network

• Pectra testing can continue on Holesky as it is finalized.
• Eth Panda team is managing most of the validators and transitioning key ownership to clients.
• Hoodi is live and finalizing:
  • Dora Explorer
  • Hoodi Dashboard

Discussion on Hoodi Launch

• Tim Beiko
  • Asked about the fork slot/epoch → Epoch 2048
• Parithosh Jayanthi
  • Shared Hoodi repo: GitHub
• Barnabas
  • Confirmed 8192 slots.
• nflaig (Lodestar, ChainSafe)
  • Lodestar update: 1.28.0-rc.0 now available with --network hoodi flag.
  • A stable release will follow soon.
• Validator Transition:
  • Already in touch with Lido and Coinbase regarding migration.
  • Som (Erigon) suggested testing RPC transactions, especially for 7702-related bugs.
  • Pari mentioned the Hive test suites might provide more insights. But, will need to be confirmed from the testing team.

Holesky Testnet & Option D Discussion

• nflaig (Lodestar, ChainSafe)
  • Asked if there is still interest in Option D for Holesky.
  • Shared a PoC: GitHub PR #7570.
• Tim Beiko
  • I thought we agreed to not do it to save core dev time?
• Parithosh Jayanthi
  • Agreed that Option D should only proceed if it’s significantly easier than expected. He thinks its better to discuss the POC and then decide if it is worth it or easier approach than expected.
  • ACD team is already considering deprecating Holesky.

Nico (Lodestar) explained the PoC for Holesky revival also argued that community may have option to run custom branch for this. However, Parithosh has concerns about this flexibility of custom branch options.

• Alex: This doesn’t affect the exit queue knowledge.
• Pari: Only exit queue is broken in Holesky; other tests can proceed. Tooling is already there, people are already deploying it. So, will increase the gas limit as Holesky has already lot of community nodes running. He suggested to continue running it for the short term (approx half a year) and then take the call.
• Barnabas: Memory issues reported on devnet 6 with 60M gas, might not see a longer non finality period with heavy blocks on holesky.
• nixo: Predicts community nodes will start shutting down.

Future of Holesky:

• Matthew Keil: Questioned the need to keep Holesky running if Hoodi is live.
• Pari: Prefers keeping it running for a few more months to ensure smooth migration. Purely for giving time for Hoodi to spin up. Tooling takes a long time to come up. Considering Lido is also looking for migration, Devops team are open to take over validators. So, if any client team wants to get rid of validators, reach out to devops team.
• Barnabas: RPC Provider available at rpc.hoodi.ethpandaops.io.

Decision:

• Option D isn't considered at the moment.
• Holesky will be there for the next 6 months as playground for Client devs and will aslo be giving the community the option to migrate to Hoodi in the meantime.

On a related note, Danno documented RPC method that provides node-relevant configuration data for the current and next fork. This will be published tomorrow, he requested review and feedback.

Devnets

• Pectra devnet is live and running.
• Devnet 6 Updates:
  • Gas limit increased to 60M.
  • Barnabas: Primarily affects Besu & Teku (Java-based clients).
  • Matthew Keil: Asked for a list of clients with memory issues.
  • Danno Ferrin (Ipsilon): Java garbage collection is memory-hungry.
  • nflaig: Requested SSZ support for Engine API.

EIP-6110 Discussion

• Jochem (EthJS)
  • wanted to discuss a clarification for EIP-6110 (this clarification is rather theoretical since it only applies to (test)nets where the DepositEvent emits data of different length than expected (i.e the emitted data on mainnet/holesky/sepolia))
  • Noted that the parse deposit method is not defined in EIP-6110.
  • Two possible solutions:
    1. Hardcode the offsets and sizes.
    2. Use ABI-based parsing.
  • Marius: Prefers hardcoding over using an ABI library in consensus code or doing nothing.
  • Łukasz Rozmej: Suggested reading first 5 words (32 bytes each) to determine length.
  • Pari: Wants to align with precompiles to avoid invalid block production.
  • Ben Adams: Clarified that ABI format is fixed and includes offsets/lengths.
• Next Steps:
  • Discussion to be continued async.
  • Clients may add comment to this EIPs PR
  • Jochem: Will link the discussion thread on ACD meeting.

Handling System Contract Failures

• Som (Erigon): Asked about failures in system contracts (e.g., withdrawals).
• Pari:
  • Safe option is skipping failed transactions to avoid mempool-triggered bugs.
  • Concerned this could create a forking issue for clients without the bug.
• Ben Adams:
  • Emphasized fixed ABI format without needing an ABI decoder.
• Mario Vega:
  • If the execution result differs, it indicates an invalid block.
• Pari:
  • Plans to engage the CL team for further discussions.

Peer DAS

• Devnet 5 is live and running.
• 60% full nodes with malicious nodes at 60-80%.
• One bug identified and fixed.
• Unrelated bugs found on Lighthouse.
• Breakout room meeting scheduled for next day.

EOF

• Besu & EVM1 working on implementations for Devnet 1.
• Pari:
  • EOF and PeerDAS are running on the same testnet, making separation difficult.
  • Need to discuss EOF checkpointing.
  • Geth and Nethermind need updates.
• Danno (Ipsilon):
  • Suggested adding a switch for better control.

Next Steps

• Testing on Devnet 6 to continue this week.
• Findings from Holesky & Hoodi to be presented on All Core Devs (ACD) call.
• Encourage teams to fill out retro feedback form: Ethereum Pad.
• Expect client releases before end of the week (EOW).

Interop Testing Call #28 (Mar 10, 2025)

Sepolia Fork

Meeting Lead by Parithosh Jayanthi

• Last week, an issue was reported about the log image of the deposit contract. This bug affected Sepolia, but it would not have occurred on mainnet.
• RETH was producing blocks with transactions.
• A coordinated rollout was successfully executed.
• Since then, Sepolia has been finalizing without issues.
• PK started running the Assortor test and provided updates.
• Tests ran as expected.
• Pari conducted a sanity check and will share more details in the next couple of days.

Issue with BLST Library (Reported by Mareak)

• Some users reported issues with the BLST library, affecting N/m and other nodes.
• Some Ethereum clients use BLST, but the issue appears specific to this library.
• Roman: Did this issue cause client crashes?
• Mareak: Yes.
• Mareak: RETH might also be using BLST.
• Mareak: Needs more time to complete the report.
• The bug affects some calls but not all.
• The issue was discovered today. A discussion thread will be shared soon.
• Mareak will post details on the All Core Devs (ACD) channel for further collaboration.
• Library Link: BLST Repository

Holesky Testnet

• The team has been working on achieving finality on Holesky.
• Finality strategy: Shared by ethPandaOps.
• Goal: Avoid using a shadow fork unless necessary.

Plan:

• Plan A: Achieve finality on Holesky.
• Plan B: Use a shadow fork if needed.

Performance Discussion

Enrico:

• Running large nodes with many validators.
• Issue: Large nodes slow down when overloaded with single attestations from the network.

nflaig - Lodestar:

• Found a bug in the attestation process.
• Thinks it's a major issue, even if the node is stable.
• Working on fixing the bug and UX improvements.
• Once the attestation issue is fixed, further improvements will be addressed.

Pawan:

• Questioned why the Epoch only had 64% participation.
• Epoch Data Link

Pari:

• Attestation propagation and backing seem to be the bottleneck.
• Theoretically, there are enough validators to finalize.
• Finality is close to being achieved.
• Discord conversation link: Discussion Channel

Next Steps (Pari):

• Switch to LH nodes and coordinate with Pawan.
• Next, test with Lodestar.
• If unsuccessful, consider using a Shadow Fork in the coming weeks.

Bug bounty

Pawel:

• Found two separate bugs outside of production code.
• Pairing bug identified in Nethermind and Constantine.
• Detailed notes will be provided by team on bug bounties.
• BLS Precompile reported at least two bugs.

Jochem-Brouwer (EthJS)

• Shared discussion on BLS pairing issue: Ethereum Magicians Thread

Dancertopaz:

• Discussed BLS issue.
• Two vectors from specific clients triggered the issue.
• Working on a fix with EEST.
• Plans to review test vector coverage over the next two weeks.
• Adding two more dashboards this week.
• Pari: Reach out to Rafael for infrastructure support.

Marek:

• Asked which EL clients use BLST.
• Suspected Nethermind, RETH, and other EL clients.

User name - iPhone:

• Confirmed that most EL clients use BLST for EIP-4844.
• Geth uses Gnark for precompiles.

Paweł Bylica:

• Found another evmone bug in BLS MSMs, related to handling points-at-infinity.
• Link to fix: Evmone PR 1150

Parithosh Jayanthi:

• Shared EIP 9460 PR: GitHub Link

Jochem:

• ABI PR 9460 – Some missing details in EIP-6110.
• Lightclient (Matt) prefers including ABI as-is.
• Next Step: Jochem will open a PR to EEST to address bugs in EIP-6110 implementation.
• Discussion ongoing on Discord: Discussion Channel

PEERDAS

• Devnet 5 is running.
• A bug was found last night and fixed.
• Network is stable, except for a full disk issue, which has been resolved.
• Moving towards more full nodes instead of super nodes.
• Documentation shared by Optimism.
• EL developers need to get involved in the implementation.
• Further discussions will continue in chat.

EOF (EVM Object Format)

• Testnet plan is in progress.
• No further development on EOF Devnet 0.
• Planning to add three EIPs on devnet.
• First release will be rounded.
• Devnet 1 will have to be stable for at least one week before moving forward.
• EOF will ship with PEERDAS.
• Development progress is going well.
• Work is continuing on EOF Devnet 1 this week.

Summary

• Sepolia is finalizing smoothly.
• Holesky finality is close; attestation propagation is a bottleneck.
• Multiple bugs reported and being addressed (BLST, BLS, EIP 6110, etc.).
• PEERDAS and EOF developments progressing well.

Further updates will be shared in the next meeting.

Interop Testing Call #27 (Mar 03, 2025)

Call Lead: Mario Vega (EthPandaOps Team)

Holesky Network Updates

Client Status

• Lodestar (nflaig): Network looks stable, no issues in the past 4 days, liveness observed.
• Teku (Enrico):
  • Encountered an overflow issue in fork choice, leading to block production failure.
  • Investigating a 9MB fork choice dump for root cause analysis.
• Lighthouse (Roman):
  • 1K validators active but with minimal impact on network recovery.
  • Ongoing fixes, but full resolution is weeks away.
• Prysm (Terence):
  • Addressing high memory usage during syncing, nodes expected to be online soon.
• Grandine (Saulius): Fixing memory growth issue (work in progress).

Slashing & Non-Finality Risks

• Mikhail Kalinin: Recovery without full validator slashing requires time due to leakage mechanics.
• nflaig: 6K validators slashed, raising concerns about network finality.
• Roman: Requested insights on slashing progress, rates, and finality recovery impact.
• Mikhail: Estimated 18 days for full correlation penalties.
• Mario Vega: Recommended waiting until Thursday for mass slashing execution to minimize stress on the network.

Key Concerns:

• Slashing penalties remain low, impacting validator exit timelines.
• Manually crafting slashings might be the best approach for controlled network recovery.

Sepolia Fork & Shadow Fork Testing

• Issue Identified:

  • Need for coordinated slashing strategies to ensure chain recovery.

Proposed Actions:

• Geth Team (Marius):
  • Plan to shadow fork Holesky before Pectra to allow staking providers to switch nodes safely.
  • Staking providers may test consolidation on this shadow fork.
• Radek: Warned mass slashing may stress network nodes and suggested targeting at least 50% of nodes online before proceeding.

BLS Consensus & Withdrawal Queue Concerns

• Marius:
  • Two BLS consensus issues fixed, but gaps in code coverage remain.
  • EIP-7002 issue: Withdrawal queues can fill up completely, causing delays for validator exits.
  • Increasing the gas limit may further exacerbate withdrawal queue congestion.
• Alex: Does not consider this a critical issue and does not want to delay Pectra.

Potential Risk:

• Some execution-layer withdrawals may be blocked due to withdrawal queue inefficiencies.
• No immediate attack vector, but long-term impacts need evaluation.

Staking Pool & Fee Calculation Discussion

• Terence: Raised concerns over staking pool awareness and impact on fee calculations.
• Marius:
  • Redeployment may be required, leading to potential confusion for staking providers.
  • Users might send funds to incorrect addresses due to miscalculations.
  • Issue was previously unnoticed but is now being addressed via Telegram & HackMD updates.

Next Steps & Coordination

Immediate Actions:

• Shadow fork documentation to be prepared for staking pools.
• Formal report summarizing findings to be shared with all stakeholders.
• Tim Beiko:
  • Sepolia fork should proceed without delay, as contracts are already deployed.
  • Move technical discussions to All Core Devs (ACD) meetings for further evaluation.
• Marius: Will reach out to staking pools to ensure smooth coordination and network stability.

Final Takeaways:

• Slashing strategies require careful coordination to avoid network stress.
• Holkesky shadow fork testing planned before Pectra upgrade.
• Validator liveness & network finality remain top priorities.
• Execution layer withdrawal queue mechanics under review.

Interop Testing Call #26 (Feb 24, 2025)

Mario Vega lead the call.

Pectra Update

MEV

• Justin Traglia

  • Flashbots identified an issue and are working on a fix, aiming for Holesky.
  • Prysm and Nimbus are progressing on SSZ & Builder API.
  • Discussion Link

• Enrico

  • Wanted to confirm some Besu-related issues, if they were fixed.
  • Justin verified it is now working.

• Other Issues Identified:

  • Lighthouse checked Content-Type instead of Header, leading to unexpected responses.
  • Minor issues related to headers and payload transmission, but they are non-breaking.
  • These issues were observed on a local testnet, not Devnet 6.
  • Using Kurtosis should be fine for further testing.

Shadow Fork Update

• PK:

  • No progress yet on the shadow fork, it’s still not up.
  • Syncing issues persist.

• Rafael:

  • Issue with fetching Mainnet data, working on a fix.
  • Expected to be resolved by today or tomorrow.

Client Updates

Holseky & Sepolia Releases - Any Major Changes?

Lodestar

• May need a hotfix due to an issue relevant to Holesky.
• Issue wasn’t caught in local testing.
• Lodestar needs to handle proposing and validation correctly.
• nflaig explained a mismatch issue specific to Lodestar.
• A few solutions were discussed; the team will test and report back in the interop channel.
• Marek mentioned that if this is the same issue reported by Nethermind, a solution might already exist.
• Marek will connect the teams.
• nflaig:
• More details on Lodestar issue in this PR.

Other Clients

• Besu (Daniel Lehrner): No major issues, release looks good.
• Lighthouse (Pawan): Still using 7.0.0-beta.0 for Holesky.
• Nethermind (Marek): No major issues, everything looks good.
• Prysm (James He):
  • Fix for attestation-related issues in progress.
  • PR #14896
  • Another potential attestation issue still under debugging.
• Grandine (Saulius):
  • Minor fixes, but no major issues reported so far.
• Reth (Roman):
  • Local nodes updated, waiting for the hard fork.
• Marek: "Good luck, everyone, with Holesky today!" 🎉

PeerDAS

• No major updates from Mario Vega.
• Rafael:
  • Devnet 5 launched last week (relaunch of Devnet 4).
  • Fixed EL client version issue (issue not present in Devnet 5).
  • Pari deployed a different version of Geth, which caused syncing problems.
  • Since the issue was on EL, the Devnet was restarted.
  • Devnet 5 Tooling: PeerDAS Dashboard
• Pawan:
  • Jimmy added a PeerDAS Readiness Checklist.

EOF (Ethereum Object Format)

• Danno:
  • Declared Devnet 0 a success! 🎉
  • 3+ clients successfully synced, fuzzing tests ran, and all tests used the same commit.
  • Decision to leave it up or take it down doesn’t matter.
  • Known bugs are fixed.
  • EOF Devnet 1 is in progress along with related EIPs.
  • Devnet 1 will be kept live for at least 3 weeks.
  • Testing is on track for testnet deployment.

EEST Deployment Discussion

• If EEST wants changes in transaction creation, there will be an EVM 1 update.
• Discussion ongoing on EEST.
• A Proof of Concept exists, which is a good sign.
• Rafael Matias:
  • Asked if EOF tests are running in Hive.
• Danno:
  • Not yet.Still exploring how to use Hive.
• Mario:
  • Explained the Hive testing process.
  • Will help set up EOF tests in Hive.
  • Goal: Have a similar setup to Pectra Devnet 6 in Hive.
    • Example: Hive Pectra Devnet 6 Dashboard
    • There will be a n need of a release for further testing.

Closing Notes

• All teams are preparing for Holseky.
• EOF and PeerDAS Devnets progressing smoothly.
• Client teams addressing minor fixes for Holesky and Sepolia.

Next steps:

• Shadow Fork fix expected today/tomorrow.
• Lodestar hotfix testing in progress.
• EOF Devnet 1 launch coming soon.

Interop Testing Call #25 (Feb 17, 2025)

-> Notes by Mario on ACD Discord

-> Notes by Pooja 👇

Mario Vega facilated the call.

MEV Devnet Status (Justin)

• No major updates.
• Ryan (Flashbots) is working on consolidation requests.
• Reth builder (rbuilder) works but fails to include execution requests.
• PK (testing): Requested a note if more consolidation & withdrawal requests are needed for devnet-6.
  Next step: - Reth team may collaborate with Ryan to resolve issues.

Mainnet Shadow-fork

• Pari: Nodes are syncing from a snapshot, taking time but expected. No major issues anticipated.
• Release expected in 1-2 days.
• EIP-7702 transactions will not be included.

Client Updates on Pectra

Erigon (Som)

• Pectra testnet compatible version is released.
• No official v3 release yet.
• Working on Caplin issues and fixes.
• Integration with all other consensus layers is working.
• Official release expected in next 1-2 days.

Nethermind (Marek)

• All Hive tests are green, no issues.

Besu (Daniel)

• Client Release last week.
• All Hive tests are passing, no issues reported.

Reth & Geth

• No representatives on the call.

Prysm (James He)

• Client release out, another release is likely for bug fixes and missed cutoff items.

Pari: Sproul mentioned a low-priority bug in Interop chat, to be fixed later.

Teku (Enrico)

• Issue with less optimal blocks.
• Nooticing a decrease in attestation count.
• Considering a change in attestation selection algorithm.
• No planned release yet but will be included in mainnet release.
• Shadow fork testing suggested – Enrico agrees it’s a good idea but uncertain if many clients are affected.

Prysm (James)

• Prysm is affected.
• A bug fix release is planned.
• Some testnet features are missing, to be addressed in the next release.

Grandine (Saulius)

• Released v1.0, testing looks good.
• Minor issues to be fixed in a patch release.

Lodestar (Phil)

• Fixes included in 1.27 release.
• Minor cleanup work, but otherwise ready.

Electra on "Ephemary"

• Pk announced that Electra is activated on "Ephemary", working well.

Contract Deployment

• No contracts deployed on Sepolia or Holesky.
• Need data ASAP.
• Alex Stokes suggested pinging EIP authors for deployment.
• Mario will attempt deployment today or reach out for assistance.
• Pari: Unsure about mainnet deployment as it costs 0.75 ETH.
  Next step: Ping lightclient if needed.

PeerDAS

• Testnet issues with ping detected.
• Potential spec update expected.
• Breakout call scheduled tomorrow, any EL team may join.
• Marek: Asked if the change affects EL.
• Pari: Changes are being discussed in the ACD Interop channel.
• Daniel (Besu):
  • Block builders requiring proposer blocks need to work with KZG proofs, which are time-consuming. Both EL & CL must handle availability sampling.

EOF by Dano

• EOF Devnet-0 launched last week.
• Some bugs detected, fixes applied.
• Validated transactions put on-chain.
• Old RPC versions went offline.
• Nethermind deployed all fixes.
• Two Reth nodes, but one didn’t fork – investigation needed.
• Issues with transaction misplacement, but no unexpected EOF problems.
• PK dealing with Devnet issues, clinets finding the tool useful.

Bot for Devnet Monitoring (PK910)

• A new bot on Eth R&D will monitor Devnet activity:
  • Devnet Monitoring Channel
• Functionality:
  • Pings clients if a node stops working.
  • Reports affected nodes without tagging.
• Marek: Asked about the trigger mechanism.
• PK: It uses multiple checks:
  • Node failure to sync.
  • Node not advancing.
  • Block failure.
  • More conditions can be added as needed.
• Currently beta software, working to integrate it into dev workflows.
• Plan: Expand bot monitoring to PeeprDAS and EOF soon.

Erigon (Som)

• Asked if Kurtosis Pectra tests are running.
• PK: Working on assertor tests (30-hour tests), updates will be shared.
• Som: CI integration ongoing, but Kurtosis breaks in some scenarios.
• PK: Will work on shorter test cycles.

New Hive UI

• Rafael (Ethpanda) working on UI revamp.
• Improved client results visibility.
• Error messages are clickable for further details.
• Planned integration into Panda Pulse for daily client bug summaries.
• Feedback welcome – reach out to Rafael with suggestions.

Interop Testing Call #24, February 10, 2025

Interop Testing Call #23 (Feb 03, 2025)

Key Decisions & Next Steps

• Devnet 6 Deployment: Announcement in Interop channel; client teams should prepare for testing.
• EIP-7702 Implementation: More documentation needed; community contributions welcome.
• Tx Pool Tests: Address specific cases, integrate with EEST and Hive testing, review test list.
• PeerDAS: Continue edge case syncing; prioritize after Pectra completion.
• EOF Testing: Prepare for Kurtosis-based testing after Pectra Devnet 6.
• ePBS & IL Testing: Proceeding with standalone domain testing before considering mainnet inclusion.

Devnet 5 & 6 Updates

Devnet 5

• Nethermind BLS issue fixed; nodes restored.
• Finalizing Devnet 5; focus shifted to Devnet 6.

Devnet 6

• ACD meeting suggested address change; so new EEST release is out.
• Kurtosis is also updated; most clients passing Hive test.
• Execution spec test running; one bug in Assertor fixed.
• Since interop went well, Pari went ahead and set up node for devnet 6 is up. Genesis file setup complete; Grandine reached out.
• Lido also reached out to test.
• Erigon Caplin request – Pari to follow up after the call.
• MEV will be there with Devnet 6, sanity check will be completed before release.

Note: Devnet 6 announcement will be made on Interop Channel.

EIP-7702 Implementation Resources

• Documentation needed; contributions welcome.
• Delegation Signing (Forge Test):Guide
• Viem 7702 Support: Docs
• Ethereum EIP PR: #9250
• Drafting 7702 Transactions (Cast): Guide

Client Updates

• Geth:
  • Most PRs merged, stable release expected this week.
  • Testnet updates by next week.

Tx Pool Testing Plans

• Matt Garnett asked if there will be testing for critical cases?
• Reference List: Geth PR #3107
• PK is working on tx invalidation tests.
• Mario’s Team:
  • Creating tx tests for EEST and Hive.
  • Looking for community feedback on Execution Spec Test Discussion.

PeerDAS

• Interop completed last week.
• PeerDAS Devnet 4 spec testing for clients ongoing.
• 2 clients fully ready, 3 partially ready.
• Once launched, high success probability.
• Out-of-memory bug in Prysm; PeerDAS test scheduled for tomorrow.
• Edge case syncing needed.
• Full focus after Pectra completion.
• PeerDAS Devnet 4 Notes

EOF Testing

• Fuzzing progressing well; N/M added.
• Reth-N/M Besu integration.
• Next Steps:
  • Kurtosis-based testing post-Pectra Devnet 6.
  • Ensure execution transaction test coverage.
  • EOF support from Reth now in the main branch.
  • Testing team available for Kurtosis support.
• Client Docker Image Builder

ePBS & Inclusion List (IL) Testing

• IL and ePBS testing support added.
• Running local devnet between Prysm & Geth; Lighthouse to be added.
• Tested both Fulu and Electra forks.
• Consensus: Premature to include ePBS in Fulu fork.
• Preferred approach: Own fork & separate epoch for measuring impact.
• Standalone domain testing for IL & ePBS ongoing.

Thanks to Pari & PK for setup!

---

Interop Testing Call (Jan 27, 2025)

Meeting Summary: Devnet 5, Devnet 6, and Related Updates

Devnet 5

• Devnet-5 is finalizing successfully after addressing key issues.
  • Dashboard link
• Client Updates:
  • Erigon: Fix implemented, resolving key issues and enabling finalization.
  • Lodestar: Syncing is functional but block import issues persist.
  • Prysm: Issue under investigation, early stages of debugging.
  • Lighthouse: Debugging ongoing; no update yet.
  • Geth: Issue appears resolved, pending verification.
  • Grandine: Node resynced, debugging continues, and it is now proposing blocks.
  • Besu: Block processing issue identified, tracing and debugging underway.
• Hive issues remain a blocker for Devnet-5 and need resolution before progressing to Devnet-6.
• Decision: Continue with Devnet-5 until fixes are in place, then transition to Devnet-6.

Devnet 6

• Timeline: Tentative release this week, dependent on client readiness and passing Hive tests.
• Consensus specifications are finalized; EL specifications are nearing completion.
• Plan:
  • Add Devnet-6 to Hive for clients already passing Devnet-5.
  • Enable MEV from the start of Devnet-6 (Bloxroute ready for testing).
• Next steps: Ensure all clients address outstanding issues to meet the timeline.

PeerDAS

• Updates to configuration values shared in the PeerDAS channel.
• Interop testing underway, progressing well.
• Lighthouse team close to PeerDAS devnet readiness, working on improvements and testing stability using Kurtosis on Kubernetes.

EOF

• Testnet spec shared: EOF Testnet Plan.
• Initial fuzzing with Geth, Besu, and REVM shows no errors—indicating readiness for Osaka Devnet.
• Decision: Launch EOF devnet next week following Pectra spec finalization.

Hive

• Hive tests now execute in parallel, resulting in a 6x speedup and cost efficiency.
• Hive integration for testing Devnet-6 is a priority, as it remains a critical tool for identifying client issues.

Key Links Shared in the Call

• Fork Monitor
• Node Monitor GitHub Repo
• Hive Testing Workflow

Next Steps

• Continue debugging and fixing outstanding issues in Devnet-5.
• Transition to Devnet-6 as soon as Hive tests are passed.
• Plan EOF devnet launch for next week.
• Ongoing PeerDAS and Kurtosis testing to ensure stability over time.

Interop Testing Call (Jan 20, 2025)

General Petcra Update

Nethermind

• Progress is on track; requires updates for blob base fee, baseFeeUpdateFraction, and EIP-7702.
• Changes are straightforward and expected to be completed within a few days.
• Encountering some test failures on Hive, particularly with the consume engine. The team is investigating.

Erigon

• Work in progress; several Hive tests are failing.
• The team is addressing the issues and expects resolution in a few days.

Besu (Daniel)

• All tests failing due to a missing genesis file for block fee, which has been merged recently.
• Execution spec tests are still in progress.
• Hive tests will soon specify the base fee, requiring updates from all clients.

Reth (Roman)

• No significant updates yet.
• Awaiting EIP-7702 changes and plans to update the base fee on the main branch.
• Question on timeline:
  • Pari: Timeline details will be shared by Tim in the coming weeks. Fork expected around the 3rd week of February, providing sufficient time for releases.
  • Focus on achieving green Hive tests.

Other Updates

• Validator deposit setup remains available for clients.
• Image updates can be requested from Rafael or Pari.
• Lodestar and Lighthouse have joined testing.
• Work continues on Flashbot builder.
• Holesky launch date is undecided but will allow ample discussion time.

EIP-7702 Discussion (Delegation Introspection)

• Julian: Overview of proposed changes discussed in the last ACD call. Changes received positive reviews, aiming for a temperature check on implementation ease.
  • Open PRs:
    • PR 9248
    • PR 9250
• Justin (Besu): Ready to proceed; timing is the main factor.
• Pari: Updates indicate readiness from Nethermind, Reth, and Erigon. Pending approvals or comments from some teams.

Next Steps:

• Move forward with Pectra devnet 6, incorporating a new genesis file and validator set increase to 50,000–100,000.
• Mario will release updates with EIP-7702 changes and fixes.
• Client teams to review and comment on PRs as needed.

PeerDAS

• Manu: Rebase completed; focus shifts to Lighthouse and others in the coming days.
• Local interop expected in a few weeks, followed by a PeerDAS devnet.

EOF Update (Danno)

• Fuzzing is ongoing but not comprehensive; clients are at varying implementation stages.
• Plans remain unchanged, targeting late Q1 to early Q2 for a devnet.

Additional Notes

• Marek highlighted test connection issues on Hive; Mario to investigate.
• Finalized base fee updates for Nethermind: PR link.
• Ongoing discussions about fork transition tests and price updating fixtures.
• Mario reassured all required tests are in the Cancun folder and will be in release 1.3 before the next ACD call.

Interop Testing Call (January 13, 2025)

Pectra

• Pari will trigger the Hive update after Mario’s/Dan's fix.
• New tests have been added for EIP-7623.
• No more test releases for devnets are expected before starting on the testnet, with a full release planned this week.
• Tests are being consumed in Assertor, and initial results look promising.
• No other open PRs; the testing is moving closer to the final release and testnet phase.
• Depending on Dan's tests, Nethermind should pass all Hive tests.
• Erigon: Passing all Spec tests; Hive test results are pending. Code review is in progress and looks positive.
• Besu: In a similar position to Erigon, with all implementations completed and potentially quicker to release.
• Open Topics:
  Eric raised the topic of SSZ support for builder flow on CL.
  Pari clarified that SSZ support is optional and not the main focus but will be tested in collaboration with the Flashbots team.
  Lighthouse is still missing from Kurtosis tests, which impacts Pectra testing as Lighthouse is essential. Updates are expected this week.

Note: Pectra Devnet 5 will launch without MEV support by default but will add it later.

PeerDAS

• Highlights from the last meeting:
  • Most clients are working on rebasing and local testing.
  • Standardizing metrics may temporarily break some clients.

EOF

• Danno will merge the branch for testing.
• Some issues with block tests were noted but are expected.
• Building for the Osaka 0 testnet will start once there are three merged clients.
• Geth and Erigon have yet to merge their changes.
• A smoother release process is anticipated when shipping is planned.