# ACD Testing Call #52 - Sep 08, 2025

Facilitator: Barnabas Busa

## Fusaka Devnet – Status Updates

Barnabas
- Devnet 3 participation remains high at 82%
- A few issues reported with Nimbus and Erigon
- Fixes from both clients are expected by tomorrow

**Erigon** (Andrew A.)
- No fixes yet; identified a flaw in Erigon logic
- Debugging continues — issue is tricky and requires more time
- Positive: Batch created to fix Fusaka Devnet 5 Hive dashboard
- Barnabas Busa: Note - Hive testnet should be covered by all clients

**Teku** (Enrico)
- Working on one syncing issue
- Merged PR for a rejection issue
- Investigating a memory issue in RETH (unrelated to earlier memory issue)

**RETH** (Roman)
- Investigating an issue with the pair
- Preparing a new release soon
- Issue has not occurred in previous devnets; will be addressed in Devnet 5

### Summary
- All known bugs were discussed
- Barnabas Busa: Looking for a 👍 from every client to confirm trunk branches are ready

References:
- [Syncing Bugs Notes](https://notes.ethereum.org/@ethpandaops/fusaka-syncing-bugs)
- [Validator Summary (Devnet 3)](https://dora.fusaka-devnet-3.ethpandaops.io/validators/summary)
- [Dora](https://dora.fusaka-devnet-3.ethpandaops.io/)

## Devnet 5 – Timeline & Readiness

**Grandine** (Saulius)
- All syncing issues fixed
- Devnet 3 is syncing properly

**Prysm** (Manu)
- Missing feature: backfill
- Ready for Devnet 5 on the develop branch

**Teku** (Enrico)
- Current state looks fine
- Bug discussed is not a blocker

**Lighthouse** (Pawan)
- PR open with a different sync strategy
- No blockers at the moment

**Lodestar** (Matthew)
- Identified a bug → hoping to fix tonight
- Refactor in progress, cleanup to be merged ASAP
- No blockers for Devnet 5, can be run locally.

**Nimbus** (Agnish)
- Blocking issue: bug in block validation pipeline for super node
- Fixes expected in the next few hours
- Team discussing another non-finality test for Devnet 3
- Devnet 5 readiness: trunk branch updates should be ready by early morning

**Geth** (lightclient)
- Ready for Devnet 5, no blockers

**Reth** (Roman)
- Ready

**Nethermind** (Lukasz)
- Fine, no blockers

**Erigon** (Andrew)
- Bug identified, investigation ongoing
- Main branch is ready

**Teku** (Justin)
- Ready

### Summary
- Most clients are ready or near-ready for Devnet 5
- Remaining blockers: Nimbus (super node bug), Erigon (under investigation)
- Devnet 5 can be scheduled for the **second half of this week**

## Shadowfork Testing
Barnabas Busa
- Main blocker: Geth and Erigon – issues due to blob schedule fields
- Shadowforks currently not possible on these two clients
- Geth (Matt): still investigating a regression noted in the syncing bugs doc; issue is related to the state history feature, not Fusaka.

## Client Releases for Holesky

Barnabas Busa

- Target: By next Monday?
- Nethermind: Yes
- Timestamp: Not yet decided → can be set async and shared on Thursday’s call
- Erigon: Difficult due to other client releases; one week is too soon, two weeks would be better

From chat
- Roman: I don’t mind. It would be close to our current release, but this shouldn’t impact the decision to proceed quicker with the Holesky fork.
- Pawan: Need to check with the rest of the team as well.
- Matt: Need to check with the team

### Summary
- Barnabas Busa: Will accelerate Devnet 5
- A full release for Holesky is not expected
- Targeting CR by 22 Sept
- BPO value still to be determined — once set, everything will be ready

## Gas Limit Testing Update

Nethermind (Marcin):
- Running EEST test
- Tooling available to run stateful tests
- Completed experiment with Hoodi
- Next step: Jochem to run the test

## Glamsterdam Testing Updates: BALs & ePBS

### **BALs**

Jared:
- BALs Testing team made an initial release
- Two issues found with EELS implementations of the spec, currently being addressed
- Timeline: Geth readiness very soon; unclear if other teams are aligned

Nethermind (N/m):
- Prototype available

Filipe:
- Specs need updates, currently, under review
- Another release expected this week

Mario Vega: [BAL Release v1.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/bal%40v1.0.0)

EEST (Spencer):
- Released [EEST v5.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v5.0.0)
- Clients are aware; see release notes
- Future test releases will come from EELS

General State Tests:
- Updated post-Cancun
- Refilled general state tests → recommended running fixture tests
- Ensure all clients are passing
- More info in release notes; reach out with questions

### ePBS
Barnabas
- Consensus specs are still under review
- Will circle back once updated

That concludes today’s agenda.

# ACD Testing Call #51 - Sep 01, 2025

Facilitator: Mario Vega

**EthPandaOps** (pk910):
- Unfinality observed, but recovered.
- Finality lost again ~30 minutes ago.

#### Client teams update:
**Prysm** (Manu):
- 3 nodes down.
- Reth and Erigon issues ongoing.
- Invalid block reported → under investigation.
- Memory issue also reported in Prysm.
- For prysm-{erigon,reth}-1: wiping DBs may solve the issue, but awaiting EL inputs to avoid masking underlying problems.

**Lighthouse** (Pawan):
- Progress made on new sync algorithm.
- 1–2 edge cases remain to be fixed.
- New sync strategy likely to be merged soon.
- Will also investigate offline nodes.
- **Request**: Once finality is recovered on Devnet 3, schedule a planned chain split (similar to last week) with prior notice. This would allow client teams to monitor logs and metrics in real time.

**Lodestar** (Matthew):
- All nodes in sync.
- Only Reth is running optimistically → will be resolved.
- Client was stable on Devnet 3 earlier, now investigating what caused the regression.

**pk910**: Acknowledged chain split request — will coordinate.

**Mario**:
- Syncing issues are starting to dissipate.
- Continue monitoring Devnet 3 before moving forward.
- Asked about timeline for Devnet 5.
- **pk910**: Tentatively next week, pending fixes.

## Syncing Issues on Devnet-3

**Mario**
- Lodestar and Lighthouse have already provided input.
- Asked if there were additional comments.
- Matthew agreed with the request for advance notice on chain splits, to better monitor metrics.
- Findings from Non-finality
- No major issues reported.
- Everything went fine during the recovery.

### Summary
- Devnet-5 is expected to launch next week.

pk910 (chat)
- Will ping offline devnet-3 nodes shortly; also observed the unfinality.
- Regarding the split test: it will be redone, but only once the chain stabilizes. Splitting again without stability would not be useful.

## Gas Limit Testing Updates

**Kamil**
- Last week: worked on a stateful scenario using the existing database.
- Currently collaborating with Jochem B. on a new testing scenario.
- Developing generic tests.
- Default setup uses the EEST scenario.
- Addressed questions from the Besu team; they will investigate the current bottleneck.

**Besu** (Ameziane)
- Team assigned a developer to improve various arithmetic opcodes.
- Implementation is new; testing will require more time.

**Mario**
- Preparing a new EEST version, expected this week.
- This will support further testing.
- Plans to coordinate with Besu once the new EEST is ready.

### On 60M Gas Limit
- **Kamil**: Increasing to 60M can only be considered after Besu’s fixes are complete.
- Once all issues are resolved, discussion on 60M can proceed.

### ModExp on Geth
- **Marius**: Progressing gradually; a PR is open on Geth.
- This remains a priority.
- Will provide an update next week on timeline and next steps.

## Glamsterdam Testing Updates: BALs and ePBS

**BALs** (Toni)
- No major updates on his side.
- Testing (Felipe):
- Spec side has been cleaned up and is passing tests.
- Codebase merging is in progress; once completed, things look good to move forward.

**ePBS** (Terence)
- Working on spec tests in the Consensus spec repo.
- Tests are already available.
- For anyone implementing the new spec: tests exist—please reach out to Terence with questions.

**Note**
[Consensus Spec v1.6.0-alpha.6](https://github.com/ethereum/consensus-specs/releases/tag/v1.6.0-alpha.6) has been released, fixing the bug reported last week. Client teams are encouraged to check and update.

## Sunnyside Labs Testnet Updates

**Sunnyside** (J)
- Preparing for Devnet-5 launch next week.
- Continuing interim testing in the meantime.

## Meeting Summary
- Finality issues on Devnet-3 are being addressed.
- Teams will sync up next week for the Devnet-5 launch.

# ACD Testing Call #49 - Aug 18, 2025
Facilitator: Mario Vega

## Fusaka devnet status updates
Mario
- Devnet 4 is currently down.

Barnabus
- Devnet 3 remains online and will continue running for now.
- Devnet 5 is targeted for launch next Tuesday, expected to be with no further spec clarifications needed. It will be pretty big.

**Conclusion**: Progress looks strong. The only pending step for clients is merging into the main branch—no other concerns have been reported.

## BPO Static Tests Updates
Mario:
- New release is ready.
- Testing includes Devnet 4 scenarios.
- Tests will be run against all clients to verify the BPO issue.
- Hive updates will follow as soon as possible.

## Gas Limit Testing Updates

### 60M Gas Pre-Fusaka
- Discussion raised by Ben regarding moving to 60M gas limit.
- Besu concerns:
- Justin (Besu) expressed hesitancy about the move.
- Current blockers include ecrecover performance, which remains a concern post-fork.
- Kamil’s updates:
- Geth work-in-progress with integrated EEST and benchmark tests.
- In contact with the Besu team; new scenarios now available.
- Working on stateful testing and deeper issue analysis.
- Observed that EC performance is slow on Besu, while other clients perform well.
- Tim Beiko (chat) asked if unchanged ecrecover pricing in Fusaka is still a blocker → Justin (Besu): Yes. Known issue: multiple opcodes at 45–60M gas range cause performance bottlenecks.

Action Items:
- Kamil to document all areas under investigation and to share resources with Besu team.

References:
- [Besu Benchmarking approach](https://github.com/hyperledger/besu/blob/main/BENCHMARKING.md)
- [Grafana benchmarking results](https://grafana.observability.ethpandaops.io/d/feo4ronhsqv40h/opcodes-benchmarking-github-action?orgId=1&from=2025-08-07T17:09:45.517Z&to=2025-08-07T19:29:44.050Z&timezone=browser&var-posgreSQL=benuragv7iuwwb&var-ClientName=$__all&var-TestTitle=$__all&var-BenchmarkClient=besu&refresh=auto)

### Other Updates
- No additional concerns raised by other client teams.
- Marius: Called for more benchmarking, especially regarding state.
- Marius (chat): All clients failed snap sync on perf/devnet4.
- Barnabas (chat): Shared failed test run – [link](https://github.com/ethpandaops/perf-devnets/actions/runs/16961730639).
- Mario: New EEST tests being added to catch potential bugs.
- Kamil: ecrecover performing well at 60M gas for all clients except Besu.
- Encouraged all clients to align on EEST benchmarking instead of developing separate tools.
- Ben: Reported Nethermind has no issues with 60M gas.
- Louis (chat): Shared EEST benchmark reference for ecrecover.

## [PR #4508](https://github.com/ethereum/consensus-specs/pull/4508) – Ready for Merge
- Barnabus brought up in the call
- CL devs are requested to review.
- If no objections, we should move forward with merging.
- Alex Stokes: “Seems fine to me.”
- Pawan (chat): “Looks good to me.”

Others: Please review, comment, and approve the PR.

## Sunnyside Labs Testnet Updates
- Minhyuk: No major updates this week, aside from the [report](https://testinprod.notion.site/Sunnyside-Devnet-Updates-08-13-24e8fc57f54680088514eccdca62c560) shared last week.

## Safe-Head Discussion
- [All Core Devs - Testing (ACDT) #45 | July 21, 2025 #1624 (comment)](https://github.com/ethereum/pm/issues/1624#issuecomment-3088955383)
• Mikhail raised the topic last week and added it for follow-up this week.
• No additional thoughts or discussion today.

**Conclusion** (Gas Benchmarkin): Review benchmark results and continue discussion in next week’s ACDT.

# ACD Testing Call #48 - Aug 11, 2025
Facilitator: Parithosh Jayanthi

## Fusaka Update
Parithosh shared updates:
- Devnet-3 has been running for 10 days, with two non-finality experiments conducted last week.
- **Experiment 1**: Took out full nodes → ~1 hour of non-finality before recovery in the following epoch.
- **Experiment 2**: Subset of supernodes removed → ~half a day of non-finality, but finality regained within 20–30 minutes, showing rapid recovery.
- Results analysis showed no major issues; bandwidth limits appear fine and overall network health is good.
### Next step:
Test with a subset of nodes (especially supernodes) capped at 1–2 Mbps bandwidth to simulate unreliable supernode participation. This is planned for next week.

## Private Mempool Update
- Bharat shared that there’s a way to directly send transactions to builders, which is useful for private mempool testing.

### Next step:
- Pari will set up spam testing for the private mempool.
- Once the force transition issue is resolved, Bharat will focus more on spamming tests.

## More on Devnets
### Devnet 3
Paris shared more updates:
- A fork of Prysm with a hook feature (allows changes) has been live on the network for 2 hours with no issues reported.
- Hooks can be extended for more scenarios — share your ideas [here]( https://discord.com/channels/595666850260713488/892088344438255616/1404441330062463086).

### Devnet 4 Setup
- Setup coordinated with Sunnyside Labs, with additional testing planned [here](https://discord.com/channels/595666850260713488/892088344438255616/1402650822285856788).
- Network is running fine; next BPO is scheduled in the next 2 hours to assess performance.
- [Grafana dashboard](https://grafana.observability.ethpandaops.io/login) available for monitoring.
- Nimbus encountered issues — Dustin is investigating, some details shared by Pari on the call.
- Pari clarified to Pawan that Devnet 4 has no non-finality tests planned but could add them if needed.
- Pawan suggested exploring client behavior under longer finality delays.
### Next step:
- Parithosh will arrange a smaller devnet at the end of Devnet 4.

## Holesky Release & Client Timeline
Parithosh shared updates
- Community feedback indicates no major issues with ad-hoc client releases for Holesky validators.
- James He relayed Preston’s note: "Releasing to holesky off develop isn't difficult, but it messes with normal flow for releases".

### Next step:
- will be brought to the next ACD for further discussion.

Pari:
- One EIP-related PR pending: [ethereum/execution-apis#678](https://github.com/ethereum/execution-apis/pull/678) — comments welcome before merging.

## Gas Limit Testing Updates
- Progress can be tracked here: [Syncoor Dashboard](https://syncoor.fusaka-devnet-3.ethpandaops.io/)
- Mario shared the plan of another benchmark release this week.
- Further on updates, Mario shared that With the updated tool, the Nethermind team can pull the genesis, run benchmarks, and report results.
- Plans to extend benchmarking with another release expected soon.

## Glamsterdam Updates
**BALs**
- No BALs devnet yet, though specs are ready and built on top of Fusaka.

### Next Steps
- Toni & Pari will coordinate on setting up a devnet, possibly using Kurtosis.
- Further discussion planned for ACD on Thursday.

**ePBS**
- Local testing is available via a local branch.
- Ansgar: No strong value in combining ePBS, BALs, and repricing in the same devnet.

### Next Steps
- Pari invited feedback on preferred Fusaka devnet setup.
- Justin Traglia: Plans to merge consensus-specs PR #4476 soon.

## Sunnyside Labs Update
- No major updates today; another report expected tomorrow.

On Gas repricing
- Toni: Highlighted EIP-7904, EIP-7778, EIP-7981, and multidimensional gas EIPs for consideration.
- Pari: Asked if there’s a Meta EIP for gas pricing.
- Ansgar: Confirmed they can create one.
- Pari: Said a Meta EIP would be more useful than referencing individual EIPs.

## Consensus Issues Found with Fusaka
- Marius raised consensus-related concerns for discussion.
- Mario: For EIP-7883, an update will be included in the next EEST release.
- Justin Traglia shared Related PR (in chat): ethereum/execution-spec-tests#1993
- Reth: Issues explained by Dragan are now resolved.
- Marius mentioned: Two issues found for ModExp gas cost, but they are unrelated.
- Pari asked if test coverage is sufficient.
- Mario: Extra coverage would be good; will prepare another devnet release for testing.
- Related PR: ethereum/execution-spec-tests#2005.

## EIP-7825 Discussion
pk910: Asked if the transaction gas limit should also apply to eth_call and which clients have implemented it. Suggested directly adding the limit for checking.
- Marius(in chat): In Geth, the limit is configurable.
- Pari: Agreed to leave it as is for now and revisit the discussion later.

# ACD Testing Call #47 - Aug 04, 2025
Facilitator: Mario Vega

## TL;DR
- Fusaka Devnet 3
- Nimbus and Lodestar are still being debugged; fixes are underway.
- Decision: Wait until both are stable before advancing.
- EIP-7825 Gas Limit
- Proposal to raise to 30M discussed; concern over app impact.
- Decision: Maintain current 16M gas limit on Devnet 3.
- Gas Limit Testing
- Stateful and large contract tests in progress; repo flag awaited.
- ModExp pricing PoC tool being developed; Geth change not expected soon.
- EIP-7951 (secp256r1) Benchmarking
- Besu results show high cost.
- Action: Mario to follow up with Besu team asynchronously.
- INTERVALS_PER_SLOT Replacement (EIP-4476)
- Adds slot timing config without protocol changes.
- Decision: No breaking changes; clients should begin prepping for implementation.
- PR-4477 Proposal
- Dev's consensus that it’s too late for Fusaka.
- Decision: Do not proceed; no Devnet 4 for this. Plan shadow fork next.
- Sunnyside Devnet / PeerDAS Syncing
- Sync failures in perfect PeerDAS; investigation ongoing.
- Decision: Feature required, clients to report progress.
- Backfill Implementation
- Prysm, Lighthouse, and Lodestar in progress.
- Decision: Next ACD to revisit Devnet 4 for backfill.

## Fusaka Devnet 3
Barnabas:
- Devnet 3 is mostly stable with ~90% participation.
- All 5 Blob Parameter Only (BPO) forks are activated and running smoothly.
- Most prior issues have been resolved.
- However, 2 consensus clients — Nimbus and Lodestar — are still facing issues.
- Caplin also involved in ongoing testing.

Lodestar:
- Matthew K is investigating genesis value sync and working on resolving the bug.

Nimbus:
- Update relayed by Agnish:
- Issue with the custody column related to hash functions in indices.
- EF test vectors are passing.
- EAS (Earliest Available Slot) was not moving — now fixed.
- General updates: MEV issues were reported and have been addressed.
- Issue raised on Eth R&D Discord - Nimbus and Geth having difficulty syncing (likely a peering issue).
- DAS Guardian fix expected within the next few days.

Bharath:
- Local transaction propagation was disabled on the RETH node, which caused transactions not to be streamed.
- Fixes of issues shared on the call.
- Also, under discussion with the RETH team to consider removing local transaction propagation entirely.
- Some complications related to spam filtering reported, and the root cause of transaction exclusion has now been identified.

Roman (RETH): Inquired about the RETH flag and its specific issue.

Bharath: Shared technical context also shared on Eth R&D Discord.

Nethermind (Marek):
- Nethermind worked with rbuilder as state provider on integration with state
- The state sync component is ready, but the EVM integration is still in progress.
- Additional testing is requested before full readiness.
- More details: [Flashbots Multi-client Support Announcement](https://collective.flashbots.net/t/announcing-multi-client-support-for-rbuilder/4895)

### Conclusion & Next Steps

Mario:
- Overall, Devnet 3 is progressing well.

Barnabas:
- Aiming to ensure Nimbus and Lodestar are bug-free before moving forward.

## Raising the Limit of EIP-7825 to 30 Million

Mario:
- Provided an update from last Thursday’s ACD discussion.
- Discussion link on [Ethereum Magicians](https://ethereum-magicians.org/t/eip-7987-transaction-gas-limit-cap-at-2-24/24746/21).
- Raised concerns that raising the gas limit may negatively impact certain applications.

Toni:
- Stated that tradeoffs favor keeping the current spec as implemented in Devnet 3.

### Decision:
- Maintain the gas limit at 16 million for now.

## Gas Limit Testing Updates

Barnabas:
- No major updates.

Kamil:
- Still working toward full state testing.
- Marcin is running additional tests and benchmarking.
- EEST is running on existing infrastructure for stateful testing.
- A test is live on a contract with a very large state size; waiting on repo to receive the necessary flag.
- Some tweaks are needed on Devnet.
- BloatNet is being prepared; results expected this week.
- Will continue iterating and updating.

Marcin:
- Developing a new tool to evaluate ModExp pricing.
- Differences in gas usage are expected; further investigation underway.
- Currently working on a PoC — actual testing still pending.

Parithosh (in chat):
- Geth team noted that upstreaming changes would take time, if at all. Not a short-term solution.

Mario:
- No Geth representative was present to elaborate further.

Louis:
- Related issue: execution-spec-tests#1976
- Related PR for configuring gas limits: [PR #1983](https://github.com/ethereum/execution-spec-tests/pull/1983)

Kamil:
- 60M gas limit update still requires more testing.
- Genesis file handling and stateful test considerations are ongoing.

## Besu’s EIP-7951 Benchmarking Results

Justin (Besu): Updates shared in [Comment #1648](https://github.com/ethereum/pm/issues/1648#issuecomment-3149294281).

Mario:
- Initial results suggest the cost is too high. Will follow up asynchronously with the Besu dev team.

## Replace INTERVALS_PER_SLOT with Explicit Slot Component Times

- [consensus-specs#4476](https://github.com/ethereum/consensus-specs/pull/4476)

Justin Traglia:
- Introduced the PR to replace INTERVALS_PER_SLOT with explicit slot component times.
- Goal: Raise awareness now and propose merging within the next 3 months to begin implementation.
- This proposal adds additional config values; no substantial changes to protocol behavior.
- Believes there’s no reason to delay merging sooner.

Mario:
- Asked whether this introduces any breaking changes for Fusaka.

Justin T.:
- No breaking changes expected.
- Clients will need to update configurations, but should not impact Fusaka.

Pawan:
- Initial review looks good.
- Will revisit the [PR](https://github.com/ethereum/consensus-specs/pull/4476) for a deeper look.

## PR-4477 Discussion & Decision
- Barnabas shared and requested decison on [PR](https://github.com/ethereum/consensus-specs/pull/4477).

Feedback from devs on call & chat:
- Justin T.: Against including this change.
- Alex S.: Suggests leaving it for now.
- Pawan: Opposes the proposal.
- Barnabas: Notes that the proposal has received support from some CL devs (Potuz, Terence, nflaig).
- Toni W.: Feels it’s already too late to consider this change.
- Csaba K.: Also against; if needed, should be done with a future BPO.
- Barnabas: Opposes bundling this change with BPO.
- Mario: Including this would require an entirely new fork.
- Pari: Suggests deferring it to Glamsterdam.
- Raul: Reminds that BPOs were explicitly designed to avoid code changes.
- Agnish: Opposes inclusion; the proposal is too tricky and more data is needed from dynamic DA conditions with supernodes.
- nflaig (Nico): Agrees it’s too late for Fusaka.

### Decision & Next step
- This [PR](https://github.com/ethereum/consensus-specs/pull/4477) will not move forward.
- Devnet 4 will not be launched.
- Focus will remain on stabilizing Nimbus and Lodestar before further progression.
- A **shadow fork** is being planned for next week. Barnabas encourages client teams to have established branches ready ahead of the shadow fork.

## Sunnyside Update

Update on [New Report](https://testinprod.notion.site/Sunnyside-Devnet-Updates-08-04-2458fc57f546808ab2c9e34480e0b7a9?source=copy_link):
- Successfully reached 60 blobs per block; observed impact on usage.
- Hit limits as expected per hardware requirements in EIP.
- Encountered genesis sync failures in perfect PeerDAS network setup.

Devs comments (from call & chat):

Manu:
- Provided explanation for Prysm issues on perfect PeerDAS.
- Actively working on a fix; expects resolution by end of the week.

Agnish:
- Requested clarity on the importance of syncing in perfect PeerDAS mode.

Pari:
- Clarified that Fusaka will not support PeerDAS with this in its initial release.
- This feature can be evaluated later, and if necessary, included in a future BPO.

Agnish agreed with this direction.

Mario:
- Agreed Perfect PeerDAS support is not required for Fusaka.
- May be revisited for future upgrades.

Agnish:
- Shared that reconstruction-based syncing offers more guarantees and would prefer that over relying on dynamic peer discovery.
- May need this feature.

Mario:
- Acknowledged that this feature might become necessary.
- Requested CL teams to confirm timelines for implementation as it may be a potential blocker for Fusaka.

Pawan:
- Currently does not have it implemented but is aware of the mechanism and will assess how long it would take to develop.

### Decision:

Mario:
- This is important enough to be tracked closely.
- Requested the rest of the CL team to respond.
- Will raise the topic again on the next ACD call for broader CL input.

Note:
Sunnyside may spin up additional testnets to support further PeerDAS testing.

## Backfill

Barnabas:
- Asked if any client implementations for backfill are in progress.

Manu:
- Prysm implementation is in progress.

Other Clients:
- Lighthouse and Lodestar are actively working on it.

Devnet 4 for Backfill?

Barnabas:
- Proposed the idea of a Devnet 4 focused on backfill testing.
- Suggested waiting for confirmation on ACD once client implementations are ready.

Backfill Implementation Challenges

Agnish:
- CPU usage is significant, depending on how quickly backfilling is expected.

Question raised: Should nodes provide slot-level data granularity while backfill is in progress?

Pawan:
* Mentioned they had to change the database structure to make Earliest Available Slot (EAS) accessible.
* Average backfill size observed: ~1 epoch.
* For their implementation, this is not a blocker.

### Conclusion:
- Awaiting updates on perfect PeerDAS syncing before proceeding further.

# ACD Testing Call #46 - July 28, 2025

Moderator: Parithosh Jayanthi

### Next Steps
* Devnet 3
* Network is live.
* Launch BPO-4 tomorrow to reduce blob count.
* Continue investigation into Nimbus peering and orphan block issues (Pari, Bharath).
* Barnabas to share async notes.
* Eth Config
* Danno to update PR removing JSON hash from eth_config.
* Gas Limit Testing
* Alexey to analyze secp256r1 gas pricing.
* Teams to report peer-related anomalies; Marius to review Geth behavior with PR logs.
* Investigate Nethermind peer discovery issue (Csaba, Raul).
* BloatNet
* Geth team to finish tracing/memory fix.
* Client teams to review Cperezz’s test plans and align with EELS integration.
* Benchmarking
* Mario to finalize EEST benchmark tests and resolve blockers.
* Sunnyside Labs
* Continue large block size propagation tests.
* Share final report on Discord once Teku backfill issues are resolved.

## Fusaka Devnet 3 Updates
Parithosh shared:
* Devnet 3 launched late last week.
* Network is finalizing with 3 BPOs already completed.
* Live tracking: [Dora Explorer](https://dora.fusaka-devnet-3.ethpandaops.io/)
* Blob Configuration**
* Currently running at 18 blobs per block.
* BPO-4 will reduce blob count (currently at 18 blobs/block).
* BPO-4 expected to go live tomorrow.
* Added rate limiters:
* 30% nodes are gigabit “super nodes”
* Rest are 100 Mbps down / 50 Mbps up
* Based on [EIP-7870](https://eips.ethereum.org/EIPS/eip-7870)
* Validator Key distribution is uneven; staking top-ups added for addditional test coverage.
* Network includes every client pair.
* Top-ups used to normalize validator stake across the board.
* Clients experiencing:
* Nimbus EL/CL unable to connect to bootnodes
* Higher-than-expected orphan blocks (no pattern, possibly MEV-related)
* MEV updates (Bharath):
- 672 validators have successfully registered for MEV.
- Block delivery confirmed from the **latest slot**.
- Investigation ongoing into **node connectivity issues**:
- Initial findings point to a **problem with Lodestar**.
- Block publishing for Nimbus is currently failing
- other than these two issues, **regular blocks are being delivered consistently**.
- [MEV Relay Dashboard](http://mev-relay-1.fusaka-devnet-3.ethpandaops.io:9060/)

**Other discussions**
- Manu: Do all nodes have the same rate limiting, or does it depend on the node type?
- Pari: It depends. Full nodes are capped at 100 Mbps, while super nodes have more. All specs are per EIP-7870. If those limits prove insufficient, we’ll flag and evaluate adjustments.
- Pari: Nimbus EL + CL clients are facing peering issues, unable to connect to bootnodes. Bharath will investigate this asynchronously.
- Pari: There’s a **higher-than-expected rate of orphan blocks.** No consistent client is at fault — random distribution, no clear pattern. The only major change from previous devnets is the presence of MEV.
There was an issue where aclient block tried to build on an old parent. Root cause is still unclear. [Block link](https://dora.fusaka-devnet-3.ethpandaops.io/slot/0xd9ee087ddce01ccac3c09ee960ae2493010e56c4457ba3c247e3b64c713a2193)

### **Next steps**:
- Bhartah will continue to investigate Nimbus and other issues.
- Barnabus will share his notes async.

## Eth Config Discussion
- Mario
- Noted that **`eth_config` is now available**.
- Testnet and mainnet configurations need to be reviewed.
- EEST (Execution Engine Spec Tests) should understand the config; queries can be placed during the testing phase.
- Pari: Asked if **RLP serialization is in progress**.
- Marius
- Raised concerns about **hashing JSON-RPC responses**.
- If all clients return the same JSON-RPC result, it can become brittle.
- Suggested we **find a better way to hash** the data.
- Prefers **RLP**, but open to **SSZ** if other clients prefer it.
- Danno
- Questioned use of **RLP**: "If we shouldn’t use RLP, why go that route?"
- Later suggested **dropping the hash entirely**.
- Tim Asked: “Why are we not supposed to use RLP?”
- Roman: Clarified that the **Execution Layer (EL)** currently **only supports RLP**, not SSZ.
- Said dropping the hash might complicate things for **testing teams**.
- Mario Vega: Suggested testing teams can **perform the hashing themselves** if needed.
- Referenced EIP-specified JSON standardization: [RFC 8785](https://www.rfc-editor.org/rfc/rfc8785).
- Personally **liked the idea of including the hash**.
- Danno: Mentioned another change
- Pari: Confirmed it seems like a similar change; **no objections** from the group.
- **Danno will include the change** in the updated PR.

### Decision:
- **Drop the hash** from the config.
- **Danno** will update the PR accordingly.

## Next Phase Testing - Timeline Discussion
Pari mentioned
- Shadow fork and large testnet are in planning.
- Reference doc: [Fusaka Devnet 3 Notes](https://notes.ethereum.org/@ethpandaops/fusaka-devnet-3)
- Alex stokes: Urged to **begin the next phase of testing as soon as possible**.
- Pari: Emphasized that client teams should **focus on orphan block issues** and other topics discussed earlier in the call.

## Gas Limit Testing Updates
- Pari updated
- Mainnet is currently running at **45 million gas** with **no major issues noted**.
- Investigation into **Consensus Layer (CL) validator behavior** is underway, and analysis will be shared soon.
- Alexey
- Plan to **check the real gas cost of `secp256r1`** this week.
- Expressed doubt about time availability to update the pricing if anything significant is found.
- Asked if **other clients** observe similar behavior; requested that findings be **reported**.

**Peering Issues**
- Marius van der Wijden
- Geth team is also looking into the issue.
- Shared a **theory**: a **Geth-specific peering optimization** might be causing issues for other clients.
- Shared a PR for adding **debug logging**:
- [go-ethereum PR #32287 – Add more debug logs](https://github.com/ethereum/go-ethereum/pull/32287/files)
- Csaba
- Highlighted a **peer discovery issue** involving the Nethermind client.
- Problem **appeared about 5 days ago**.
- This testing has only been live on mainnet for less than a month, so needs further understanding.
- Raul
- Suggested sharing **peer IDs** to investigate potential common patterns.
- Pari
- Both **Nethermind** and **Geth** issues appear to be **unrelated** to the **45M gas limit** setting itself.

## BloatNet Testing Discussion
- Cperezz
- Shared context and resources:
- [BloatNet HackMD 1](https://hackmd.io/WUe1KgwSS3iT8xoIbDg3Ng)
- [BloatNet HackMD 2](https://hackmd.io/9icZeLN7R0Sk5mIjKlZAHQ?both)
- [Go-Ethereum Issue #32290](https://github.com/ethereum/go-ethereum/issues/32290)
- Plans to expand testing with support from additional teams.
- Requested **review from client teams** to grow BloatNet testing coverage.
- Raised the question: **Can we align with the EELS team** on state-growth-related tests?
- [Related EEST GitHub Issue #1923](https://github.com/ethereum/execution-spec-tests/issues/1923)
- Noted: EELS will be integrated with spammer for better testing.
- Marius van der Wijden
- Reported a serious **issue on BloatNet with Geth**:
- When Geth detects a state change, it tries to dump **all states**.
- This behavior **consumes a lot of memory**, causing **nodes to crash**.
- Geth had **55 node crashes**, and the **database got corrupted**.
- **Root cause**: nodes **ran out of memory** during tracing.
- Fixes implemented:
- Working on tracing with **less memory usage**.
- Shared infrastructure resource usage:
> "We allocated ~10TB because of tracing and <1TB of RAM due to other chain operations."
- Pari
- Confirmed **resyncing the node** from the corrupted database has been started.
- Will **investigate tracing issues** further.
- Mario
- Commented on the **limitations of state growth testing**:
- Tests produced cannot contain **very large state** due to **technical constraints**.
- Most current tests are computationally intensive rather than focused on state growth.
- Highlighted the need for **state-specific benchmarking**.
- Cperezz (follow-up)
- Emphasized that the new tests are **more state-related**.
- Ideally, such tests should be executed using **BloatNet state**.
- Marius van der Wijden
> "It would be ideal to write the tests in a way so they can be replayed on arbitrary networks."
- Pari
- Shared that the **Nethermind team has started EELS integration**.
- Suggests continued asynchronous coordination on integration efforts.

## EEST Integration, Benchmarking, and Testing Reports

- Mario
- Announced a new release: [EEST Benchmark v0.0.3](https://github.com/ethereum/execution-spec-tests/releases/tag/benchmark%40v0.0.3), which includes **all benchmark tests**.
- Noted that Nethermind tools can now use genesis files.
- Stated that **genesis will be the starting point** for future benchmarks.
- Shared that this is still a work in progress.
- **Blocker:** All benchmarks still need to be functional and verified.

## CLZ Benchmark Results

- Louis
- Presented findings via [this slide deck](https://docs.google.com/presentation/d/1jjLM4cWQghMpiyPcQzZ_3di-Euz65idgZ45BUPndKBs/edit#slide=id.g373c3cf0dc8_0_69).
- Shared methodology and test notes.
- Tests based on **EIP-7825** and focus on **worst-case block sizes**.
- Two gas limit scenarios tested: **72M and 100M**.
- Call participants were generally supportive of the **CLZ benchmark methodology**.
- Pari
- On reducing gas pricing, he recommended **sticking to 5**.
- Mentioned the intent to **reduce pricing** in the upcoming **Glamsterdam** fork.
- Mario
- Noted the benchmarking methodology:
> “Use EELS first, then test across client implementations in future forks.”

## Sunnyside Labs Testing Update
- Minhyuk (Sunnyside Labs)
- Provided a progress update:
- Running **two testnets** with **regular transactions** to monitor how **larger block sizes affect network data propagation**.
- Began test execution.
- **Teku** is currently the only client supporting **backfill**, though some issues remain.
- Team is working on resolving backfill issues and **aiding with broader test coverage**.
- Report will be **shared on Discord** once ready.
- Confirmed that the **Perfect Column Devnet** appears to be **functioning correctly at first glance**.

# ACD Testing Call #45 - July 21, 2025

## Fusaka Devnet Updates
**Mario:**
- Devnet 3 not gone live as of now

**Spenser:**
- EEST feature release for Devnet 3 includes:
- Modexp gas repricing updates
- CLZ opcode changes (will be discussed today)
- Blob base fee and gas reduction
- New test added for PeerDAS and [EIP-7907](https://eips.ethereum.org/EIPS/eip-7907) testing

**Pari:**
- Two items pending before launch (Builder spec & Engine API v2 PR [#123](https://github.com/ethereum/builder-specs/pull/123))
- Kurtosis testing in progress
- If unblocked, Devnet 3 release expected this week

**Client Status:**
- Besu, Geth (except config endpoint), Nethermind, Reth are ready
- Lighthouse, Grandine, Lodestar (pending builder API) are ready

#### Conclusion:
- No major blockers from clients side is anticipated for next 2 days.

## Gas Limit Testing Updates

**Pari:**
- 45M gas limit enabled across all clients on mainnet - Congratulations!
- Perf Devnet 2 was worked on the last weekend.
- New sync testing tool is available
- New RPC testing tool in development, team will reach out to RPC providers.

**Kamil:**
- Opcode benchmarking is WIP.
- Connecting with EEST team on full integration of opcode becnhmarking.
- Computation benchmarks mostly complete.

**Marcin Sobczak:**
- [EIP-7883 Comprehensive ModExp Analysis](https://github.com/nerolation/eth-7883-analysis/blob/master/eip7883_comprehensive_analysis.md)
- [Entity Impact Report](https://github.com/nerolation/eth-7883-analysis/blob/master/eip7883_entity_analysis.md)
- Ask is for client teams to take a look.

## BloatNet Data Collection

**CPrezz:**
- BloatNet is approaching **2x the size of mainnet**.
- Ongoing **compaction testing** aims to help evaluate safety for a potential **100M gas limit**.
- Several metrics have been implemented, though some are still missing.
- **Sync data is missing from Besu and Erigon.**
- **Grafana integration** is challenging, especially due to difficulties extracting metrics from logs for Nethermind and others.
- Shared reference: [BloatNet Metrics & Progress – HackMD](https://hackmd.io/boZCC4eNR66zN3m9VDYFWQ)
- Request to client teams: reach out if assistance is needed.

**Kamil Chodoła:**
- Will report additional metrics and is working to include more.
- Reminder: **All changes to the `performance` branch** are automatically used by:
- Opcode benchmarking tools
- Perfnet
- BloatNet
- Ensure `performance` branches are updated regularly with experimental changes and rebased on `master`.

**Amezaine (Besu):**
- Besu does have some metrics for sync and compaction.
- Needs to review implementation to confirm correctness and ensure coverage.

**Milen (Erigon):**
- Has been helpful in providing data and coordination.

**Additional Notes (CPrezz):**
- For compaction metrics, timing must cover **insertion and removal operations** in detail.

**Pari:**
- Emphasized the importance of keeping `performance` branches up to date across all clients.

## Engine API: Fast Confirmed Safe Block Discussion
- [Discussion Link](https://github.com/ethereum/pm/issues/1624#issuecomment-3088955383)

**Mikhail Kalinin** initiated a conversation about incorporating a **fast confirmed block** reference in the Engine API. The goal is to enhance safety signals shared between clients and external consumers (like exchanges and L2s).

### Key Questions Raised:
- How is the `safe block` currently used by EL (Execution Layer) clients?
- How do consumers like exchanges and rollups use it via the JSON-RPC API?
- Should the `safe block` field be **overloaded** to include the fast confirmed block (currently mapped to `justified`)?
- Or should a **new field** such as `fast_confirmed` be introduced?
- How difficult would it be to add this from an engineering perspective?

**Mikhail Kalinin**
- There's an open PR and ongoing experimentation with a slightly modified implementation.
- Historically, `safeBlock` was meant to serve as the confirmed block, but **no robust algorithm** exists today to guarantee this.
- Asks whether introducing a new tag in the Engine API is a better long-term solution.

**Marius van der Wijden**
- Suggests that `safeHead` should be the default for most applications currently relying on `finalized`.
- Notes underutilization of `safeHead` by clients.

**Terence (Prysm)**
- Points to consecutive slot disparity as a source of discrepancy.
- Prysm includes a flag that can expose this state to users.

**Further Considerations**
- Marius emphasized that **no major EL changes are needed** if CL (Consensus Layer) already marks blocks as safe.
- Shared if RPC providers and infrastructure operators running multiple clients: discrepancies in safe block may already be there.
- Noted that the current definition of `safeBlockHash` is **vague enough** to permit flexibility:
> `"safeBlockHash": DATA, 32 Bytes – the "safe" block hash of the canonical chain under certain synchrony and honesty assumptions. This value MUST be either equal to or an ancestor of headBlockHash.`

#### Conclusion
- Continue discussion in **Ethereum R&D Discord**.
- Bring the topic back to the **next All Core Devs (ACD)** call for wider input.

## Builder Response Discussion

**Bharath-123** proposed to [discuss](https://github.com/ethereum/pm/issues/1624#issuecomment-3095192635):

### [Deprecate JSON payloads in favor of SSZ](https://github.com/bharath-123/builder-specs/pull/1)
- Suggests **deprecating JSON-formatted requests and responses** in favor of SSZ encoding.
- This update was raised to gather any final feedback or objections from client teams.

**Parithosh Jayanthi**
- Asked for final input from the group on PR [#123](https://github.com/ethereum/builder-specs/pull/123):
> "Can we merge it? Or any open discussion topics?"

**Bharath**
- Confirmed **client developer consensus** exists.
- If no objections are raised, this update can be **targeted for Devnet 3**.

### Decision

**Alex Stokes**
- Supports the proposal:
> "We should merge it."
- Notes that **work is already underway on MEV-Boost** to support the changes.

The update is cleared to proceed and will be part of Devnet 3 scope unless new concerns are raised.

## `getBlobsV3` Discussion

The proposal to introduce `getBlobsV3` was evaluated during ACD Testing Call #45. The discussion centered on whether it should be implemented now for future use or deferred due to testing and coordination concerns.

**Dustin**
- Suggests that if EL clients ship `getBlobsV3` now, it could be retroactively validated through testing later.
- Notes it wouldn't be trusted immediately but might become trusted if supporting tests are run in the future.

**Raúl Kripalani**
- Clarified that from an optimization perspective, the change would be purely on the **Consensus Layer (CL)**.
- `getBlobsV3` is an **extension of the gossip subprotocol**, already included in the `master` spec.
- No spec changes on CL are necessary at this point.
- Noted that CL clients are still using `getBlobsV2` and **have not yet implemented v3**.

Some of the other client devs on call were not in favor.

**Mario**
- Raised concerns about the contentious nature of the proposal.
- Suggested deferring the discussion to an **ACDE meeting**.

**Parithosh**
- Flagged that there's a **BPO (Blobstream Precompile Opportunity)** upgrade between Fusaka and Glamsterdam.
- This sequencing should be considered when deciding the scope of Devnet testing.

**Mario**
- Expressed uncertainty about having enough time to properly test `getBlobsV3` in **Devnet 3**.

#### Decisions

- **Do not include `getBlobsV3`** for now.
- **Francesco**: Recommended going forward with `getBlobsV2` only, as current CL implementations support that version.
- **Raúl Kripalani**:
- Closed [PR #671](https://github.com/ethereum/execution-apis/pull/671) which proposed `getBlobsV3`.
- Opened a new PR to make **`getBlobsV2` all-or-nothing**, so it's uniformly adopted by all clients.

## `CLZ` Opcode Gas Cost Discussion
The gas cost of the `CLZ` (Count Leading Zeros) opcode was increased from 3 to 5. This change prompted discussion regarding whether it was justified or should be reverted based on updated benchmarking data.

**Dragan Rakita**
- Raised concerns over bumping `CLZ` from 3 to 5 gas.
- Cited comparative analysis of similar opcodes like `ISZERO` and `MUL`, which suggests a cost of 3 would be more accurate.
- Shared benchmarking data in [Discord](https://discord.com/channels/595666850260713488/745077610685661265/1396813811381440512).
- Referenced [Rust benchmark comparison in `bluealloy/revm#2744`](https://github.com/bluealloy/revm/pull/2744) showing comparable performance among opcodes.
- Expressed willingness to reprice the opcode if consensus emerges.

**Mario**
- A rollback is complicated since the fork is close to finalization.
- Warned against hasty changes without broader client benchmarking data.
- Noted EEST is currently working on a more comprehensive benchmark, but it's not yet available.

**Marius van der Wijden**
- Also hesitant to change the gas price without robust benchmarks.
- Shared benchmarking results for Geth:
- `opAdd`: 8ns
- `opCLZ`: 48ns
Indicating that `CLZ` may be costlier than initially assumed.

**Dragan Rakita**
- Emphasized that benchmarking can vary by implementation.
- His team's data shows `CLZ` might not even justify a gas cost of 3.

**Mario (on benchmarking status)**
- Not all clients have published reliable benchmarks yet.
- Encouraged finishing benchmarking work before making changes.

#### Decision

- **No rollback** of the `CLZ` gas cost at this time.
- Gas cost remains **5** for **Devnet 3**.
- Reconsider for **Glamsterdam** based on comprehensive benchmarking.
- General leaning is to **adjust gas upward** rather than downward if uncertainty exists.

> _“Let’s keep it at 5 for now. Benchmark more thoroughly, revisit for Glamsterdam if needed.”_ — Mario

## 60M Gas Limit Discussion

Raised by: **Kamil Chodoła**

- Current blocks are already reaching **45M gas**, so **60M** is a foreseeable next step.
- Proposal to start thinking about **re-pricing and scaling efforts** to support 60M blocks, potentially even **before Fusaka**.
- Prompted a **temperature check** among client teams:
- Should we proactively work toward **60M gas blocks**?
- Known areas for improvement include **ModExp** performance.

**Marius van der Wijden**
- Noted that with **refund mechanisms**, actual gas pressure can be **reduced by ~20%**.
- Requested more data to understand the **impact on state growth** from scaling up to 60M gas.
- Hopes the **BloatNet** initiative can provide valuable insights.

> _"I would also like more data on state growth for 60M, hope that BloatNet can help us there."_ — Marius

**CPerezz**
- Urged for a **step-by-step** approach:
- First goal is achieving **2x mainnet state** on BloatNet.
- Working on collecting **EL client performance data** to model a wide range of test cases.

> _"One step at a time! 🙂 Getting to 2x mainnet for now and gathering EL client’s info to come up with as many cases as possible."_ — CPerezz

**Ben**
- Highlighted that **receipt data** remains a **bottleneck** in scaling up to **100M gas**.

**Kamil**
- Framed the discussion as exploratory:
- Aimed to **spark reflection** and gather feedback.
- Will revisit the topic **in the following week**.
- Believes 60M is **feasible before Fusaka** if work continues as planned.

**Mario Havel**
- Welcomed the discussion.
- Emphasized **no immediate decision is needed** but encouraged teams to keep the topic in mind.

> _"Nice to be brought up. No need for a decision now."_ — Mario

### Follow-Up Items
- Merge builder-specs PR #123
- Continue safe block discussion on Discord
- Revisit CLZ gas in next testnet
- Monitor BloatNet metrics and performance benchmarks
- Prepare Devnet 3 as blockers resolve

# ACD Testing Call #44 - July 14, 2025

**Facilitator:** Parithosh Jayanthi
**Links:**
- [Fusaka Devnet 3 EIP PR #9981](https://github.com/ethereum/EIPs/pull/9981)
- [Sunnyside Labs Testing Update](https://testinprod.notion.site/Sunnyside-Devnet-Updates-07-14-2308fc57f5468054b18fcbff31cc032c?source=copy_link)
- [engine_getBlobsV3 PR #674](https://github.com/ethereum/execution-apis/pull/674)
- [GetPayload builder-specs PR #123](https://github.com/ethereum/builder-specs/pull/123)

## Fusaka Devnet 2 Updates
**Barnabas** provided an update:
- Reaching out to client teams
- No major bugs reported
- **eth-das-guardian** tool is now in active use

## Gas Limit: 45 Million
- **Nimbus** and **Lighthouse** have released specs with the 45M gas limit
- **Prysm**, **Lodestar**, and **Grandine** to release later this week
- Clients are aligning around **45M → up from 36M**

## Fusaka Devnet 3
- PR open: [EIP #9981](https://github.com/ethereum/EIPs/pull/9981)
- Awaiting broader approval — **Raul has approved**
- BPO EIP is no longer part of Devnet-3 but for the main fork
- **Next:** Merge remaining 2 PRs and set timeline post EIP-7907 decision

## EIP-7907: Raise Code Size Limit (Jochem's presentation)
Proposal to increase code size limit from **24 KiB to 48 KiB**

**Highlights:**
- Increases the contract code size limit introduced in EIP-170 and adds a gas metering to code loading
- *Warm vs. cold code state*
- Benchmarks at 100M to be collected and presented by ACDE. Client teams are to have an opinion about implementation difficulty for indexes as well as a timeline estimation by ACDE.

**Concerns raised:**
- Bigger witness sizes
- Gas cost implications
- Implicit `codeHash` behavior
- Risk of increased technical debt
- Likely to **break some existing contracts**

**Comments:**
- **Mario:** Underspecified
- **Marius:** Has concerns
- **Charles:** Large contracts may be more expensive
- **Pari:** Further discussion planned for ACDE
- **Pari:** If Fusaka is planned before Devconnect, the focus should be on hardening specs.

## Other Gas Limit Discussions
- **Two targets in EIPs:** 45M and 16M
- **Milen (Erigon):** Asked how large contracts deploy with 16M
- **Pari:** Will evaluate in the future
- 45M Gas target is part of **Devnet 3**

## Related PRs for Review
**Flagged by Barnabas:**
- [engine_getBlobsV3 (partial return)](https://github.com/ethereum/execution-apis/pull/674)
- [GetPayload update (remove execution payload/blob bundle)](https://github.com/ethereum/builder-specs/pull/123)

## Sunnyside Labs – Devnet Testing Update
- Ran across **all CL and EL clients**
- CL tested with **up to 60 blobs/block**
- All clients supported **72 blobs**
- Testing state is strong; recommend **spec hardening** next

# ACD Testing Call #39 – June 02, 2025

#### Summary
- **Gas Limit Increase to 60M Accepted:**
Client teams aligned on raising the gas limit to 60 million. Any increase beyond 60M should be gradual.
- **Fusaka Devnet 0 Stable:**
All clients successfully tested BPOs. Devnet 0 is looking promising. Sync testing and tool integration (eth-das-guardian) are next.
- **Fusaka Devnet 1 Launch on June 9:**
Will include implementation of EIP-7917 and Validator Custody. PR #4190 needs review from all client teams.
- **Validator Custody in Progress:**
Most clients are implementing or finalizing custody features. Targeting readiness by Devnet 1. PR #4320 reviewed as canonical reference.
- **Metrics Inclusion Approved for Devnet 1:**
Both CL and EL side metrics will be added. A tracking repo for metric changes will be created.
- **Column Sidecar Improvements Coming:**
Custom implementation in Teku; plan to propose changes for wider client support via PR.
- Next week’s Interop call is **cancelled**.

## Quick Notes

**Parithosh J** lead the call.

- [Retrospective post on EIP-7691](https://ethpandaops.io/posts/eip7691-retrospective/) is out
**TL;DR:**
- Estimations were accurate
- Bring further discussions to the ACDT issue

## Gas Limit Retrospective

**Pari:**
- Hoodi gas limit: 60 million gas
- Suggests slowly increasing limit in future.

**Barnabas:**
- Should we move in two phases? (e.g., signal to 48M first, then 60M)

**Kamil:**
- Decent percentage of validators already signaling 60M
- 60M seems reasonable
- Smaller incremental steps (e.g., 15–20M) are unlikely
- 60M is a no-brainer

**Barnabas:**
- Doubling gas limit is a significant change

**Ben:**
- 60M seems fine
- Above that, be more cautious

**Justin F.:**
- 48M step seems unnecessary

**Decision:** Client teams are okay with a move to 60M. Future increases should be gradual.

**Next Step:**
Focus on max CL size this week.

**Toni Wahrstätter:**
> If max CL size is 2.5 MiB, 65M gas may reach the 10min limit. 60M is safe.

## Fusaka Devnet 0 Status
[PeerDAS & Fusaka Devnet Updates](https://github.com/ethereum/pm/issues/1561#issuecomment-2927893242) by Will C.

**Pari:**
- PeerDAS testing channel and PeerDAS Devnet 7 will be deprecated
- Moving forward PeerDAS testing will happen on Fusaka testing stream
- Fusaka Devnet 0 is the new focus
- Communication moved to Fusaka Interop channel. Suggested to keep in thread.

**Barnabas:**
- Fusaka Devnet 0 started out with Genesis.
- Max blob cout at 9 on epoch 256.
- Increased the max blob cout to 12 with the first BPO and at the second BPO, it is increased to 15 than to 18 and then went down to 9. Will be increased to up to 20.
- All BPOs functioning well.
- All clinet successfully upgrading and downgrading
- No forks reported to BPO.

**Pari:** Devnet 0 is looking promising!

Note: ProbeLab team's tool - [eth-das-guardian](https://github.com/probe-lab/eth-das-guardian) is now open source.
will be integrated to testing stack. Clients Teams should check if their node is live.

**Next Steps (Barnabas):**
- Repeat Sync testing planned to check if clients are able to backfill.
- Building UI for integrating the new tool

## PeerDAS Validator Custody Updates

### Client's update
**Prysm (Manu):**
- Implemented validator custody
- Backfill still in progress

**Lighthouse (Pawan):**
- WIP, expected by end of week
- Backfill not yet implemented
- Tricky implementation wise, yet on track for interop

**Nimbus (Agnish):**
- No backfill required at Nimbus
- Validator custody can change within slot start

**Francis:**
- Encourages everyone to review the [PR Merged](https://github.com/ethereum/consensus-specs/pull/4320/files)
- Addresses concerns with backfill and custody changes

**Justin:**
- PR was made to address frequent custody changes

**Agnish:**
> For multi-Beacon Node setups, it's better to act as a "super node"

**Q&A:**
- **Pawan:** What endpoint does Prysm use to find attached validator node?
- **Manu:** Beacon proposer from API
- **Kasey:** Validator registration via Beacon API
- **Agnish:** Could non-enforcement of custody lead to network partitioning?

**Raul:**
> Some edge cases require deeper analysis – discussion to continue offline.

**Next Step:** Target validator custody for next week.
**Barnabas:** Will add PR to Devnet 0.

## Fusaka Devnet 1 Plans

**EIP-7917:**
- [PR #4190](https://github.com/ethereum/consensus-specs/pull/4190)
- Launch planned for **June 9, 2025**
[Fusaka Devnet 1 Notes](https://notes.ethereum.org/@ethpandaops/fusaka-devnet-1)

**Client Updates for EIP-7917:**
- **Justin:** PR open, clients should review
- **Prysm:** In progress
- **Lodestar (Phil):** In progress

**Decision:** Devnet 1 will focus on:
- Validator custody
- EIP-7917 implementation

## `getBlobsV2` Metrics (EL Side)
**Updates:**
- **Besu (Gabriel):** On track
- **Reth, Erigon, Nethermind:** On track
- **Geth:** Check async

## BPO Networking Updates

**Raul:**
- Adding a key to ENR to signal next upgrade
- EIP incoming, tweaks expected
- No network partition observed yet
- May delay until Devnet 2

**Justin T.:**
- Suggests keeping it for Devnet 2
- It will allow time to add more tests in Devnet 2 for BPO

Other links
- [EIP PR #9840](https://github.com/ethereum/EIPs/pull/9840)
- [Consensus Specs PR #4346](https://github.com/ethereum/consensus-specs/pull/4346)

## Metrics Implementation
**CL side metrics**
**Katya:**
- Added metrics; potential spec change needed
- Lodestar is testing
- PR still in progress

**EL side metrics**
**SunnySide (J):**
- [Discord Reference](https://discord.com/channels/595666850260713488/1252403418941624532/1374373828691497064)
- Reth, N/M, EthJS implemented original version
- Awaiting EL feedback

**Decision:**
Include metrics changes in Devnet 1 and create a repo to track them.

## SunnySide Labs Testing Update

[Update – May 30, 2025](https://testinprod.notion.site/Sunnyside-Devnet-Updates-05-30-Internal-2008fc57f54680ea81cde6acaa6d55fc)

- Ran 7 devnets
- Blob count more stable
- Focused on bottlenecks
- **LH:** Up to 53 blobs per block
- **Prysm:** Also high
- **Grandine:** Failed beyond 10 blobs due to old image

**Findings:**
- Grandine sometimes succeeded with retries
- Column processing and gas combos might affect `getBlobs`
- **LH:** Reported many column failures

**Next Step:**
J (Sunny side) will create thread on Interop channel for client by client analysis.

## Column Sidecar

**Dimitri:**
- Not available in Beacon API yet
- **Teku:** Custom implementation exists

**Pari:**
- Will propose PR to add this event

## Meeting Scheduling Notes
- Next week’s ACDT (Interop) call: **Cancelled**
- Client teams meeting in interop.

# Interop Testing Call #38 – May 26, 2025

**Call led by:** Mario Vega

## Summary
- **Fusaka-devnet-0**
- Devnet-0 to be launched tomorrow with Fulu activation following the next day.
- **BPO & Config**
- BPO is live on devnet-0, but the final version of the EIP is still being discussed and will go live in devnet-1 or devnet-2.
- Future devnets will align on genesis dates.
- **Devnet 1**
- **EIP-7918** confirmed for inclusion.
- **CL Spec PR #4323**
- Mixed client feedback on usefulness.
- Decision: Keep open; await further client input.
- **PeerDAS Testing**
- Decision: Use **PeerDAS Devnet 7** for Electra → Fulu blob testing.
- Malicious EL block tests planned to validate CL rejection behavior.

6. **History Expiry**
- No updates on rollout, testing, or documentation.

## Fusaka-devnet-0 Status

### Test Status

**Mario**
- Latest EEST release includes Fusaka SFI EIPs.
- Hive instance is running, but only **Go-Ethereum** is currently live.
- Other clients should reach out to Pari or EthPandaOps for access.
- **Simulator** not ready yet due to significant EEST changes; expected soon.

**Parithosh**
- Fusaka Devnet 0 hive instance: [Hive Dashboard](https://hive.ethpandaops.io/#/?group=fusaka-devnet-0)

**Barnabas**
- Devnet-0 to be launched tomorrow with Fulu activation following the next day..

### Client Updates

- **Erigon (EL):** WIP branch exists, not reviewed by Barnabas yet.
- **Nimbus (CL):** Testing an image internally.
- **Lodestar & Lighthouse:** No updates.

### Devnet 0 – BPO & Config Discussion

**Pawan**
- BPO EIP is available.

**Pari**
- Live on devnet 0. Specs still in flux; final version may be included in Devnet 1 or 2.

**ethDreamer (Mark)**
- Question on whether the BPO config includes blob schedule.

**Barnabas**
- Yes, genesis includes blob schedule (but only Electra blobs, not Fulu).
- Reference: [config.yaml#L194](https://github.com/ethpandaops/fusaka-devnets/blob/master/network-configs/devnet-0/metadata/config.yaml#L194)
- Future devnets will align on genesis date.

## Devnet 1 Planning

**SFI'd EIPs:**
- Includes **EIP-7918**
- Could be slightly delayed to align with **Interop in Berlin**

**Anders**
- Updated EIP-7918 specs recently: [Discussion](https://github.com/ethereum/pm/issues/1554#issuecomment-2908457240)

**Decision:** EIP-7918 will be included in Devnet 1.

### Demitri’s CL Specs – PR Discussion

**Parithosh**
- [CL spec PR 4323](https://github.com/ethereum/consensus-specs/pull/4323) still open.

**Pawan (Lighthouse)**
- Not useful for LH currently, unsure if it helps bandwidth.

**Gajinder (Lodestar)**
- Could reduce latency during multiple peer requests.
- Not useful if most clients use gossip pools.
- Favor of closing [this PR](https://github.com/ethereum/consensus-specs/pull/4323)

**Manu (Prysm)**
- Doesn’t currently use it but plans to. Wants the PR merged for future readiness.

**Decision:** Wait for more client feedback before closing the PR.

## PeerDAS Testing

### Client Updates

**Manu (Prysm)**
- Development is progressing well on PeerDAS testing.

**Pawan (Lighthouse)**
- BPO implementation is still in progress for Devnet 0.
- Merged a few synchronization-related fixes.
- Resolved a bug with `getblock`; the fix has been merged.
- Expected to be ready for the **Fulu fork** on Devnet 0.

**Sunnyside Labs**
- Conducted PeerDAS testing last week: [Test Report – May 20](https://testinprod.notion.site/Sunnyside-Devnet-Updates-05-20-1f88fc57f5468034a397d29017942763)
- Setup: 60 nodes with Execution Layer (EL) + Consensus Layer (CL).
- Focus: Assess number of blobs that can be reached; current goal is to reach **72 blobs per block**.

#### Observations
- Bottlenecks identified in `getblobs v2`.
- No significant performance difference observed between enabling and disabling `getblobs`.
- **Grandine** was requesting blobs multiple times.
- Grandine tested in standalone configuration.
- Lighthouse and Prysm were performing better with or without `getblobs` enabled.
- Further analysis and findings will be shared in the **Discord channel**.

#### Discussion & Troubleshooting

**Pawan**
- Asked if fresh sync tests were done with current parameters.

**Sunnyside**
- Yes, syncing was challenging:
- Multiple nodes failed to start.
- Required manual force-syncs and full network restarts.

**Francesco**
- Asked about **high CPU usage** on supernodes.

**Finding**
- High CPU usage was only observed on **Lodestar** supernode, not other clients.

### Additional Testing Requests

**Parithosh**
- Asked if any client teams had specific scenarios they want tested for PeerDAS.

**Manu**
- Suggested testing:
- Syncing in both **finalizing** and **non-finalizing** networks.
- Running a few hours of blob transactions from **Electra to Fulu**.

**Pari**
- Recommended using **PeerDAS Devnet 7**, which already contains **Fulu**.
- Agreed on basic sync testing.

**Manu**
- Confirmed **Electra** can be run on PeerDAS Devnet 7 for testing purposes.

### Malicious Block Testing

**Marius**
- Proposed creating an **EL block with invalid/cell block transactions** to ensure:
- It gets correctly **rejected by CL clients**.

**Pari**
- Acknowledged the need for a **malicious block** and that Marius can assist with its creation.

**Pawan**
- Compared the testing with the rejection scenario to a **deserialization failure over the Engine API**.

### EEST Testing Opportunity

**Mario**
- Mentioned a **new testing type in EEST** that could support this use case.

**Marius**
- Clarified that `getblobs` tests are different; this is focused more on **consensus-level block validation**.

**Mario**
- Acknowledged the distinction.

## History Expiry

No updates shared.

# Interop Testing Call #37 – May 19, 2025

**Facilitator**: Parithosh Jayanthi

## Summary
- ModXP + BPO changes confirmed for Fusaka Devnet 0.
- EL-only BPO hardfork agreed.
- CL-side signaling discussion ongoing. Likely to avoid h/f.
- PeerDAS testing and spec PRs progressing.
- Perfnet v2 & Hoodi upgrades planned.

**Next Steps**:
- Terence: Create Consensus Spec Issue
- Pari: Update EIP for EL handling of BPO
- Raúl: Finalize spec structure and open Consensus Dev thread
- All clients: Prepare for Devnet 0 testing next week

## Pectra Fork Retro
- No issues reported post data collection.

## Fusaka Devnet 0 Status
- Interop EIPs for Fusaka listed [on Discord](https://discord.com/channels/595666850260713488/892088344438255616/1373975537789304862).
- ModXP EIPs approved by all EL clients: Geth, Besu, Reth, Nethermind.
- [Fusaka Devnet 0 spec](https://notes.ethereum.org/@ethpandaops/fusaka-devnet-0)

### BPO Proposal (Marius)
- Proposal includes scheduling explicit forks:
- Example format:

    "bpo1": {
      "timestamp": 1747670415,
      "target": 9,
      "max": 12,
      "baseFeeUpdateFraction": 5007716
    }

- Treated as **EL-only hardfork**; CL remains unchanged.

**Discussion Points**:
- CL teams prefer avoiding unnecessary fork boilerplate.
- Marius: keep changes minimal, EL-only is cleaner.
- Potuz: prefer handshake signaling if hardfork can be avoided.
- Gajinder: handshake-only leaves room for inconsistent state.

**Next Steps**:
- [Consensus Spec Issue #4331](https://github.com/ethereum/consensus-specs/issues/4331) to be created by Terence.
- EIP to be updated by Pari.
- Raúl to lead long-term spec design and create a Consensus Dev channel thread.

**Decision**: Agreement reached for Fusaka Devnet 0 with ModXP + BPO changes.

## PeerDAS Testing
- **Devnet 7** is live.
- [Checkpoint sync available here](https://checkpoint-sync.peerdas-devnet-7.ethpandaops.io)
- All clients actively testing codecs.

**Status**:
- Target release next week for ModXP and PeerDAS.
- BPO tests to be run on Kurtosis (confirmed by Mario).

## PeerDAS Spec Discussion
- PR: [beacon-APIs #524](https://github.com/ethereum/beacon-APIs/pull/524)
- Discussed local flag and validator-blinded block behavior.
- Pari to reach out to maintainers for merge.

## P2P Propagation Testing

**Discussion between Potuz and Raúl** (Zoom chat):
- Proposal to rerun PandaOps-style tests with 1000+ distributed nodes.
- [Reference: EthResearch thread](https://ethresear.ch/t/faster-block-blob-propagation-in-ethereum/21370/37)

## Builder Specs
- Builder spec for Fusaka updated (Barnabas, Alex Stokes confirmed).

## Devnet & Release Planning
- Pari suggested, Devnet 0 completion will trigger team prep for release.
- **Perfnet v2** (mainnet-aligned) in progress – led by Kamil.

**Hoodi Testing**:
- Plan to test **60M block size** and attestation slashing.

**Decision**: Proceed with Hoodi block size test plan.

## History Expiry
- No new updates.
- Portal client call scheduled post-meeting for interested teams.

# Interop Testing Call #36 – May 12, 2025

**Facilitator**: Mario Vega

**Topics**: Pectra, Fusaka CFI, History Expiry

### Next steps:
- **Fusaka Devnet 0**
- **Launch Target**: ~2 weeks
- Devnet will include proposals:
- BPO
- PeerDAS
- ModExp proposals - [EIP-7823](https://eips.ethereum.org/EIPS/eip-7823), [EIP-7883](https://eips.ethereum.org/EIPS/eip-7883)
- **History Expiry**
- History Expiry drop and Documentation (on Sepolia) target: **June 1, 2025**

## Pectra-Related Discussions
- **Mario**: Generally, the upgrade was smooth and uneventful.
- **Mikhail**: `eth1_data` polling can be safely removed — smooth upgrade, great news!
- **James (Prysm)**: Still investigating performance issues.

## Testing Fusaka

- A [tracker for Fusaka Devnet 0 sentiment](https://notes.ethereum.org/@marioevz/fusaka-sfi-tracker) is available.

### Updates:
- **Barnabas**: PeerDAS Devnet 7 launched last week with 100% participation.
- **Will C**:
- Shipping Devnet 7 is the biggest one.
- BPO blob schedule decision needed.
- **Justin T**: Will decide in this call.
- **Will**: Major components are documented; minor PRs remain open.

**Next Step**: BPO to be included in Fusaka Devnet 0.

### Client Readiness:
- **Terence (Prysm)**: If Devnet 0 only includes BPO, Prysm will be ready in 1 week.
- **Som (Erigon)**: 2 weeks time requested.
- **Marcin (Nimbus) + Gajinder (EthJS)**: Ready whenever a date is set.
- Barnabas proposed **Target Launch Date**: May 26, 2025 – Fusaka Devnet 0 with BPO.
- **Justin (Besu)**: Prefers Devnet 0 with BPO + PeerDAS only.

### Additional Proposal Discussions for Devnet 0:
- **Som**: Mentioned to include PAY opcode.
- **Mario**: Suggests PAY be part of Devnet 1 due to complexity.
- **Tim**: PAY / EIP-7212 / contract code size increase also being considered for Devnet 1.

Please contact Mario for any suggestions for [Fusaka SFI Tracker](https://notes.ethereum.org/@marioevz/fusaka-sfi-tracker)

**Francis Li**: Suggested including [EIP-9623](https://github.com/ethereum/EIPs/pull/9623) — originally an EIP, now part of PeerDAS spec.

### Decision:
**Launch Target**: ~2 weeks
**Fusaka Devnet 0** will include:
- BPO
- PeerDAS
- ModExp proposals ([EIP-7823](https://eips.ethereum.org/EIPS/eip-7823), [EIP-7883](https://eips.ethereum.org/EIPS/eip-7883))

## Introduce Blob Schedule

- PR Merged: [Consensus Spec PR #4277](https://github.com/ethereum/consensus-specs/pull/4277)

## History Expiry
- [History Endpoints Doc](https://eth-clients.github.io/history-endpoints/)

### Status:
- **Barnabas**:
- Proposal: Start dropping history on Sepolia from **June 1** (originally May 1).
- **Justin F. (Besu)**: Agrees with June 1.
- **Geth (Marius)**:
- Code exists but not enabled.
- Will produce documentation in two weeks.
- **Andrew (Erigon)**:
- PR exists but not yet merged.
- Won’t make June 1 release, documentation possibly within 2 months.
- **Reth**: [No update provided]
- **Pari**: All clients Will publish documentation and user guide when ready.

### Decision:
**Sepolia History Expiry Documentation** target: **June 1, 2025**

🔗 Resources:
- [Consensus PR #4277](https://github.com/ethereum/consensus-specs/pull/4277)
- [PeerDAS Devnet 7 Notes](https://notes.ethereum.org/@ethpandaops/peerdas-devnet-7)
- [Tx Pool Changes](https://github.com/ethereum/EIPs/pull/9378/files)

# Interop Testing Call #35 - May 5, 2025

**Facilitated by:** Parithosh

## Pectra Update
### DevOps
**Parithosh**
* Blogpost update.
* 3 bug fixes released; no critical issues reported.
* Flashbots infra improvement needed — team has been informed.

### EEST
**Mario**
* A few Hive tests failing — documented and reviewing edge cases.
* Some scenarios seem unrealistic — will consult client teams.
* Manual review completed across all client repos: [Manual Review Notes](https://notes.ethereum.org/@marioevz/pectra-manual-review).

### Mainnet Shadow Fork

* Deployed last week.
* Sync issues on Erigon — now resolved.
* Override flag bugs observed and patched.
* Pectra features tested successfully.
* Shadow fork now removed.

### Latest EEST Release

* First release includes all Ethereum tests.
* **Plan**: migrate all tests to the unified **execution-tests** repo.
* May affect CI pipelines, but no immediate action required.

## Communication Channel for Pectra

* Pectra Mainnet Plan document is added in PM repo.
* A dedicated **Pectra Upgrade channel** will be created pre-fork and retired post-fork.
* Each team to have a primary & backup contact monitoring `Eth R&D` Discord.
* [Mainnet Checklist](https://ethpandaops.io/posts/pectra-mainnet-checklist/)
* Lukasz suggested advance setup for emergency call link — Tim confirmed.

## PeerDAS

**Update by Barnabas**

* [Devnet-7](https://notes.ethereum.org/@ethpandaops/peerdas-devnet-7) expected to launch tomorrow.
* Devnet-6 will be deprecated (except for Sunnyside labs which will stay for a bit but eventually deprecated).
* Client teams to upload updated images.
* PeerDAS discussions will move to ACDT after Pectra; **PeerDAS calls end after tomorrow.**

### Gas Limit Testing

* Jared: no current blockers.
* Next step: decide testnet for gas limit testing in next week’s call.

## History Expiry

* May 1 deadline passed.
* Documentation is lacking — clearer guidance needed.
* Pari to follow up in the **History Expiry** channel.

### Client Updates:

* **Nethermind (Kamil):** Ready to support drop, will share docs.
* **Barnabas:** Inquired about dropping by block height — confirmed possible.
* **Nethermind (Lukasz):** Live pruning in PRs.
* Suggestion to add history expiry info to release notes.
* Need improved validator/user communication — plan for dedicated DevDocs page.
* **Reth & Geth:** Work in progress.
* **Besu (Daniel):** Still testing.
* EF blog post planned once all clients releases are available.
* Need to decide new "History Drop" activation target date.
* Piper (History Expiry champion) should join next call.

## Next Steps
**Pectra Upgrade Monitoring**
• Parithosh to set up and share a scheduled meeting link for monitoring during the Pectra upgrade. Set up and publicize the Pectra Upgrade Discord channel, including an emergency meeting link.

**Testing & Devnets**
* Mario and the EEST team to assess and potentially add a new test for the EIP-7702 wording update.
* Launch Devnet-7 for PeerDAS and deprecate Devnet-6.
* Client teams to prepare and publish updated images for Devnet-7.
* Confirm the testnet to be used for gas limit testing in next week’s meeting.

**History Expiry**
* Parithosh to initiate discussion in the History Expiry channel to align on a new implementation plan and deadline.
* Client teams to update release notes and clearly state their history expiry support status.
* Create a dedicated documentation page to educate validators and users on History Expiry.
* Schedule a follow-up with Piper Merriam for deeper coordination on rollout.

**Communications**
• Publish an Ethereum Foundation blog post to announce History Expiry once all client implementations are complete.

# Interop Testing Call #34 and before
[here](https://hackmd.io/@poojaranjan/InteropTestingNotes)