# Besu fusaka-devnet-5 Issues

## TL;DR

Investigated two main issues during devnet-5:
1. High memory usage - blobs in the txpool
2. Empty blocks related to Nimbus sending payloadAttributes late (only 500ms block building window)

In order to mitigate the memory issue, we added:

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-prioritized=1000
  - --tx-pool-max-prioritized-by-type=BLOB=72

To mitigate empty block issue, we added:

- --block-txs-selection-max-time=150

These configs helped mitigate the issues seen on devnet-5, but we don't plan on changing Besu default config for other networks because:
1. We can't recreate the memory issue yet; we'd like to see and debug it next time it happens.
2. Nimbus resolved the issue on their side so we're not expecting empty blocks for the next test.

We can add this config to Besu release notes as a backup solution if it is a problem.

The configs likely resulted in some stacktraces we wouldn't expect to see otherwise (and don't see on devnet-3).

Even with the config, We did still see low blob numbers especially for supernodes.

**We hope to resolve/harden against these issues in code ready for the Mainnet release.**

## Outstanding Issues

1. [Low blob numbers](#2025-09-19-0941-Barnabus-announced-increase-in-blob-traffic-to-test-72-blob-max-I-noticed-Besu-had-low-blob-numbers-lt-6%E2%80%A6)
2. [Blob memory issues](#2025-09-16-0318)
3. [Attempting to access worldstate Stack trace](#2025-09-19-2100-Barnabus-reported-Attempting-to-access-closed-worldstate-stacktrace) - not seen on devnet-3 so could be related to the devnet-5 specific besu config we added
4. [engine_getBlobsV2 timeouts](#2025-09-16-0700---engine_getBlobsV2-timeouts-reported-by-lighthouse)
5. [Finalize block upon receipt of getPayload](https://github.com/hyperledger/besu/pull/9194) - would help return lower quality block instead of empty block when block building window is short

## Resolved Issues

1. [BAL feature flag leak bug](#2025-09-19-0248-BAL-fix-deployed)
2. [Nimbus 500ms FCU+payloadAttributes -> getPayload window](#2025-09-20-1034---nimbus-fixed-500ms-issue)

## Useful Notes

> https://grafana.xatu-experimental.ethpandaops.io/goto/pmQO7ACHg?orgId=1 can track the average blob count here, might align with your memory usage

### Spamoor

> @siladu
> Do you know how many blobs per tx are being sent?
> @Barnabas
> we have 1 blob per tx
> We could increase that a bit, as I assume that 1 blob per tx is not super realistic in a 72 blob env

> @siladu Do you know if there are lots of duplicate blobs sent in the spam via different txs?
> @pk910
No, these should be all unique blobs with random data.

### Memory issue notes:

1. [MONITORING] add config for txpool limits
2. [TODO] reproduce locally with kurtosis + blob spammoor
3. [TODO] can we change TransactionPool.mapOfBlobsInTransactionPool data structure to be more minimal?
4. [TODO] Add configurable limits to TransactionPool.mapOfBlobsInTransactionPool

## Timeline

Times in UTC (screenshot filenames in UTC+10)

### 2025-09-10 14:00 Devnet-5 Launched

[~440 Besu nodes](https://grafana.observability.ethpandaops.io/d/_Mqt4ksnz2/besu-full?orgId=1&from=now-13d&to=now&timezone=utc&var-system=fusaka-devnet-5-teku-besu-1-arm&var-system=fusaka-devnet-5-teku-besu-super-1&var-system=fusaka-devnet-5-teku-besu-super-2&var-system=fusaka-devnet-5-teku-besu-super-3&var-system=fusaka-devnet-5-teku-besu-super-4&var-system=fusaka-devnet-5-prysm-besu-1-arm&var-system=fusaka-devnet-5-prysm-besu-2-arm&var-system=fusaka-devnet-5-prysm-besu-3-arm&var-system=fusaka-devnet-5-prysm-besu-full-1&var-system=fusaka-devnet-5-prysm-besu-full-2&var-system=fusaka-devnet-5-prysm-besu-full-3&var-system=fusaka-devnet-5-prysm-besu-full-4&var-system=fusaka-devnet-5-prysm-besu-full-5&var-system=fusaka-devnet-5-prysm-besu-full-6&var-system=fusaka-devnet-5-prysm-besu-full-7&var-system=fusaka-devnet-5-prysm-besu-full-8&var-system=fusaka-devnet-5-prysm-besu-full-9&var-system=fusaka-devnet-5-prysm-besu-full-10&var-system=fusaka-devnet-5-prysm-besu-full-11&var-system=fusaka-devnet-5-prysm-besu-full-12&var-system=fusaka-devnet-5-prysm-besu-full-13&var-system=fusaka-devnet-5-prysm-besu-full-14&var-system=fusaka-devnet-5-prysm-besu-full-15&var-system=fusaka-devnet-5-prysm-besu-full-16&var-system=fusaka-devnet-5-prysm-besu-full-17&var-system=fusaka-devnet-5-prysm-besu-full-18&var-system=fusaka-devnet-5-prysm-besu-full-19&var-system=fusaka-devnet-5-prysm-besu-full-20&var-system=fusaka-devnet-5-prysm-besu-full-21&var-system=fusaka-devnet-5-prysm-besu-full-22&var-system=fusaka-devnet-5-prysm-besu-full-23&var-system=fusaka-devnet-5-prysm-besu-full-24&var-system=fusaka-devnet-5-prysm-besu-full-25&var-system=fusaka-devnet-5-prysm-besu-full-26&var-system=fusaka-devnet-5-prysm-besu-full-27&var-system=fusaka-devnet-5-prysm-besu-full-28&var-system=fusaka-devnet-5-prysm-besu-full-29&var-system=fusaka-devnet-5-prysm-besu-full-30&var-system=fusaka-devnet-5-prysm-besu-full-31&var-system=fusaka-devnet-5-prysm-besu-full-32&var-system=fusaka-devnet-5-prysm-besu-full-33&var-system=fusaka-devnet-5-prysm-besu-full-34&var-system=fusaka-devnet-5-prysm-besu-full-35&var-system=fusaka-devnet-5-prysm-besu-full-36&var-system=fusaka-devnet-5-prysm-besu-full-37&var-system=fusaka-devnet-5-prysm-besu-full-38&var-system=fusaka-devnet-5-prysm-besu-full-39&var-system=fusaka-devnet-5-prysm-besu-full-40&var-system=fusaka-devnet-5-prysm-besu-full-41&var-system=fusaka-devnet-5-prysm-besu-full-42&var-system=fusaka-devnet-5-prysm-besu-full-43&var-system=fusaka-devnet-5-prysm-besu-full-44&var-system=fusaka-devnet-5-prysm-besu-full-45&var-system=fusaka-devnet-5-prysm-besu-full-46&var-system=fusaka-devnet-5-prysm-besu-full-47&var-system=fusaka-devnet-5-prysm-besu-full-48&var-system=fusaka-devnet-5-prysm-besu-full-49&var-system=fusaka-devnet-5-prysm-besu-full-50&var-system=fusaka-devnet-5-prysm-besu-full-51&var-system=fusaka-devnet-5-prysm-besu-full-52&var-system=fusaka-devnet-5-prysm-besu-full-53&var-system=fusaka-devnet-5-prysm-besu-full-54&var-system=fusaka-devnet-5-prysm-besu-full-55&var-system=fusaka-devnet-5-prysm-besu-full-56&var-system=fusaka-devnet-5-prysm-besu-full-57&var-system=fusaka-devnet-5-prysm-besu-full-58&var-system=fusaka-devnet-5-prysm-besu-full-59&var-system=fusaka-devnet-5-prysm-besu-full-60&var-system=fusaka-devnet-5-prysm-besu-full-61&var-system=fusaka-devnet-5-prysm-besu-full-62&var-system=fusaka-devnet-5-prysm-besu-full-63&var-system=fusaka-devnet-5-prysm-besu-full-64&var-system=fusaka-devnet-5-prysm-besu-full-65&var-system=fusaka-devnet-5-prysm-besu-full-66&var-system=fusaka-devnet-5-prysm-besu-full-67&var-system=fusaka-devnet-5-prysm-besu-full-68&var-system=fusaka-devnet-5-prysm-besu-full-69&var-system=fusaka-devnet-5-prysm-besu-full-70&var-system=fusaka-devnet-5-prysm-besu-full-71&var-system=fusaka-devnet-5-prysm-besu-full-72&var-system=fusaka-devnet-5-prysm-besu-full-73&var-system=fusaka-devnet-5-prysm-besu-full-74&var-system=fusaka-devnet-5-prysm-besu-full-75&var-system=fusaka-devnet-5-prysm-besu-full-76&var-system=fusaka-devnet-5-prysm-besu-full-77&var-system=fusaka-devnet-5-prysm-besu-full-78&var-system=fusaka-devnet-5-prysm-besu-super-1&var-system=fusaka-devnet-5-prysm-besu-super-2&var-system=fusaka-devnet-5-prysm-besu-super-3&var-system=fusaka-devnet-5-prysm-besu-super-4&var-system=fusaka-devnet-5-prysm-besu-super-5&var-system=fusaka-devnet-5-prysm-besu-super-6&var-system=fusaka-devnet-5-prysm-besu-super-7&var-system=fusaka-devnet-5-prysm-besu-super-8&var-system=fusaka-devnet-5-nimbus-besu-1-arm&var-system=fusaka-devnet-5-nimbus-besu-full-1&var-system=fusaka-devnet-5-nimbus-besu-full-2&var-system=fusaka-devnet-5-nimbus-besu-full-3&var-system=fusaka-devnet-5-nimbus-besu-full-4&var-system=fusaka-devnet-5-nimbus-besu-full-5&var-system=fusaka-devnet-5-nimbus-besu-full-6&var-system=fusaka-devnet-5-nimbus-besu-full-7&var-system=fusaka-devnet-5-nimbus-besu-full-8&var-system=fusaka-devnet-5-nimbus-besu-full-9&var-system=fusaka-devnet-5-nimbus-besu-full-10&var-system=fusaka-devnet-5-nimbus-besu-full-11&var-system=fusaka-devnet-5-nimbus-besu-full-12&var-system=fusaka-devnet-5-nimbus-besu-full-13&var-system=fusaka-devnet-5-nimbus-besu-full-14&var-system=fusaka-devnet-5-nimbus-besu-full-15&var-system=fusaka-devnet-5-nimbus-besu-full-16&var-system=fusaka-devnet-5-nimbus-besu-full-17&var-system=fusaka-devnet-5-nimbus-besu-full-18&var-system=fusaka-devnet-5-nimbus-besu-full-19&var-system=fusaka-devnet-5-nimbus-besu-full-20&var-system=fusaka-devnet-5-nimbus-besu-full-21&var-system=fusaka-devnet-5-nimbus-besu-full-22&var-system=fusaka-devnet-5-nimbus-besu-full-23&var-system=fusaka-devnet-5-nimbus-besu-full-24&var-system=fusaka-devnet-5-nimbus-besu-full-25&var-system=fusaka-devnet-5-nimbus-besu-full-26&var-system=fusaka-devnet-5-nimbus-besu-super-1&var-system=fusaka-devnet-5-nimbus-besu-super-2&var-system=fusaka-devnet-5-nimbus-besu-super-3&var-system=fusaka-devnet-5-grandine-besu-full-1&var-system=fusaka-devnet-5-grandine-besu-full-2&var-system=fusaka-devnet-5-grandine-besu-full-3&var-system=fusaka-devnet-5-grandine-besu-full-4&var-system=fusaka-devnet-5-grandine-besu-full-5&var-system=fusaka-devnet-5-grandine-besu-full-6&var-system=fusaka-devnet-5-grandine-besu-full-7&var-system=fusaka-devnet-5-grandine-besu-full-8&var-system=fusaka-devnet-5-grandine-besu-full-9&var-system=fusaka-devnet-5-grandine-besu-full-10&var-system=fusaka-devnet-5-grandine-besu-full-11&var-system=fusaka-devnet-5-grandine-besu-full-12&var-system=fusaka-devnet-5-grandine-besu-full-13&var-system=fusaka-devnet-5-grandine-besu-full-14&var-system=fusaka-devnet-5-grandine-besu-full-15&var-system=fusaka-devnet-5-grandine-besu-full-16&var-system=fusaka-devnet-5-grandine-besu-full-17&var-system=fusaka-devnet-5-grandine-besu-full-18&var-system=fusaka-devnet-5-grandine-besu-full-19&var-system=fusaka-devnet-5-grandine-besu-super-1&var-system=fusaka-devnet-5-grandine-besu-super-2&var-system=fusaka-devnet-5-lodestar-besu-1-arm&var-system=fusaka-devnet-5-lodestar-besu-full-1&var-system=fusaka-devnet-5-lodestar-besu-full-2&var-system=fusaka-devnet-5-lodestar-besu-full-3&var-system=fusaka-devnet-5-lodestar-besu-full-4&var-system=fusaka-devnet-5-lodestar-besu-full-5&var-system=fusaka-devnet-5-lodestar-besu-full-6&var-system=fusaka-devnet-5-lodestar-besu-full-7&var-system=fusaka-devnet-5-lodestar-besu-full-8&var-system=fusaka-devnet-5-lodestar-besu-full-9&var-system=fusaka-devnet-5-lodestar-besu-full-10&var-system=fusaka-devnet-5-lodestar-besu-full-11&var-system=fusaka-devnet-5-lodestar-besu-full-12&var-system=fusaka-devnet-5-lodestar-besu-full-13&var-system=fusaka-devnet-5-lodestar-besu-full-14&var-system=fusaka-devnet-5-lodestar-besu-full-15&var-system=fusaka-devnet-5-lodestar-besu-full-16&var-system=fusaka-devnet-5-lodestar-besu-full-17&var-system=fusaka-devnet-5-lodestar-besu-full-18&var-system=fusaka-devnet-5-lodestar-besu-full-19&var-system=fusaka-devnet-5-lodestar-besu-super-1&var-system=fusaka-devnet-5-lodestar-besu-super-2&var-system=fusaka-devnet-5-ssl-teku-besu-1&var-system=fusaka-devnet-5-ssl-teku-besu-2&var-system=fusaka-devnet-5-ssl-teku-besu-3&var-system=fusaka-devnet-5-ssl-teku-besu-4&var-system=fusaka-devnet-5-ssl-teku-besu-5&var-system=fusaka-devnet-5-ssl-teku-besu-6&var-system=fusaka-devnet-5-ssl-teku-besu-7&var-system=fusaka-devnet-5-ssl-teku-besu-8&var-system=fusaka-devnet-5-ssl-teku-besu-9&var-system=fusaka-devnet-5-ssl-teku-besu-10&var-system=fusaka-devnet-5-ssl-teku-besu-11&var-system=fusaka-devnet-5-ssl-teku-besu-12&var-system=fusaka-devnet-5-ssl-teku-besu-13&var-system=fusaka-devnet-5-ssl-teku-besu-14&var-system=fusaka-devnet-5-ssl-teku-besu-15&var-system=fusaka-devnet-5-ssl-teku-besu-16&var-system=fusaka-devnet-5-ssl-teku-besu-17&var-system=fusaka-devnet-5-ssl-teku-besu-18&var-system=fusaka-devnet-5-ssl-teku-besu-19&var-system=fusaka-devnet-5-ssl-teku-besu-20&var-system=fusaka-devnet-5-ssl-teku-besu-21&var-system=fusaka-devnet-5-ssl-teku-besu-22&var-system=fusaka-devnet-5-ssl-teku-besu-23&var-system=fusaka-devnet-5-ssl-teku-besu-24&var-system=fusaka-devnet-5-ssl-teku-besu-25&var-system=fusaka-devnet-5-ssl-teku-besu-26&var-system=fusaka-devnet-5-ssl-teku-besu-27&var-system=fusaka-devnet-5-ssl-teku-besu-28&var-system=fusaka-devnet-5-ssl-teku-besu-29&var-system=fusaka-devnet-5-ssl-teku-besu-30&var-system=fusaka-devnet-5-ssl-teku-besu-31&var-system=fusaka-devnet-5-ssl-teku-besu-32&var-system=fusaka-devnet-5-ssl-teku-besu-33&var-system=fusaka-devnet-5-ssl-teku-besu-34&var-system=fusaka-devnet-5-ssl-teku-besu-35&var-system=fusaka-devnet-5-ssl-teku-besu-36&var-system=fusaka-devnet-5-ssl-teku-besu-37&var-system=fusaka-devnet-5-ssl-teku-besu-38&var-system=fusaka-devnet-5-ssl-teku-besu-39&var-system=fusaka-devnet-5-ssl-teku-besu-full-1&var-system=fusaka-devnet-5-ssl-teku-besu-full-2&var-system=fusaka-devnet-5-ssl-teku-besu-full-3&var-system=fusaka-devnet-5-ssl-teku-besu-full-4&var-system=fusaka-devnet-5-ssl-teku-besu-full-5&var-system=fusaka-devnet-5-ssl-teku-besu-full-6&var-system=fusaka-devnet-5-ssl-teku-besu-full-7&var-system=fusaka-devnet-5-ssl-teku-besu-full-8&var-system=fusaka-devnet-5-ssl-teku-besu-full-9&var-system=fusaka-devnet-5-ssl-teku-besu-full-10&var-system=fusaka-devnet-5-ssl-teku-besu-full-11&var-system=fusaka-devnet-5-ssl-teku-besu-full-12&var-system=fusaka-devnet-5-ssl-teku-besu-full-13&var-system=fusaka-devnet-5-ssl-teku-besu-full-14&var-system=fusaka-devnet-5-ssl-teku-besu-full-15&var-system=fusaka-devnet-5-ssl-teku-besu-full-16&var-system=fusaka-devnet-5-ssl-teku-besu-full-17&var-system=fusaka-devnet-5-ssl-teku-besu-full-18&var-system=fusaka-devnet-5-ssl-teku-besu-full-19&var-system=fusaka-devnet-5-ssl-teku-besu-full-20&var-system=fusaka-devnet-5-ssl-teku-besu-full-21&var-system=fusaka-devnet-5-ssl-teku-besu-full-22&var-system=fusaka-devnet-5-ssl-teku-besu-full-23&var-system=fusaka-devnet-5-ssl-teku-besu-full-24&var-system=fusaka-devnet-5-ssl-teku-besu-full-25&var-system=fusaka-devnet-5-ssl-teku-besu-full-26&var-system=fusaka-devnet-5-ssl-teku-besu-full-27&var-system=fusaka-devnet-5-ssl-teku-besu-full-28&var-system=fusaka-devnet-5-ssl-teku-besu-full-29&var-system=fusaka-devnet-5-ssl-teku-besu-full-30&var-system=fusaka-devnet-5-ssl-teku-besu-full-31&var-system=fusaka-devnet-5-ssl-teku-besu-full-32&var-system=fusaka-devnet-5-ssl-teku-besu-full-33&var-system=fusaka-devnet-5-ssl-teku-besu-full-34&var-system=fusaka-devnet-5-ssl-teku-besu-full-35&var-system=fusaka-devnet-5-ssl-teku-besu-full-36&var-system=fusaka-devnet-5-ssl-teku-besu-full-37&var-system=fusaka-devnet-5-ssl-teku-besu-full-38&var-system=fusaka-devnet-5-ssl-teku-besu-full-39&var-system=fusaka-devnet-5-ssl-teku-besu-nonval-1&var-system=fusaka-devnet-5-ssl-teku-besu-nonval-2&var-system=fusaka-devnet-5-ssl-teku-besu-nonval-3&var-system=fusaka-devnet-5-ssl-teku-besu-nonval-4&var-system=fusaka-devnet-5-ssl-teku-besu-nonval-5&var-system=fusaka-devnet-5-ssl-prysm-besu-nonval-1&var-system=fusaka-devnet-5-ssl-prysm-besu-nonval-2&var-system=fusaka-devnet-5-ssl-prysm-besu-nonval-3&var-system=fusaka-devnet-5-ssl-prysm-besu-nonval-4&var-system=fusaka-devnet-5-ssl-prysm-besu-nonval-5&var-system=fusaka-devnet-5-lighthouse-besu-1-arm&var-system=fusaka-devnet-5-lighthouse-besu-2-arm&var-system=fusaka-devnet-5-lighthouse-besu-3-arm&var-system=fusaka-devnet-5-lighthouse-besu-super-1&var-system=fusaka-devnet-5-lighthouse-besu-super-2&var-system=fusaka-devnet-5-lighthouse-besu-super-3&var-system=fusaka-devnet-5-lighthouse-besu-super-4&var-system=fusaka-devnet-5-lighthouse-besu-super-5&var-system=fusaka-devnet-5-lighthouse-besu-super-6&var-system=fusaka-devnet-5-lighthouse-besu-super-7&var-system=fusaka-devnet-5-lighthouse-besu-super-8&var-system=fusaka-devnet-5-ssl-nimbus-besu-nonval-1&var-system=fusaka-devnet-5-ssl-nimbus-besu-nonval-2&var-system=fusaka-devnet-5-ssl-nimbus-besu-nonval-3&var-system=fusaka-devnet-5-ssl-nimbus-besu-nonval-4&var-system=fusaka-devnet-5-ssl-nimbus-besu-nonval-5&var-system=fusaka-devnet-5-ssl-grandine-besu-nonval-1&var-system=fusaka-devnet-5-ssl-grandine-besu-nonval-2&var-system=fusaka-devnet-5-ssl-grandine-besu-nonval-3&var-system=fusaka-devnet-5-ssl-grandine-besu-nonval-4&var-system=fusaka-devnet-5-ssl-grandine-besu-nonval-5&var-system=fusaka-devnet-5-ssl-lodestar-besu-nonval-1&var-system=fusaka-devnet-5-ssl-lodestar-besu-nonval-2&var-system=fusaka-devnet-5-ssl-lodestar-besu-nonval-3&var-system=fusaka-devnet-5-ssl-lodestar-besu-nonval-4&var-system=fusaka-devnet-5-ssl-lodestar-besu-nonval-5&var-system=fusaka-devnet-5-ssl-lighthouse-besu-1&var-system=fusaka-devnet-5-ssl-lighthouse-besu-2&var-system=fusaka-devnet-5-ssl-lighthouse-besu-3&var-system=fusaka-devnet-5-ssl-lighthouse-besu-4&var-system=fusaka-devnet-5-ssl-lighthouse-besu-5&var-system=fusaka-devnet-5-ssl-lighthouse-besu-6&var-system=fusaka-devnet-5-ssl-lighthouse-besu-7&var-system=fusaka-devnet-5-ssl-lighthouse-besu-8&var-system=fusaka-devnet-5-ssl-lighthouse-besu-9&var-system=fusaka-devnet-5-ssl-lighthouse-besu-10&var-system=fusaka-devnet-5-ssl-lighthouse-besu-11&var-system=fusaka-devnet-5-ssl-lighthouse-besu-12&var-system=fusaka-devnet-5-ssl-lighthouse-besu-13&var-system=fusaka-devnet-5-ssl-lighthouse-besu-14&var-system=fusaka-devnet-5-ssl-lighthouse-besu-15&var-system=fusaka-devnet-5-ssl-lighthouse-besu-16&var-system=fusaka-devnet-5-ssl-lighthouse-besu-17&var-system=fusaka-devnet-5-ssl-lighthouse-besu-18&var-system=fusaka-devnet-5-ssl-lighthouse-besu-19&var-system=fusaka-devnet-5-ssl-lighthouse-besu-20&var-system=fusaka-devnet-5-ssl-lighthouse-besu-21&var-system=fusaka-devnet-5-ssl-lighthouse-besu-22&var-system=fusaka-devnet-5-ssl-lighthouse-besu-23&var-system=fusaka-devnet-5-ssl-lighthouse-besu-24&var-system=fusaka-devnet-5-ssl-lighthouse-besu-25&var-system=fusaka-devnet-5-ssl-lighthouse-besu-26&var-system=fusaka-devnet-5-ssl-lighthouse-besu-27&var-system=fusaka-devnet-5-ssl-lighthouse-besu-28&var-system=fusaka-devnet-5-ssl-lighthouse-besu-29&var-system=fusaka-devnet-5-ssl-lighthouse-besu-30&var-system=fusaka-devnet-5-ssl-lighthouse-besu-31&var-system=fusaka-devnet-5-ssl-lighthouse-besu-32&var-system=fusaka-devnet-5-ssl-lighthouse-besu-33&var-system=fusaka-devnet-5-ssl-lighthouse-besu-34&var-system=fusaka-devnet-5-ssl-lighthouse-besu-35&var-system=fusaka-devnet-5-ssl-lighthouse-besu-36&var-system=fusaka-devnet-5-ssl-lighthouse-besu-37&var-system=fusaka-devnet-5-ssl-lighthouse-besu-38&var-system=fusaka-devnet-5-ssl-lighthouse-besu-39&var-system=fusaka-devnet-5-ssl-lighthouse-besu-40&var-system=fusaka-devnet-5-ssl-lighthouse-besu-41&var-system=fusaka-devnet-5-ssl-lighthouse-besu-42&var-system=fusaka-devnet-5-ssl-lighthouse-besu-43&var-system=fusaka-devnet-5-ssl-lighthouse-besu-44&var-system=fusaka-devnet-5-ssl-lighthouse-besu-45&var-system=fusaka-devnet-5-ssl-lighthouse-besu-46&var-system=fusaka-devnet-5-ssl-lighthouse-besu-47&var-system=fusaka-devnet-5-ssl-lighthouse-besu-48&var-system=fusaka-devnet-5-ssl-lighthouse-besu-49&var-system=fusaka-devnet-5-ssl-lighthouse-besu-50&var-system=fusaka-devnet-5-ssl-lighthouse-besu-51&var-system=fusaka-devnet-5-ssl-lighthouse-besu-52&var-system=fusaka-devnet-5-ssl-lighthouse-besu-53&var-system=fusaka-devnet-5-ssl-lighthouse-besu-54&var-system=fusaka-devnet-5-ssl-lighthouse-besu-55&var-system=fusaka-devnet-5-ssl-lighthouse-besu-56&var-system=fusaka-devnet-5-ssl-lighthouse-besu-57&var-system=fusaka-devnet-5-ssl-lighthouse-besu-58&var-system=fusaka-devnet-5-ssl-lighthouse-besu-59&var-system=fusaka-devnet-5-ssl-lighthouse-besu-60&var-system=fusaka-devnet-5-ssl-lighthouse-besu-61&var-system=fusaka-devnet-5-ssl-lighthouse-besu-62&var-system=fusaka-devnet-5-ssl-lighthouse-besu-63&var-system=fusaka-devnet-5-ssl-lighthouse-besu-64&var-system=fusaka-devnet-5-ssl-lighthouse-besu-65&var-system=fusaka-devnet-5-ssl-lighthouse-besu-66&var-system=fusaka-devnet-5-ssl-lighthouse-besu-67&var-system=fusaka-devnet-5-ssl-lighthouse-besu-68&var-system=fusaka-devnet-5-ssl-lighthouse-besu-69&var-system=fusaka-devnet-5-ssl-lighthouse-besu-70&var-system=fusaka-devnet-5-ssl-lighthouse-besu-71&var-system=fusaka-devnet-5-ssl-lighthouse-besu-72&var-system=fusaka-devnet-5-ssl-lighthouse-besu-73&var-system=fusaka-devnet-5-ssl-lighthouse-besu-74&var-system=fusaka-devnet-5-ssl-lighthouse-besu-75&var-system=fusaka-devnet-5-ssl-lighthouse-besu-76&var-system=fusaka-devnet-5-ssl-lighthouse-besu-77&var-system=fusaka-devnet-5-ssl-lighthouse-besu-78&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-1&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-2&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-3&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-4&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-5&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-6&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-7&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-8&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-9&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-10&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-11&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-12&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-13&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-14&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-15&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-16&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-17&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-18&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-19&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-20&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-21&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-22&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-23&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-24&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-25&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-26&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-27&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-28&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-29&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-30&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-31&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-32&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-33&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-34&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-35&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-36&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-37&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-38&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-39&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-40&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-41&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-42&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-43&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-44&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-45&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-46&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-47&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-48&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-49&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-50&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-51&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-52&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-53&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-54&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-55&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-56&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-57&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-58&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-59&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-60&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-61&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-62&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-63&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-64&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-65&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-66&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-67&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-68&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-69&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-70&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-71&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-72&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-73&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-74&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-75&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-76&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-77&var-system=fusaka-devnet-5-ssl-lighthouse-besu-full-78&var-system=fusaka-devnet-5-ssl-lighthouse-besu-nonval-1&var-system=fusaka-devnet-5-ssl-lighthouse-besu-nonval-2&var-system=fusaka-devnet-5-ssl-lighthouse-besu-nonval-3&var-system=fusaka-devnet-5-ssl-lighthouse-besu-nonval-4&var-system=fusaka-devnet-5-ssl-lighthouse-besu-nonval-5&var-sync_task_quantile=$__all&var-executor=$__all&var-vertx_pool_name=$__all&var-protocol_inbound=$__all&var-protocol_outbound=$__all&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=$__all&var-txpool_layer=$__all&var-network=$__all&var-Filters=&refresh=1m)

A small number of supernodes had memory issues, particularly nimbus-besu-super-1 and nimbus-besu-super-3

### **2025-09-15 16:29**
> besu is proposing lots of empty blocks on devnet 5
on devnet 3 however, it seems to only be happening with nimbus

Empty Block investigation started

**2025-09-15 01:40** - no obvious issues in txpool metrics. Potential concern that there's high number of txs in Prioritised layer versus Ready layer (mainnet is the other way around) - could just be the nature of the spamoor txs, e.g. high fees paid


DEBUG logs show lots of

2025-09-16 01:23:00.858+0000 | pool-2-thread-6 | DEBUG | MainnetTransactionProcessor | Invalid transaction: transaction nonce 29068 does not match sender account nonce 29067.

TODO investigate further? - potentially non-issue as may just be how spamoor works?

### 2025-09-16 03:18
> Getting an OOM on nimbus-besu-super-1


Noted nodes are 32GB, Xmx defaulting to 8GB
Noticed network is 100M gas.

Note, memory issues started during BPO1 (10 target / 15 max).

Collected heap dump without then with garbage collection (had to install JDK in the container)

# without GC
jcmd $PID GC.heap_dump -all "$OUT"

# with GC
jcmd $PID GC.heap_dump "$OUT"

Both heap dumps had ~8GB of data, so GC didn't remove - whole heap was retained, suggesting a leak

90% taken up by `TransactionPool` `ListMultimap<VersionedHash, BlobProofBundle> mapOfBlobsInTransactionPool`

Memory was also high on nimbus-besu-super-2

Tried to correlate memory with incoming blob txs - tricky because think we're only seeing the blob txs allowed into the pool, seems to top out at ~200 at any given time and drop rapidly.


TODO Continue memory investigation

### **2025-09-16 07:00** - engine_getBlobsV2 timeouts reported by lighthouse


TODO - investigate. Did not have chance to investigate this while other issues were ongoing.

### **2025-09-16 07:36**

> quick workaround that does not require any development, and that can give us time, is to reduce the size of the txpool, so less blobs are kept, a very strict conf could be:
--tx-pool-layer-max-capacity=10000000 (10MB)
--tx-pool-max-size=1000
--tx-pool-max-prioritized-by-type=BLOB=6
This way the txpool should not take more that 100MB of memory, and we can see if there is a memory leak

### **2025-09-16 10:00 UTC** - txpool limiting config deployed: initially buggy flag which killed besu nodes, Barnabas reverted
https://github.com/ethpandaops/fusaka-devnets/pull/84

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-size=1000
  - --tx-pool-max-prioritized-by-type=BLOB=6

### **2025-09-16 23:43** - Fixed config deployed
https://github.com/ethpandaops/fusaka-devnets/pull/86

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-prioritized=1000
  - --tx-pool-max-prioritized-by-type=BLOB=6

The expectation was this would limit memory usage by limited number of txs and blob txs.

Later tried to correlate memory usage with https://grafana.xatu-experimental.ethpandaops.io/goto/pmQO7ACHg?orgId=1 which should show average blob count:


It appeared the config worked though hard to be sure we were seeing the same levels of traffic as some could have been via tx gossip too.

Unfortunately we didn't take another heap dump with this config applied.

TODO - try to replicate the memory issue locally to prove this

### **2025-09-17 20:39** Informed @tersec about potential issue with Nimbus:
> @tersec some nimbus-besu nodes are failing to build a block so returning an empty block.
> On nimbus-besu-super-1 and nimbus-besu-super-2 this appears to be due to:
besu is taking longer than 500ms to build a block with selected txs
nimbus is calling engine_getPayload 500ms after the last FCU

Empty blocks, particularly with Nimbus: Nimbus had a bug where they were sending FCU+payloadAttributes at the start of the current slot, only giving us 500ms to build a block

### **2025-09-17 13:52** - selection max time reduced from 5 seconds default to 400ms https://github.com/ethpandaops/fusaka-devnets/pull/87

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-prioritized=1000
  - --tx-pool-max-prioritized-by-type=BLOB=6
  - --block-txs-selection-max-time=400

This didn't help since there is a grace time added to complete the current tx being selected.
Grace time = max time + 100ms = 500ms
So actual total selection timeout was max + grace time = 400 + 500 = 900ms

### **2025-09-17 23:41** - selection max time reduced from 400 to 150ms https://github.com/ethpandaops/fusaka-devnets/pull/90

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-prioritized=1000
  - --tx-pool-max-prioritized-by-type=BLOB=6
  - --block-txs-selection-max-time=150

grace time = max time + 100ms = 150 + 100 = 250 ms
selection timeout = max time + grace time = 150 + 250 = 400ms
leaving us with 100ms to finalize the block after selection

From the 150ms config deployment until 02:57am (~3 hours), pandaops (accidentally?) enabled MEV on the besu nodes which bypassed the issue, once I pointed it out (by noticed the produced blocks with "MEV Block" label in Dora, they then disabled it.

Without MEV...the 150ms config seemed to mostly fix the empty block issue, however we still had some issues due to various errors, most notably a BAL feature toggle leak.

https://github.com/hyperledger/besu/issues/9186
fixed: https://github.com/hyperledger/besu/pull/9189

### **2025-09-19 02:48** BAL fix deployed

### **2025-09-19 09:41** Barnabus announced increase in blob traffic to test 72 blob max. I noticed Besu had low blob numbers (< 6)...

### **2025-09-19 10:10** increase max by type=BLOB from 6 -> 72
https://github.com/ethpandaops/fusaka-devnets/pull/91

  - --tx-pool-layer-max-capacity=10000000
  - --tx-pool-max-prioritized=1000
  - --tx-pool-max-prioritized-by-type=BLOB=72
  - --block-txs-selection-max-time=150

We saw a small number of 72 blob blocks, mostly with lighthouse and a couple with teku - all full nodes, no supernodes produced 72 blobs, this screenshot was taken ~11 hours after blob traffic increased:


Lots of blocks still had < 6 blobs.


Memory usage during the `--tx-pool-max-prioritized-by-type=BLOB=72` timeframe seemed OK (the uptick of blob tx count right at the end)...


### 2025-09-19 21:00 Barnabus reported "Attempting to access closed worldstate" stacktrace

We had also seen this stacktrace while investigating the BAL issue, but this confirms it remained after the BAL fix was in. I suspect it appeared after changing the max-selection-time and the block was getting interupted but haven't verified that.

2025-09-19 21:02:11.671+0000 | EthScheduler-BlockCreation-39 | WARN  | BonsaiSnapshotWorldStateKeyValueStorage | Attempting to access closed worldstate
java.lang.Throwable: Attempting to access closed worldstate
	at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.isClosedGet(BonsaiSnapshotWorldStateKeyValueStorage.java:63)
	at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.storage.BonsaiSnapshotWorldStateKeyValueStorage.getAccount(BonsaiSnapshotWorldStateKeyValueStorage.java:80)
	at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:391)
	at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.loadAccount(PathBasedWorldStateUpdateAccumulator.java:318)
	at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.java:304)
	at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.java:57)
	at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:58)
	at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:30)
	at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.getAccount(AbstractWorldUpdater.java:118)
	at org.hyperledger.besu.evm.worldstate.WorldUpdater.getOrCreate(WorldUpdater.java:76)
	at org.hyperledger.besu.evm.worldstate.WorldUpdater.getOrCreateSenderAccount(WorldUpdater.java:88)
	at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:223)
	at org.hyperledger.besu.ethereum.blockcreation.txselection.BlockTransactionSelector.processTransaction(BlockTransactionSelector.java:585)
	at org.hyperledger.besu.ethereum.blockcreation.txselection.BlockTransactionSelector.evaluatePendingTransaction(BlockTransactionSelector.java:463)
	at org.hyperledger.besu.ethereum.blockcreation.txselection.BlockTransactionSelector.evaluateTransaction(BlockTransactionSelector.java:425)
	at org.hyperledger.besu.ethereum.eth.transactions.layered.LayeredPendingTransactions.selectTransactions(LayeredPendingTransactions.java:332)
	at org.hyperledger.besu.ethereum.eth.transactions.TransactionPool.selectTransactions(TransactionPool.java:601)
	at org.hyperledger.besu.ethereum.blockcreation.txselection.BlockTransactionSelector.lambda$internalTimeLimitedSelection$6(BlockTransactionSelector.java:252)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

### **2025-09-19 UTC EOD** - Devnet-5 shutdown.

### **2025-09-20 10:34** - nimbus fixed 500ms issue
> @tersec:
> fix for the 500ms fcU/getPayload gap should be in

## Discussion Threads

https://discord.com/channels/595666850260713488/1417442195580911626
https://discord.com/channels/595666850260713488/1418553979150798929

empty block thread: https://consensys.slack.com/archives/C08SA44PS4Q/p1757984634133709
nimbus-besu-super-1 OOM thread:
https://consensys.slack.com/archives/C08SA44PS4Q/p1757992692117859