# EPF Final Report

## tl;dr

1. [Project Proposal](https://hackmd.io/@Odinson/Sk5w04lVxg)
2. [Pectra Holesky Incident](https://blog.sigmaprime.io/pectra-holesky-incident.html)
3. [Project PR](https://github.com/sigp/lighthouse/pull/7803)
4. [Conclusion](#Conclusion)

## Abstract

Being a participant in the Ethereum Protocol Fellowship cohort 6, the project, I worked on dynamic memory-aware caching implementation in Lighthouse, an Ethereum Consensus Layer Client in Rust. Under the mentorship of [Michael Sproul](https://github.com/michaelsproul) and [Dapplion](https://github.com/dapplion), I developed a memory aware caching, on top of the current count based implementation, which was motivated by an incident on the Holesky testnet during the Pectra upgrade.

### Pectra Holesky Incident

On February 24th, 2025, shortly after the Pectra upgrade on Holesky, an invalid block was accepted by Geth, Nethermind, and Besu due to a misconfigured `depositContractAddress` value. Since these three clients represented a super-majority of the network, most validators attested to this invalid block, causing it to become part of a justified checkpoint. This triggered a prolonged period of non-finality where the network couldn't finalize new blocks, activating "inactivity leak" mode that quadratically increased penalties for validators. For Lighthouse specifically, this created severe operational challenges: the inactivity penalties added ~70MB of data to each Beacon State at epoch boundaries (memory optimizations weren't applied to the `inactivity_scores` field), disk usage ballooned from 20GB to over 1TB as the hot database couldn't perform normal migrations to the more efficient freezer database, and nodes experienced out-of-memory crashes due to expensive state lookups when serving non-finalized blocks to peers attempting to sync. The incident exposed critical gaps in client resilience during adverse network conditions and prompted the Lighthouse team to implement numerous optimizations for handling non-finality scenarios.

### Caching implementation in Lighthouse

Currently, Lighthouse uses a fixed-size LRU cache for beacon state that performs well during normal operations but becomes inefficient during network stress. The cache is bounded by count(default 32 states), which is around 500MB during normal conditions but expands to multiple GBs as Beacon State size increases. This risks validator liveness and network stability, particularly affecting solo stakers with limited hardware resources.

### My Work

#### My implementation

My project focused on implementing dynamic, memory-aware beacon state caching in Lighthouse to address the memory behaviour exposed by the Pectra Holesky non-finality incident. I moved Lighthouse from a rigid, count-based cache (e.g., “128 states”) to an optional, byte limit based caching. At the core of this, I used Milhouse’s `MemorySize` trait (thanks to Michael for previously having this setup, where I just had to make some minor additions) for `BeaconState` inside the `consensus/types` crate, using Lighthouse’s existing macros to traverse all relevant sub-trees and lists. The implementation relies on a differential `MemoryTracker` so that shared `Arc` structures (such as committee caches and other shared sub-trees) are only counted once, avoiding over-counting in different scenarios. On top of this, I extended the `StateCache` to implement an optional `max_cached_bytes` field, so that the cache can be capped either by count alone (current behaviour) or by count plus a memory ceiling. I didn't want to calculate memory on every operation as that would be too expensive. Instead, I track how many inserts we've done, and every N inserts, it does a full measurement pass. This uses a single`MemoryTracker` to add up all the cached states and updates the`store_beacon_state_cache_memory_size` metric. When a memory limit is configured and the measured usage exceeds it, the cache prunes states in batches whose size is derived from the existing `state_cache_headroom` parameter and clamped to a safe range, re-measuring between batches until usage falls back under the limit. The delete paths remain simple (they just drop entries). I also exposed the new behaviour via a `--state-cache-max-mb` CLI flag, which is disabled by default to preserve current behaviour unless operators explicitly opt in, and added dedicated metrics for both whole-cache and per-state memory sizing latencies to make performance overhead visible in production. Overall, this turns Lighthouse’s state cache into an adaptive, memory-aware component that can be tuned to the constraints of solo stakers and behaves robustly when beacon states bloat during extended periods of non-finality.

### Metrics and Benchmarks

To evaluate the new memory-aware state cache, I tried to recreate Holesky conditions by running Lighthouse and hammered it with HTTP requests to force the
state cache to fill up with large beacon states and compared behaviour with and without the byte cap. In all experiments I repeatedly spammed HTTP endpoints to keep the state cache hot. The node configuration was kept constant across the entire time, the only change was whether `--state-cache-max-mb` was set. All measurements were visualized in Grafana using the new metrics `store_beacon_state_cache_memory_size`, `store_beacon_state_cache_size` and the timing histograms for `beacon_state_memory_size_calculation_time`.


Image 1 shows the behaviour of the memory-tracking thread when the node is under continuous HTTP stress. The top panel plots % time spent in the memory measurement code, both for a single item (yellow) and for full size (green). During the benchmark the per-item thread time fluctuates between roughly 20–80%, with occasional spikes close to 100% at the moments where a recompute covers many states, but it remains bounded and does not saturate the core. The bottom panel plots total node memory usage (yellow) versus the portion attributed to the state cache by the new memory gauge (green). As the HTTP spam ramps up, node memory rises into the 32–36 GiB range while the state cache slowly grows to a few GiB, but the state cache memory size is constant at around 8 GiB, with occasional hikes followed by dips, when the eventual recomputation kicks in.


2nd image is the same workload without enabling `--state-cache-max-mb`. The dashboard only has access to the old `store_beacon_state_cache_size` metric, so the top row shows `State Cache Size = 128` but `State Cache Memory` is reported as `N/A`. In the bottom panel, “Memory usage: Node vs StateCache”, only the node-wide memory line (green) is available, which climbs from roughly 42 GiB to ~49 GiB as the cache fills while HTTP requests are spammed.


The 3rd image shows the same benchmarks, but this time with the memory cap enabled via `--state-cache-max-mb=8192`(≈ 8 GiB). The dashboard now reports `State Cache Memory ≈ 7.51 GiB` and `Total Node Memory Usage ≈ 40.2 GiB`, and the cache size hovers around 60–70 entries instead of sticking at 128. The thread-time sections show that whole-cache recomputation takes around 15–18 s on this machine, with a per-item cost in the 300–350 ms range and about 40–45% of a single core spent inside the memory tracker while recomputations are happening. In the bottom “Memory usage: Node vs StateCache” graph, the green line stays just under 8 GiB as expected, while the yellow node-memory line climbs but stays substantially below the ~48–49 GiB seen in the uncapped run.

**CLI metrics snapshots**

$ curl -s http://127.0.0.1:5054/metrics \
  | egrep -i 'store_beacon_state_cache_memory_size|store_beacon_state_cache_size'

store_beacon_state_cache_memory_size 7380783997
store_beacon_state_cache_size 37

store_beacon_state_cache_memory_size 8015966929
store_beacon_state_cache_size 28

store_beacon_state_cache_memory_size 7576144761
store_beacon_state_cache_size 17

These samples show the `store_beacon_state_cache_memory_size` gauge oscillating between roughly 6.87 GiB, 7.47 GiB and 7.06 GiB while the cache size shrinks from 37 to 17 entries. This is exactly the expected behaviour with `--state-cache-max-mb=8192`: the cache keeps pruning states until its measured memory sits just below the configured 8 GiB ceiling, even if that means discarding a large fraction of the cached entries.

### Conclusion

With all of this together, the benchmarks proved the approach works: yes, there's CPU overhead for measuring memory (about 15-18 seconds per full scan), but it's bounded and predictable. More importantly, operators now have a `--state-cache-max-mb` flag they can actually use to prevent OOM crashes and keep Lighthouse within a comparatively safer space during the kind of stressed, non-finalizing conditions that broke Holesky.

### Future plans post cohort

After the fellowship is over, I am looking forward to keep contributing to Lighthouse, try to get some legwork done, and find some other stuffs to work on. I have spent quite some amount of time fixing different issues ranging from validator client to my own project, state cache, for which I have explored a good amount of the codebase, and I plan on keep doing that. The main goal is to become a core Ethereum dev, so I will also keep looking into and researching the latest work in the ecosystem, attending the calls ACDC/E, and also work on my rust skills.

### Shoutouts and Acknowledgement

Firstly I would like to thank Mario and Josh for giving all of us this opportunity through the cohort, and ever since the beginning of the study group, organizing the lectures, and the fellowship itself, alongside the office hours, for us to learn, work and gain experience in our path to potentially become a core Ethereum developers.

Next I would like to thank Michael and Dapplion for continuously reviewing my work amidst being caught up with their own responsibilites, pointing out edge-cases I had missed, and suggesting cleaner implementations, testing out the work in real time. I am grateful to have both of them and have been able to learn a lot in these past few months.
Other than them, Eitan, who is an amazing mentor and person to talk to, and get to learn about the consensus. I have been able to gain so much of knowledge from him in various aspects while solving some issues in Lighthouse. Because of all of them, I have been able to become a better dev, think better or write better code.

Finally, to all the fellow fellows with whom I got to discuss so much and learn a lot, and all the core devs in the ACDC/E which is the best place to be at and get to know what are the aspects where work is being done right now or to be done in the future, its amazing to be a part of this ecosystem and I hope to keep working like I currently do and do my best to contribute meaningfully. Thank you!