Ethereum Protocol Fellowship Final Update

Abstract

As a participant in cohort four of the Ethereum Protocol Fellowship, my goal was to contribute to Lighthouse, an Ethereum consensus client written in Rust. Specifically I was focused on adding new database implementations to the slasher backend, modularizing the beacon node backend, adding block v3 endpoint functionality to the Beacon API, and providing a continous stream of contributions to the Lighthouse codebase. I also decided to contribute to rust-libp2p by adding tracing + structured logging.

The slasher is a piece of software that deteces "slashable" events from validators and reports them to the protocol. The Lighthouse slasher has several existing database implementations with batch processing times of roughly one second. There was interest in adding additional db implementations, namely Redb and Sqlite. I was able to reach 5 second batch times with the redb implementation and 23 second batch times with the sqlite implementation. Theres still some more optimization work that can be done to reduce batch times further.

The beacon node is at the core of the Ethereum proof-of-stake protocol. It is responsible for running the beacon chain, and uses distributed consensus to agree on blocks both proposed and attested on by validators in the network. Beacon nodes communicate their processed blocks to their peers via a peer-to-peer network, which also manages the lifecycle process of active validator clients. Lighthouses beacon node backend is setup to use LevelDB, an on-disk key value database store. Parts of the Lighthouse codebase assume the database is always LevelDB which makes it difficult to add other database implementations. I spent some time modularizing the beacon node backend and abstracting over the current LevelDB implementation. The changes here should hopefully make it easier to switch between different database implementations within the beacon node.

The block v3 endpoint is the new block production endpoint that is introduced as part of the Deneb upgrade. The endpoint is used by validator clients to produce an unsigned block. In the old block v2 endpoint, the validator client would have to specificy wether they want to use the builder relay or the local exectuion client when requesting a new block. This added unecessary complexity to the validator client and resulted in two seperate endpoints, one for the builder relay, i.e. blinded block endpoint and one for the local exeuction client, i.e. full payload end point. The new v3 endpoint consolidates the two v2 endpoints and removes the need for the validator to specify wether they want to fetch blocks from the builder relay vs the local execution client.

rust-libp2p is a modular peer to peer networking framework and is core to the Lighthouse consensus client. I helped implement tracing/structured logging to the rust-libp2p library with the aim of providing a richer, more contextualized logging experience. This should improve the rust-libp2p developer experience and allow for better debugging tools. I believe the changes I made here will eventually add benefit to the Ligthouse team as they continue working with the rust-libp2p library.

Throughout the cohort, I also made an effort to continously contribute to the Lighthouse codebase outside of my "bigger" projects. I was able to get 12 seperate contributions merged into Lighthouse and have several other contributions awaiting review. These contributions include adding SSZ to the full block production flow, adding/updating new beacon api endpoints, improving error messages and more.

Slasher database implementations

The slasher backend in Lighthouse currently has support for two databases implementations, LMDB and MDBX. There was interest from the Lighthouse team to increase the number of database implementations that the slasher backend can support. I worked to add support for two new database implementations: Redb and Sqlite

Redb

PR: https://github.com/sigp/lighthouse/pull/4529
Metrics: https://snapshots.raintank.io/dashboard/snapshot/bnIAVIgu0Hl4FY0VbdBdlG5i4tinFhIf

Redb is a simple, portable, high-performance, ACID, embedded key-value store. Adding Redb to the slasher was relatively straightforward. However making it performant took some significant effort. I had to deal with some good ol' fashion Rust borrow checker issues and made some changes to the underlying slasher db abstraction layer to make use of some Redb specific features (i.e. delete_while). We were able to get batch processing times to roughly 5 seconds, which is 'on par' with the slashers existing database implementations. Theres still some additional optimization opportunties that may exist here, and I'm also spending some time cleaning up this PR before requesting another review

Sqlite

PR: https://github.com/sigp/lighthouse/pull/4666
Metrics: https://snapshots.raintank.io/dashboard/snapshot/hIwPCJmC26Yjg2REEAdb0ueJFPpyNz22?orgId=2&refresh=5s

Sqlite is an embedded relational database implementation. Adding Sqlite to the slasher was also relatively straightfoward. However building a performant database implementation was a different story. My initial implementation was unusable, batch times were at around 17 minutes! Eventually aftter a few rounds of optimizations I got batch times down to roughly 23 seconds. Theres still some changes that can be made here to potentially get batch times down even further. Some of the optimizations I made here include

• using a 'barebones' db schema (i.e. no ID primary key)
• using prepare_cached which prepares a SQL statement for execution, returning a previously prepared statement from the cache if one is available.
• set pragma journal_mode: wal
• set pragma synchronous: NORMAL
• set pragma locking_mode: EXCLUSIVE
• set pragma cache_size: 10000
• using r2d2 connection pool

Modularize beacon node backend

PR: https://github.com/sigp/lighthouse/pull/4718

Lighthouse's beacon node backend uses LevelDB. LevelDB is quite dated and there will come a time where a new database implementation will be needed. I worked to abstract away the LevelDB implementation so that other databases could be added more easily in the future. The PR is ready to be reviewed and has passed all test cases. I assume that once reviewed there will be some changes that will be requested by the Lighthouse team. Theres also some big changes to the beacon node coming in with the move to "tree-states", so this PR may not be reviewed until the tree-state stuff is merged first.

Block v3 endpoint

Main PR: https://github.com/sigp/lighthouse/pull/4629
VC PR: https://github.com/sigp/lighthouse/pull/4813

The Deneb spec has introduced a new block generation endpoint for validators called "block v3":

Spec: https://github.com/ethereum/beacon-APIs/pull/339

Previously there were separate endpoints for returning blinded and unblinded blocks. This new endpoint combines the old endpoints, which helps reduce some complexity from the validator client.

Most of the effort here was related to refactoring the existing block production flow in Lighthouse. In the old flow there was a payload type parameter that had to be specified at the start of the block production flow. This payload type parameter propogated down to other function calls and dicated wether a blinded or full payload was returned.

In the new flow, the payload type was determined much farther down the function call list and so there was a need to remove the payload type parameter. Since Rust has a pretty strict type system, there was also a need to add several new enums and consolidate logic to handle the new and old flows.

Finally I had the task of merging my block v3 changes with the rest of the upcoming deneb changes. The deneb changes were being built on a seperate branch and were merged into the codebase before my changes were finalized. Once the deneb changes were merged I had to deal with some pretty significant merge conflicts. Resolving those merge conflicts gave me a better understanding of some of the upcoming deneb changes. I also found some small additional opportunties for refactoring!

Tracing in rust-libp2p

PR: https://github.com/libp2p/rust-libp2p/pull/4282

rust-libp2p is designed as an async, event-based system. This makes it fairly hard to add rich logs in user land because peer information is not always available. This is where tracing comes in.

Tracing: A scoped, structured logging and diagnostics system.

Tracing allows rust-libp2p to offer more detailed logs for users. These “traces” contain more context than traditional logging would.

Tracing offers the ability to also record "span"s. A span lasts until its dropped and describes the entire duration that it is active for. All logs (in tracing term "events") are hierarchically embedded in all parent-spans. We introduced several spans:

• On debug level: One for new_outgoing_connection, new_incoming_connection and new_established_connection
• On debug level: Connection::poll, Swarm::poll and Pool::poll
• On trace level: NetworkBehaviour::poll for each implementation of NetworkBehaviour
• On trace level: ConnectionHandler::poll for each implementation of a protocols ConnectionHandlers

My work to add tracing to rust-libp2p can be summarized as:

• migrating the entire workspace from log to tracing
• adding several spans to interesting functions
• showcasing how to ingest this data into Jaeger

I received a TON of help from the rust-libp2p team, specifically Thomas. I am very grateful for their support and I learned a lot by working with them. This PR touched almost every single file in the rust-libp2p repository!

Continous stream of contributions

From what I’ve seen in my short time contributing to Lighthouse, Ethereum protocol developers are constantly working on various issues simulataneously. In order to mimic the daily life of a protocol developer I worked on more than one or two issues. I attempted to be a frequent and productive contributor. I was able to get 14 separate pull requests merged, 12 into Lighthouse and 2 into rust-libp2p. Some of my contributions included:

• Adding SSZ across the full block production flow
• Adding the expected withdrawals endpoint
• Adding SSZ tests for the block v3 endpoint
• Improving the readability of transport connection errors
• Adding additional linting rules which helped reduce lines of code and complexity to both Lighthouse and rust-libp2p
• Removing unecessary write lock in block_verification
• Updating the node health endpoint
• Adding a debounce mechanism to logs
• Keeping the payload cache idempotent
• Simplifying calcluate_committee_fraction
• Small documentation/help text improvements

I also have 5 other pull requests open that are pending a review or need some additional work. I am proud of the work I was able to accomplish, it was a great feeling to see some of my contributions make it into production.

Conclusion

I am proud of the contributions I made during the Ethereum Protocol Fellowship. I was able to dive into some larger tasks and make good progress, provided a continous stream of contributions to Lighthouse, worked on rust-libp2p and in general became a productive contributor within the Ethereum ecosystem. I learned a new programming language (Rust), I gained a ton of insight and context into the consensus side of the Ethereum protocol, I learned about the libp2p framework, and got to work with some really intelligent people.

Post EPF Plans

My post-EPF plans are to continue contributing to Lighthouse and rust-libp2p. I hope to eventually find a full time role as a core protocol developer. I understand that the path to a core developer can take time and I will need to continously work towards that goal. Some of my planned next steps include:

• Finish up adding new slasher database implementations
• Finish up the work to modularize the beacon node backend and work towards adding another database implementation
• Pick up more complicated tasks within the rust-libp2p repo
• Pick up more complicated tasks within the Lighthouse repo
• Continously attend core dev meetings and keep track of new changes to the Ethereum protocol
• Improve my Rust programming skills

I think as long as I continue learning and contributing I will eventually become a core protocol developer. But even if not, I enjoy contributing to the Ethereum ecosystem and will continue doing that.

Shoutouts and Thank Yous

I'd like to thank Mario and Josh for helping faciliate the EPF program and building a community of likeminded individuals that want to learn, grow and contribute to the Ethereum ecosystem. The EPF is such a unique and rewarding opportunity, and you guys are the backbone of this whole thing. Thank you, thank you, thank you!

I'd like to thank the Lighthouse team for providing mentorship, support and lots of code reviews. Michael, Jimmy, Sean, Paul, Age and the rest of the Sigma Prime team: I really enjoyed contributing to Lighthouse, I learned a lot from the experience and look forward to making more contributions in the future. Thank you for being so responsive, helpful and patient with me. I am a better developer today in large part because of your support.

I'd like to thank the rust-libp2p team. I came in knowing next to nothing about rust-libp2p, and tracing in general. Also, my Rust skills were suspect at best. But you guys were extremely patient with me and provided a ton of support along the way. Thank you! Special shoutout to Thomas, you're the best!

Big shout out to my girlfriend for dealing with me during this whole process. Lots of long nights and weekends coding which meant less date nights and quality time for us. I think she's now triggered by the words Rust, Ethereum and Lighthouse. I appreciate all your support and understanding, you helped me so much throughout this process and I really appreciate it. I love you!

Also shout out to my Mom, you the real MVP.