Ethereum Protocol Fellowhsip Week 7-8

I'm really proud of the work I've done so far during this cohort. I find myself looking back to six months ago when I was just begining to poke around the Lighthouse codebase. I was new to the Rust programming language and client development in general. Fast forward to now and I am finally feeling productive programming in Rust. I'm also much more comfortable with certain parts of Lighthouse codebase. There's still a lot to learn, it feels like I'm just barely scratching the surface, but I'm excited about the progress I've made so far.

• Block v3 endpoint progress
• rust-libp2p tracing updates
• modularizing the beacon node
• benchmarking slasher backend implementations

Block V3 endpoint

Over the past month I've worked on implementing the new block v3 endpoint. In an ideal scenario, implementing the v3 endpoint would have simply required some small tweaks to the existing full/blinded v2 endpoint logic. However in the Lighthouse v2 flow, there were some heavy usage of abstract function type parameters. To illustrate the issues I faced, take the following example:

pub fn get_execution_payload<
    T: BeaconChainTypes,
    Payload: AbstractExecPayload<T::EthSpec> + 'static, >(
    chain: Arc<BeaconChain<T>>,
    state: &BeaconState<T::EthSpec>,
    proposer_index: u64,
    builder_params: BuilderParams,
) -> Result<PreparePayloadHandle<T::EthSpec, Payload>, BlockProductionError> {
  // function body
}

To call this function you would need to provide a Payload type at the time of the function call, i.e.

// for a full payload
get_execution_payload<FullPayload>(...)

// for a blinded payload
get_execution_payload<BlindedPayload>(...)

This doesnt work in v3 flow, as the Payload type can be either a Full or Blinded payload depending on the state of the network, block profitability, builder availability etc. etc. (see the spec for more details).

The simplest approach seemed to be creating a separate v3 flow, so that's where I began. I duplicated much of the existing v2 flow and eliminated all instances of abstract function type parameters. This required modifying the return types of several functions, since their original return types were dependent on these type parameters. I needed to have these new functions return either a "Full" or a "Blinded" payload. I introduced some useful enums to make this possible.

In the example above, the return type of get_exectution_payload is

Result<PreparePayloadHandle<T::EthSpec, Payload>, BlockProductionError>

// type definitions
pub type PreparePayloadHandle<E, Payload> = JoinHandle<Option<PreparePayloadResult<E, Payload>>>;
pub type PreparePayloadResult<E, Payload> = Result<BlockProposalContents<E, Payload>, BlockProductionError>;

In order to support the v3 flow I made the following changes

Result<PreparePayloadHandle<T::EthSpec>, BlockProductionError>

// updated type definitions
pub type PreparePayloadResult<E> = Result<BlockProposalContentsType<E>, BlockProductionError>;
pub type PreparePayloadHandle<E> = JoinHandle<Option<PreparePayloadResult<E>>>;


pub enum BlockProposalContentsType<E: EthSpec> {
    Full(BlockProposalContents<E, FullPayload<E>>),
    Blinded(BlockProposalContents<E, BlindedPayload<E>>),
}

I eventually got the new v3 flow to work, but now we had two separate block production flows with a ton of duplicated code. In order to clean up this mess, I consolidated the v2 and v3 flows. I created a new enum that is fed into the block production flow to indicate wether this is the new v3 flow or the existing blinded/full v2 flow

pub enum BlockProductionVersion {
    V3,
    BlindedV2,
    FullV2,
}

I'm now finally at a point where both the new and old flows are consolidated and passing all tests. I opened a pull request and am awaiting feedback from the Lighthouse team.

rust-libp2p tracing updates

Theres still a lot of work to be done here. I got a ton of feedback from the libp2p team. I'm going to dedicate the next few weeks to really cleaning things up here.

Modularizing the beacon node backend

I've sucesfully modularized the beacon node backend. It should now be relatively simple to switch between different database implementations.

PR: https://github.com/sigp/lighthouse/pull/4718

Theres an existing KeyValueStore trait that already does some abstraction, however the Lighthouse codebases assumes the KV store is always LevelDB.

I created a BeaconNodeBackend type that implements the KeyValueStore and ItemStore traits. I then replaced all references of LevelDb to the new BeaconNodeBackend type. Within the BeaconNodeBackend type I used the cfg macro which should allow us to switch between different database implementations via config changes.

Benchmarking Slasher backend implementations

We're going to start benchmarking the slasher backend implementations to see if they are competitive.

From Michael Sproul

Now that we have a couple of new backends in the works, namely Redb and Sqlite, it would be great to see a performance shoot-out. For the slasher the key metric is the time taken to process each batch, which is recorded in Prometheus as slasher_process_batch_time. We have a Grafana dashboard for this here: https://github.com/sigp/lighthouse-metrics/blob/master/dashboards/Slasher.json

A good test would be to start-up a slasher on the same machine with each of the backends – LMDB, Redb, Sqlite – and compare the performance. To prevent them competing for CPU and I/O it's probably best to run the slashers one at a time (not concurrently). Running each one for a few hours should be enough to get a feel for its performance. If one or both of the new backends are competitive in this test we can then look into running a longer test (2 weeks+) in which we can measure the total disk usage and the long-term performance.

I have a shared execution node on mainnet (running eleel) which I could grant access to for this experiment. If this is something you're interested in doing @eserilev, DM me your SSH pubkey on Discord.

I'm really excited to see what results we get from these benchmarks!!