EPF Final Dev Update #16

--- tags: epf,ethereum --- # EPF Final Dev Update #16 ## Project Abstract My goal of this project is to contribute to the implementation and testing of [EIP-4844](https://www.eip4844.com), and learn about the end-to-end protocol development process. During the exploration phase, we reached out to the Lighthouse team to discuss about contributing to EIP-4844, and eventually we decided towork on the [Builder API updates for EIP-4844](https://hackmd.io/@jimmygchen/B1dLR74Io). The reason is that it wasn't on the critical path at the time and the spec discussion hadn't been started, so we had an opportunity to learn about protocol feature development and contribute from the very beginning. Originally we thought the scope for Builder API updates was not big enough for the duration of the program (it turned out that we were completely wrong! :sweat_smile:), so we also to incldued a bunch of smaller tasks as part of our project. Details of the project plan and roadmap can be found here: [Project: EIP-4844 Consensus Client](https://github.com/eth-protocol-fellows/cohort-three/blob/master/projects/4844-consensus-client.md). I did a short presentation at the end of the EPF, and the slides can be found [here](https://docs.google.com/presentation/d/1TZSbcW4oE4LO7BRQ9IyWT76viSJsl50FoXIPuDdk_1g/edit?usp=sharing). It's a less wordy version of this update if anyone is interested. ## Project Status **TL;DR:** The [builder API update task](https://github.com/sigp/lighthouse/issues/3689) is _still a WIP_ due to ongoing spec discussions over the last few months. However it's progressing well and it seems to be on the right track. While waiting for feedback on the spec update PRs, I spent some time working on a few other areas, and I’ve grouped them into a few categories below. I didn't end up working on the Blob Explorer as I already had a few things on my plate, but [@0xGabi](https://github.com/0xGabi) headed towards that direction, and I'm looking forward to see his work! ### Builder API Updates for Blobs (and Capella) Issue: https://github.com/sigp/lighthouse/issues/3689 Design: https://hackmd.io/a8eyJ505Q62gXMbYr0v7_w As mentioned above, there are still a few outstanding PRs which are listed below. The spec update PRs didn't get much attention during the early stage, as most client teams were focusing on getting the initial 4844 features implemented and getting the clients ready for [devnet v3](https://notes.ethereum.org/@timbeiko/4844-devnet-3) at the time. Nevertheless, it was good to have the PRs out there early to kick start the discussion. After **@0xGabi** and I drafted the spec updates, I jumped onto a [draft implementation](https://github.com/sigp/lighthouse/pull/3808) in Lighthouse. At the time of writing this, the [consensus spec for Deneb](https://github.com/ethereum/consensus-specs/tree/dev/specs/deneb) is pretty much finalised and I am in the process of revising the [builder-specs update for Deneb](https://github.com/ethereum/builder-specs/pull/61), and I will be continuing on these PRs in the coming weeks. #### Pull requests | PR Link | Description | PR Merged? | | ------------------------------------------------------------------------------------ | -------------------------------------------------- |:----------:| | [beacon-APIs#271](https://github.com/ethereum/beacon-APIs/pull/271) | Add Capella & 4844 types to Beacon API | ✅ | | [builder-specs#60](https://github.com/ethereum/builder-specs/pull/60) | Builder spec update for Capella | ✅ | | [builder-specs#61](https://github.com/ethereum/builder-specs/pull/61) | Builder spec update for Deneb | 🚧 | | [ethereum-consensus#168](https://github.com/ralexstokes/ethereum-consensus/pull/168) | Rust library update: add Capella types and presets | ✅ | | [ethereum-consensus#170](https://github.com/ralexstokes/ethereum-consensus/pull/170) | Rust library update: add 4844 types and presets | 🚧 | | [mev-rs#77](https://github.com/ralexstokes/mev-rs/pull/77) | Rust library update: add 4844 types | 🚧 | | [lighthouse#3808](https://github.com/sigp/lighthouse/pull/3808) | Lighthouse draft implementation | 🚧 | ### EIP-4844 Functioanlities & Testing The below are other 4844 tasks that I've worked on. The `beacon-API` spec update ([beacon-APIs#282](https://github.com/ethereum/beacon-APIs/pull/286)) has taken a bit longer, as we are waiting for `Capella` to be released first before we can merge the `Deneb` changes in. The blob signing PR is a recent one that I have started, but also depends on a [spec change](https://github.com/ethereum/beacon-APIs/pull/302) that is currently under discussion. I'll be working on these in the next couple of weeks. | PR Link | Description | PR Merged? | | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |:----------:| | [eip4844-interop#77](https://github.com/Inphi/eip4844-interop/pull/77) | Added Lighthouse to `eip4844-interop`, a testing tool created by [@Inphi](https://github.com/Inphi) to help with 4844 interop testing and testing some of the key features of EIP-4844. | ✅ | | [devnet-v3](https://github.com/jimmygchen/devnet-v3) | docker-compose setup for devnet-v3 (Lighthouse + Geth) | ✅ | | [lighthouse#3830](https://github.com/sigp/lighthouse/pull/3830) | Implement a Beacon API endpoint in Lighthouse to support retrieval of blobs, and endpoint for users and testing | ✅ | | [beacon-APIs#282](https://github.com/ethereum/beacon-APIs/pull/286) | Add the above download blob endpoint to the standard spec | 🚧 | | [lighthouse#3979](https://github.com/sigp/lighthouse/pull/3979) | Implement Lighthouse Blob signing APIs and VC signing functionalities | 🚧 | ### Lighthouse: Rate limiting historical backfill sync I heard about this [backfill sync issue](https://github.com/sigp/lighthouse/issues/3212) from a meeting with the Lighthouse team and got interested into working on this problem. At the time of writing, the implementation has been completed, and I am doing some testing to finalise the configuration on number of batches to process per slot. The change is likely to be included in the next release after 3.5.0. Some relevant links to my work: - Notes from investigating Lighthouse code base: https://hackmd.io/RUCGCQ5HSM66uo6aeYSn8A - Issue: https://github.com/sigp/lighthouse/issues/3212 - Benchmarking: https://hackmd.io/h9RvBwR-RiuzX1Okt-Jmnw | PR Link | Description | PR Merged? | | --------------------------------------------------------------- | ------------------------------------ |:----------:| | [lighthouse#3936](https://github.com/sigp/lighthouse/pull/3936) | Rate limiting historic backfill sync | 🚧 | ## Future of the project This project turned out to be a bunch of smaller tasks across many different areas rather than a standalone project, which was also great because I think I got a good feel for what being a core dev is like from this project. My plan is to wrap up the outstanding PRs, and continue to contribute to the protocol and ecosystem as a full time member of the core dev team. ## Self evaluation Although I didn't get to "finish" the project, I believe I've achieved all of [my goals](https://github.com/eth-protocol-fellows/cohort-three/blob/master/projects/4844-consensus-client.md#goal-of-the-project) that I set out for this project. I've learnt a lot throughout the EPF program, not just the technical knowledge, collaboration approach, but also what I truely enjoy doing and helped me figure out my next steps after EPF - big thanks to everyone who helped me along the way! Things I've learned: - Ethereum basic knowledge, especially in EIP-4844, but there's so much more 🤩 - Improved in writing, both technical and non-technical, from writing weekly updates and doing most of the comms asynchornously on GitHub :writing_hand: - Basics of the Rust lanaguage. It's been fun and can't wait to dig deeper :crab: :hole: - In terms of working on spec: - Spec changes can take a while and go back and forth, but it's valuable to gather inputs from various teams and do some experiments on different approaches. I've also found it valuable to start the discsussion early, either by creating an issue / a draft PR on the spec. - Sometimes it's worth implementing a PoC before spec is drafted or finalised. Even though it could result in an early implemention being completely re-worked or even thrown away - which doesn't seem uncommon, but it's almost never a complete waste because of the valuable learnings and potetntial issues we may discover. - Reach out to R&D channel for help / questions, people are usually super helpful and try to help when they can. I've really enjoyed the challenges, the learnings and working with other core devs. It's an amazing community - everyone is welcoming, collaborative, and respectful of each other, and it seems that we all share a common goal to make Ethereum better and deeply care about creating a positive impact to not just the Ethereum ecosystem but the wider society. For me, it's been very satisfying to be able to work on this and learn about core development full time for the last couple of months. As a dev who's been in the industry for quite some time (13+ years) and have a family dependent on my income, it was a big decision for me to choose to participate in EPF full time over getting a well-paid and comfortable job - thanks to my wife for the full support! And I have to say I'm glad that I stepped out of my comfort zone and be the dumbest person in the room again, and most importantly finding something that get me up excited every morning (or at least most :sweat_smile: )! So it's a wrap! It's been a wonderful journey and pleasure to be part of the Ethereum Protocol Fellowship and cohort. I'm excited to continue to build Etherum and can't wait to the start of my next chapter @ [Sigma Prime](https://sigmaprime.io)! :blush: :v: Thanks to the Lighthouse team, especially Sean & Diva, for their patience and help during EPF! :pray: Big thanks to Mario, Josh, and EF for making all this happen! and thank you all for reading! Bye :wave: