Final dev update (robzajac)

# Final dev update (robzajac) This is the final dev update for [robzajac](https://github.com/robzajac) in [EPF cohort three](https://github.com/eth-protocol-fellows/cohort-three). ## Project summary The [light client protocol](https://mycelium.xyz/research/world-of-light-clients-ethereum) is a way for nodes in low-computation environments to sync to the latest block header without needing to trust a third party. It is built on the introduction of the sync committee in Altair, a randomly selected set of 512 validators (rotating every 256 epochs) which signs the headers of newly attested blocks. The goal of this project was to implement support for the [light client APIs](https://github.com/ethereum/beacon-APIs/tree/master/apis/beacon/light_client) in the consensus client [teku](https://github.com/ConsenSys/teku). These APIs are the method by which light clients poll the data required to participate in their sync protocol. A secondary goal was to make progress in the research area of node RPC incentivization or otherwise design a way to incentivize the serving of light client data. More information on light clients can be found in the [original project proposal](https://hackmd.io/@robzajac/rJG_1gjro). ## Status report In the project proposal, I specified the following goals: 1. Light client endpoints are fully implemented, tested and deployed in teku 2. Demonstrated a sync with a light client against a teku full node 3. Researched and proposed a mechanism to incentivize full nodes serving light client data Below is the progress made towards those goals. Please also refer to the [tracking issue in teku](https://github.com/ConsenSys/teku/issues/4230#issuecomment-1357933576). ### Light client endpoints - `GetLightClientBootstrap` - Implemented [interface and SSZ types](https://github.com/ConsenSys/teku/pull/6384) - Implemented [utility for merkle proofs](https://github.com/ConsenSys/teku/pull/6538) - Implemented [`GetLightClientBootstrap`](https://github.com/ConsenSys/teku/pull/6577) - Remaining work: - Implement the [gossip topic](https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/light-client/p2p-interface.md#getlightclientbootstrap) for light client bootstrap - `GetLightClientUpdatesByRange` - Implemented [interface and SSZ types](https://github.com/ConsenSys/teku/pull/6623) - Implemented [custom octet serialization](https://github.com/ConsenSys/teku/pull/6685) - Wrote [design doc](https://hackmd.io/@robzajac/B1sAY9M3j) with options for implementation - Prototyped [implementation reading historic states](https://github.com/robzajac/teku/commit/0335e7bc2ccf2c9ddb5b2050863cd371caf35c63) - Remaining work: - Light client update internals - Gossip topic - `GetLightClientFinalityUpdate` - Unimplemented - `GetLightClientOptimisticUpdate` - Unimplemented - Miscellaneous - Added [flag to enable light client support](https://github.com/ConsenSys/teku/pull/6652) - Updated SSZ types according to [new Altair definitions (backwards-incompatible change)](https://github.com/ConsenSys/teku/pull/6692) ### Light client sync - Synced a client pair with an archive mode teku for testing with historic states - Installed and ran [helios](https://github.com/a16z/helios), an implementation of the light client sync protocol from [a16z](https://a16zcrypto.com/building-helios-ethereum-light-client/) - Opened issue for an [unchecked error](https://github.com/a16z/helios/issues/199) - Wrote notes on the [testing plan](https://hackmd.io/@robzajac/HkYxAMWhj) and how we can leverage helios for testing of the light client APIs in teku - Remaining work: - Rerun helios with an updated teku (including the APIs) to demonstrate a light client sync ### Full node incentivization Given the complexity of this research area, I did not make headway into designing a new incentivization mechanism. However, the data served for light clients is cheap, and nodes may choose to serve it altruistically for the timebeing. There a few other takeaways: - Solutions for RPC incentivization such as [POKT Network](https://www.pokt.network/) exist. Whether to deploy those solutions is best left for node operators to decide in their own environments. - Proposals such as [rate limiting for the LES protocol](https://blog.chainsafe.io/light-client-task-force-call-1-1aaf559230fb) have previously been explored. - Long-term, the network may be served by moving towards a more decentralized model such as [Portal Network](https://blog.chainsafe.io/light-client-task-force-call-1-1aaf559230fb). ## Project future Based on the current status of the project, I have a few suggestions for how to go forward. ### Redesign with light client table While implementing [light client updates](https://hackmd.io/@robzajac/B1sAY9M3j), we realized that querying for historic states to construct light client data would have several disadvantages: - Only available in archive nodes, which are limited in practice - Less likely to retrieve correct data, according to [`is_better_update`](https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/light-client/sync-protocol.md#is_better_update) - Poor code maintainability, handling edge cases around sync committee period boundaries, pre-Altair states, etc. Therefore, the best way forward is to refactor to read from a [light client table](https://hackmd.io/@robzajac/H1XWzscTj). We would introduce new columns to store `LightClientBootstrap` and `LightClientUpdate`, and populate the data as we sync the chain. This has the advantages of: - Higher data availability, as all nodes that see a block will have the relevant light client data - Data correctness, as we are guaranteed to find the best update - Simpler code maintainability ### Roadmap Based on these changes, an updated roadmap would look like: 1. Implement the light client table in teku - Add new columns - Add logic to populate columns while applying blocks 2. Refactor `GetLightClientBootstrap` and `GetLightClientUpdatesByRange` to read from table 3. Implement `GetLightClientFinalityUpdate` and `GetLightClientOptimisticUpdate` 4. Run a full light client sync with [helios](https://github.com/a16z/helios) ## Self-evaluation In this EPF cohort, I learned a great deal about programming for the Ethereum blockchain, working in a core dev environment, and my own skills as a software engineer. ### Technical progress In the beginning, I wrote an ambitious plan for implementing the light client API endpoints in teku. Down the road, it turned out there was a lot to learn about the details of the Ethereum consensus layer, the internals of the teku implementation, the details of the light client sync protocol, and many other pieces. Ultimately, we wrote an implementation of the API that relies on retrieving historic beacon states and computing light client data from them. We learned that this wouldn't be the best implementation going forward, and that teku is best served moving towards a light client table. For this updated roadmap we wrote a design doc and a plan for how to proceed. I learned a lot about what it takes to plan and execute a large project independently, as well as how unexpected complexity can creep up and how to deal with it. I feel more confident going forward with my ability to tackle large software projects and bring them through to fruition. ### EPF program I benefitted greatly from the structure of the cohort and the EPF program. I appreciated sharing the experience with other EPF fellows from whom I could draw inspiration, and from collaborating with my teku mentor Paul Harris. I felt that the program was incredibly well-organized and that it was an awesome way to learn the details of core dev. ### Personal development I learned a lot about what kind of environments I function in best as a software engineer. While I appreciated having the freedom and flexibility of working independently and scoping my own project in core dev, I learned that ultimately I would benefit from having a team around me with whom I could share ideas and drive a project forward. I learned that I benefit from the structure of a more traditional software company, and that with freedom and flexiblity there also come the tradeoffs of greater responsibility and self-direction. I feel more informed and more confident going forward with seeking environments that best support my creativity and productivity in writing software. ### Closing I would like to thank the cohort organizers Mario Havel and Josh Davis for an incredibly well-run program that gave great insight into working on Ethereum core dev. I'd also like to thank my mentor Paul Harris for his excellent technical support and brainstorming around teku. Finally, I'd like to thank the EPF fellows for creating a supportive and inspiring environment. I'm endlessly grateful for the opportunity to participate in this cohort, and I'm looking forward to seeing what comes next for the program fellows and for the Ethereum blockchain.