Try   HackMD

Final dev update (robzajac)

This is the final dev update for robzajac in EPF cohort three.

Project summary

The light client protocol is a way for nodes in low-computation environments to sync to the latest block header without needing to trust a third party. It is built on the introduction of the sync committee in Altair, a randomly selected set of 512 validators (rotating every 256 epochs) which signs the headers of newly attested blocks.

The goal of this project was to implement support for the light client APIs in the consensus client teku. These APIs are the method by which light clients poll the data required to participate in their sync protocol. A secondary goal was to make progress in the research area of node RPC incentivization or otherwise design a way to incentivize the serving of light client data.

More information on light clients can be found in the original project proposal.

Status report

In the project proposal, I specified the following goals:

  1. Light client endpoints are fully implemented, tested and deployed in teku
  2. Demonstrated a sync with a light client against a teku full node
  3. Researched and proposed a mechanism to incentivize full nodes serving light client data

Below is the progress made towards those goals. Please also refer to the tracking issue in teku.

Light client endpoints

Light client sync

  • Synced a client pair with an archive mode teku for testing with historic states
  • Installed and ran helios, an implementation of the light client sync protocol from a16z
    • Opened issue for an unchecked error
    • Wrote notes on the testing plan and how we can leverage helios for testing of the light client APIs in teku
  • Remaining work:
    • Rerun helios with an updated teku (including the APIs) to demonstrate a light client sync

Full node incentivization

Given the complexity of this research area, I did not make headway into designing a new incentivization mechanism. However, the data served for light clients is cheap, and nodes may choose to serve it altruistically for the timebeing. There a few other takeaways:

  • Solutions for RPC incentivization such as POKT Network exist. Whether to deploy those solutions is best left for node operators to decide in their own environments.
  • Proposals such as rate limiting for the LES protocol have previously been explored.
  • Long-term, the network may be served by moving towards a more decentralized model such as Portal Network.

Project future

Based on the current status of the project, I have a few suggestions for how to go forward.

Redesign with light client table

While implementing light client updates, we realized that querying for historic states to construct light client data would have several disadvantages:

  • Only available in archive nodes, which are limited in practice
  • Less likely to retrieve correct data, according to is_better_update
  • Poor code maintainability, handling edge cases around sync committee period boundaries, pre-Altair states, etc.

Therefore, the best way forward is to refactor to read from a light client table. We would introduce new columns to store LightClientBootstrap and LightClientUpdate, and populate the data as we sync the chain. This has the advantages of:

  • Higher data availability, as all nodes that see a block will have the relevant light client data
  • Data correctness, as we are guaranteed to find the best update
  • Simpler code maintainability

Roadmap

Based on these changes, an updated roadmap would look like:

  1. Implement the light client table in teku
    • Add new columns
    • Add logic to populate columns while applying blocks
  2. Refactor GetLightClientBootstrap and GetLightClientUpdatesByRange to read from table
  3. Implement GetLightClientFinalityUpdate and GetLightClientOptimisticUpdate
  4. Run a full light client sync with helios

Self-evaluation

In this EPF cohort, I learned a great deal about programming for the Ethereum blockchain, working in a core dev environment, and my own skills as a software engineer.

Technical progress

In the beginning, I wrote an ambitious plan for implementing the light client API endpoints in teku. Down the road, it turned out there was a lot to learn about the details of the Ethereum consensus layer, the internals of the teku implementation, the details of the light client sync protocol, and many other pieces.

Ultimately, we wrote an implementation of the API that relies on retrieving historic beacon states and computing light client data from them. We learned that this wouldn't be the best implementation going forward, and that teku is best served moving towards a light client table. For this updated roadmap we wrote a design doc and a plan for how to proceed.

I learned a lot about what it takes to plan and execute a large project independently, as well as how unexpected complexity can creep up and how to deal with it. I feel more confident going forward with my ability to tackle large software projects and bring them through to fruition.

EPF program

I benefitted greatly from the structure of the cohort and the EPF program. I appreciated sharing the experience with other EPF fellows from whom I could draw inspiration, and from collaborating with my teku mentor Paul Harris. I felt that the program was incredibly well-organized and that it was an awesome way to learn the details of core dev.

Personal development

I learned a lot about what kind of environments I function in best as a software engineer. While I appreciated having the freedom and flexibility of working independently and scoping my own project in core dev, I learned that ultimately I would benefit from having a team around me with whom I could share ideas and drive a project forward. I learned that I benefit from the structure of a more traditional software company, and that with freedom and flexiblity there also come the tradeoffs of greater responsibility and self-direction. I feel more informed and more confident going forward with seeking environments that best support my creativity and productivity in writing software.

Closing

I would like to thank the cohort organizers Mario Havel and Josh Davis for an incredibly well-run program that gave great insight into working on Ethereum core dev. I'd also like to thank my mentor Paul Harris for his excellent technical support and brainstorming around teku. Finally, I'd like to thank the EPF fellows for creating a supportive and inspiring environment. I'm endlessly grateful for the opportunity to participate in this cohort, and I'm looking forward to seeing what comes next for the program fellows and for the Ethereum blockchain.

Last changed by 
 
robzajac
0
308

Read more

Light client consensus layer development

This is a project proposal for EPF Cohort Three by robzajac. The goal of this project is to enable greater decentralization and trustless access to the Ethereum consensus layer by implementing the light client protocol in teku. This will increase the availability of light client data in the CL and make it easier for light clients to sync to the latest block header without needing to trust a third party. Motivation The light client protocol is a way for clients in minimal computational environments to sync to the latest block header without needing to trust the data provider (e.g. a node hosted on Infura). For more on the protocol and its motivation, see background reading. This protocol describes both the light client itself, and a "light client server" which provides the data needed for the light client to execute its sync protocol. While there have been several light client projects in development (e.g. recently announced RPC-proxy light clients Kevlar and Helios), there are not as many implementations of the light client server. Currently, the majority providers for light client data in beacon nodes are the consensus clients Nimbus and Lodestar. Serving this data is done altruistically, as light clients do not contribute any computation to the network. In order to contribute to greater availability of this data, as well as expand client diversity, more consensus clients should implement the light client protocol.

Feb 23, 2023
teku light client table

As described in the latest dev update, persisting light-client data on disk and serving requests with a table lookup will be the best way to provide light client support in teku. It has the advantages of: Providing the best light client update per is_better_update, given that light client updates can be compared and saved per sync-committee period Providing the highest rate of data availability, as all nodes that see a block will be able to serve light client data corresponding to the block (previously an archive node would be required for historic states) Code maintainability, as the logic of updating when we sync and serving from the table is much simpler than computing the slot required for light client updates and retrieving the relevant beacon states on-the-fly This doc will briefly describe the interface and details of this table in teku. Light client data schemas Add two columns with type:

Feb 17, 2023
Dev Update Week 15

This week I focused mainly on two things: Worked on getting my mainnet besu + teku pair up and running on a node provided by Mario. Initially had some issues with the configurations for the clients, but now the EL & CL syncs are both complete with teku up-to-date in archive mode.Archive mode in teku is necessary for accessing historic states, which are required to derive light client data. We can test the light client API already implemented (light client bootstrap) on this mainnet client pair. Worked on the light client updates by range API and the light client data availability problem. Light client data, at least for LightClientUpdate, is derived from beacon state. The light client API specifies that servers should provide light client data for a historic period of ~5 months. Most nodes, at least in teku, run in prune mode, meaning that they do not store any state beyond the latest finalized. Archive nodes have a significant storage requirement that most nodes will not provide (especially archive nodes running with >= 5 months of historic data, not just e.g. from the latest checkpoint).

Feb 12, 2023
Light client updates design in teku

Background The GetLightClientUpdatesByRange API requests the light client update (LCU) for a given sync committee period start_period and for count periods after start_period. LCUs are derived from beacon state. They are assigned to sync committee periods (8192 slots) based on their attested_header.beacon.slot. There are a few other considerations for what makes a valid LCU. There are a few heuristics to determine the "best" LCU for a sync committee period. According to the spec: Full nodes SHOULD provide the best derivable LightClientUpdate(according to is_better_update) for each sync committee period covering any epochs in range [max(ALTAIR_FORK_EPOCH, current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS), current_epoch] where current_epoch is defined by the current wall-clock time. In the definition of is_better_update, two updates are compared based on sync committee participation, finality, slot, etc.

Jan 31, 2023
Read more from robzajac
Published on HackMD