Try   HackMD

EPF Cohort 3 Dev Update 9 (pavignol)

I didn't post an update last week because I hadn't made much progress on the EPF front and was I busy working on a client for the KZG Ceremony. This week, I have 2 updates:

Added a REST implementation for the GetDuties endpoint

GetDuties is an endpoint that I started working on weeks ago, but because there was a bug with the prysm beacon node REST API, I was unable to complete it. Now that the bug has been fixed, I was finally able to merge this PR.

The GetDuties endpoint aggregates 8 Beacon REST APIs into a single one:

1 call to /eth/v1/validator/duties/attester/{epoch}
1 call to /eth/v1/validator/duties/attester/{epoch+1}
1 call to /eth/v1/validator/duties/proposer/{epoch}
1 call to /eth/v1/validator/duties/proposer/{epoch+1}
1 call to /eth/v1/validator/duties/proposer/{sync}
1 call to /eth/v1/validator/duties/proposer/{sync+1}
1 call to /eth/v1/beacon/states/head/committees?{epoch}
1 call to /eth/v1/beacon/states/head/committees?{epoch+1}

Once all the calls are made, aggregating the information is relatively straightforward. The way the information is presented in the REST API is different from how it is presented in the gRPC API, but with a few hashmaps it's pretty simple to reconstruct it. Because of the high number of Beacon API calls, the tests were the biggest chunk of this work since there are many edge cases are validation cases to test.

Made a C++ KZG Ceremony Client

This task is unrelated to the EPF, but I was interested in building a client for the KZG Ceremony and use this client to send my own contribution. The client is now mostly done and has the following features:

  • Supports Windows, Mac and a wide range of Linux distributions
  • Supports Ethereum and GitHub accounts
  • BLS signing
  • ECDSA signing for Ethereum accounts
  • Transcript verification
  • Contribution verification
  • Lightweight (the executable is around 3MB)

For the sake of language and framework diversity, I decided to build the client with C++. Current implementations are centered around Go, Rust or JavaScript, so it adds some diversity to the list of clients that people can use.

Update on the PRs

There are only 5 endpoints remaining! We've come a long way from the start of this project and it's really nice to see that we're probably going to be able to 100% complete this project. 2 of the remaining tasks are on me, and now that my KZG Ceremony client implementation is done, I'll be able to focus more time in the next 2 weeks on those. I think they will also be relatively easy to implement since they are the stream versions of endpoints that have already been implemented (e.g. GetDuties -> StreamDuties).

Legend:

Symbol State
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Not ready for PR
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
PR opened
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
Merged into develop
Done Owner Endpoint
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol GetDuties
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol GetBeaconBlock
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol ProposeBeaconBlock
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol PrepareBeaconProposer
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol GetAttestationData
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv ProposeAttestation
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv SubmitAggregateSelectionProof
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv SubmitSignedAggregateSelectionProof
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol ProposeExit
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol SubscribeCommitteeSubnets
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu SubmitValidatorRegistrations
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu ValidatorIndex
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu ValidatorStatus
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu MultipleValidatorStatus
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol SubmitSignedContributionAndProof
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu SubmitSyncMessage
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv GetSyncSubcommitteeIndex
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv GetSyncMessageBlockRoot
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol StreamDuties
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol WaitForChainStart
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu WaitForActivation
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol StreamBlocksAltair
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
pavignol DomainData
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu GetFeeRecipientByPubKey
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
manu CheckDoppelGanger
Image Not Showing Possible Reasons
  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported
Learn More →
dhruv GetSyncCommitteeContribution
Last changed by 
 
Patrice Vignola
0
100

Read more

Final dev update (pavignol)

Project summary Before the Beacon REST API was standardized, the Prysmatic Labs team was strongly advocating for the use of a gRPC API instead of a REST API for communication between the validator client and the beacon chain. When it was decided that a REST API should be the standard, the Prysm validator client was already using a bunch of gRPC endpoints to communicate with the beacon chain. This resulted in the Prysm validator and beacon node clients not being standard-compliant and not being able to communicate with VCs and beacon nodes form other teams. Eventually, as third parties wanted to be able to query the beacon chain, Prysmatic Labs started adding more REST endpoints to the beacon node until it became 100% compliant. But since the VC didn't have that chance and was stuck using the gRPC API, it meant that the beacon node also had to keep supporting both the REST and gRPC endpoints. This resulted in a huge maintenance burden. Therefore, the goal of this project was to make the Prysm validator client compliant with the Beacon API standard in order to eventually remove this tech debt and being able to use the VC with beacon nodes made by other teams. For a more details explanation of what this project entails, read the project description. Status report

Feb 27, 2023
EPF Cohort 3 Dev Update 8 (pavignol)

This week's update will be relatively short since there are not many endpoints to implement on my end, and some of the endpoints that I have implemented are blocked on PRs that haven't been merged yet. Added Capella support to the ProposeBeaconBlock and GetBeaconBlock REST API implementations Originally, when I added ProposeBeaconBlock and GetBeaconBlock, I wasn't able to add support for Capella since the functionality wasn't ready yet on the beacon node side, and therefore there wasn't even a gRPC implementation on the validator side either. But since the Prysm team has been working hard on implementating withdrawals, it was time to revisit those 2 endpoints and add the missing functionality. Adding Capella support was pretty easy since I have already written all of the boilerplate code, heavylifting and tests a few weeks ago for Bellatrix, so all I needed to do was code the conversion of the withdrawals array between the gRPC and the REST API JSON structs, which looks like this: func jsonifyWithdrawals(withdrawals []*enginev1.Withdrawal) []*apimiddleware.WithdrawalJson { jsonWithdrawals := make([]*apimiddleware.WithdrawalJson, len(withdrawals)) for index, withdrawal := range withdrawals {

Jan 9, 2023
EPF Cohort 3 Dev Update 7 (pavignol)

Here are the Prysm validator features that I’ve been working on last week: Wrote the REST implementation for Validator's GetDuties While the beacon REST API separates the duties retrieval in 3 distinct endpoints (/eth/v1/validator/duties/attester/{epoch}, /eth/v1/validator/duties/proposer/{epoch} and /eth/v1/validator/duties/sync/{epoch}), the Prysm gRPC API does everything in a single function called GetDuties. Moreover, GetDuties automatically retrieves the duties for the current epoch and the next epoch while the beacon REST API endpoints only target a single epoch at a time. For these reasons, this endpoint required combining a lot of endpoints together and massaging the data to make it fit into a single giant data structure containing all duties for epoch and epoch+1. To make sure that the mapping respected all requirements of the gRPC API, a few rules and details have to be kept in mind: Even validators that don't have duties for a given epoch should be returned if they were part of the request. This includes all validators in the pending state. Attester and sync duties should be retrieved for both epoch and epoch+1 Proposer duties should only be retrieved for the current epoch (not epoch+1)

Dec 31, 2022
EPF Cohort 3 Dev Update 6 (pavignol)

Here are the Prysm validator features that I’ve been working on last week: Added a REST implementation for Validator's SubscribeCommitteeSubnets This endpoint was relatively straightforward to implement and can be summarized in 3 steps: For each slot in the input:If the slot is not found in the cached slot -> committees_at_slot map, query the attester duties corresponding to the slot's epoch by querying the /eth/v1/validator/duties/attester/{epoch} beacon node API; otherwise, retrieve the duties from the cache and go to step 2 For each duty:Retrieve its slot and committees_at_slot values and add them to the slot -> committees_at_slot map Return an object containing the following values: CommitteeIndex, CommitteesAtSlot, Slot, IsAggregator and ValidatorIndex. Except for CommitteesAtSlot, all other values are given as the input.

Dec 24, 2022
Read more from Patrice Vignola
Published on HackMD