Try   HackMD

EPF Cohort 3 Dev Update 7 (pavignol)

Here are the Prysm validator features that I’ve been working on last week:

Wrote the REST implementation for Validator's GetDuties

While the beacon REST API separates the duties retrieval in 3 distinct endpoints (/eth/v1/validator/duties/attester/{epoch}, /eth/v1/validator/duties/proposer/{epoch} and /eth/v1/validator/duties/sync/{epoch}), the Prysm gRPC API does everything in a single function called GetDuties. Moreover, GetDuties automatically retrieves the duties for the current epoch and the next epoch while the beacon REST API endpoints only target a single epoch at a time. For these reasons, this endpoint required combining a lot of endpoints together and massaging the data to make it fit into a single giant data structure containing all duties for epoch and epoch+1.

To make sure that the mapping respected all requirements of the gRPC API, a few rules and details have to be kept in mind:

  1. Even validators that don't have duties for a given epoch should be returned if they were part of the request. This includes all validators in the pending state.
  2. Attester and sync duties should be retrieved for both epoch and epoch+1
  3. Proposer duties should only be retrieved for the current epoch (not epoch+1)
  4. Since sync committee duties are an Altair feature, they should only be retrieved for epoch>=AltairForkEpoch.
  5. For each validator, we have to return the array of validators that are part of the committee for the given epoch.
  6. In addition to the duties, we also have to return the status of each validator .

With those rules in mind, the implementation was relatively straightforward:

  1. Query the status of all validator public keys given in the input via the /eth/v1/beacon/states/head/validators endpoint to get the indices and status of each validator.
  2. Query the list of committees for the given epoch and cache its validators in a {committeeIndex, slot} -> validatorIndices map
  3. Query the list of attester duties for epoch and epoch+1 and cache them in a validatorIndex -> {committeeIndex, slot} map
  4. Query the list of sync duties for epoch and epoch+1 (if epoch>=AltairForkEpoch) and cache the validator indices that have a sync dutie for this epoch in a set
  5. Query the list of proposer duties for epoch and cache them in a validatorIndex -> slot map
  6. Finally, for each index obtained in step 1:
    6.1 If the attester duties map from step 3 has an entry for the current validator index, get its committee from the map obtained in step 2 and save the info in the output struct
    6.2 If the sync duties set from step 4 has an entry for the current validator index, set the IsSyncCommittee value of the output struct to true
    6.3 If the proposer duties map from step 5 has an entry for the current validator index, get its proposer slot from the map and set it in the output struct

Although the implementation is done and I have verified that it passes all tests, I haven't created a PR yet because it's dependent on 2 pending PRs:

Also, implementing this endpoint revealed bugs in the implementation of all 3 duties endpoint, where an epoch would be erroneously rejected because it was in the future. When the Prysm team comes back from the holidays break, I'll work with them to tackle this issue.

Going forward

Next week, I'm planning to implement the StreamBlocksAltair API. This is pretty exciting because it's the last endpoint that hasn't been assigned to someone yet, which means that all endpoints have either been merged into the develop branch, have a PR opened for them or are actively being worked on by an EPF fellow:

Legend:

Symbol State
:x: Not ready for PR
:o: PR opened
:heavy_check_mark: Merged into develop
Done Owner Endpoint
:x: pavignol GetDuties
:heavy_check_mark: pavignol GetBeaconBlock
:heavy_check_mark: pavignol ProposeBeaconBlock
:heavy_check_mark: pavignol PrepareBeaconProposer
:heavy_check_mark: pavignol GetAttestationData
:o: dhruv ProposeAttestation
:x: dhruv SubmitAggregateSelectionProof
:x: dhruv SubmitSignedAggregateSelectionProof
:heavy_check_mark: pavignol ProposeExit
:o: pavignol SubscribeCommitteeSubnets
:o: manu SubmitValidatorRegistrations
:heavy_check_mark: manu ValidatorIndex
:heavy_check_mark: manu ValidatorStatus
:o: manu MultipleValidatorStatus
:o: pavignol SubmitSignedContributionAndProof
:o: manu SubmitSyncMessage
:x: dhruv GetSyncSubcommitteeIndex
:x: dhruv GetSyncMessageBlockRoot
:x: pavignol StreamDuties
:heavy_check_mark: pavignol WaitForChainStart
:heavy_check_mark: manu WaitForActivation
:x: pavignol StreamBlocksAltair
:heavy_check_mark: pavignol DomainData
:x: manu GetFeeRecipientByPubKey
:x: manu CheckDoppelGanger
:x: dhruv GetSyncCommitteeContribution
Last changed by 
 
Patrice Vignola
0
125

Read more

Final dev update (pavignol)

Project summary Before the Beacon REST API was standardized, the Prysmatic Labs team was strongly advocating for the use of a gRPC API instead of a REST API for communication between the validator client and the beacon chain. When it was decided that a REST API should be the standard, the Prysm validator client was already using a bunch of gRPC endpoints to communicate with the beacon chain. This resulted in the Prysm validator and beacon node clients not being standard-compliant and not being able to communicate with VCs and beacon nodes form other teams. Eventually, as third parties wanted to be able to query the beacon chain, Prysmatic Labs started adding more REST endpoints to the beacon node until it became 100% compliant. But since the VC didn't have that chance and was stuck using the gRPC API, it meant that the beacon node also had to keep supporting both the REST and gRPC endpoints. This resulted in a huge maintenance burden. Therefore, the goal of this project was to make the Prysm validator client compliant with the Beacon API standard in order to eventually remove this tech debt and being able to use the VC with beacon nodes made by other teams. For a more details explanation of what this project entails, read the project description. Status report

Feb 27, 2023
EPF Cohort 3 Dev Update 9 (pavignol)

I didn't post an update last week because I hadn't made much progress on the EPF front and was I busy working on a client for the KZG Ceremony. This week, I have 2 updates: Added a REST implementation for the GetDuties endpoint GetDuties is an endpoint that I started working on weeks ago, but because there was a bug with the prysm beacon node REST API, I was unable to complete it. Now that the bug has been fixed, I was finally able to merge this PR. The GetDuties endpoint aggregates 8 Beacon REST APIs into a single one: 1 call to /eth/v1/validator/duties/attester/{epoch} 1 call to /eth/v1/validator/duties/attester/{epoch+1} 1 call to /eth/v1/validator/duties/proposer/{epoch}

Jan 31, 2023
EPF Cohort 3 Dev Update 8 (pavignol)

This week's update will be relatively short since there are not many endpoints to implement on my end, and some of the endpoints that I have implemented are blocked on PRs that haven't been merged yet. Added Capella support to the ProposeBeaconBlock and GetBeaconBlock REST API implementations Originally, when I added ProposeBeaconBlock and GetBeaconBlock, I wasn't able to add support for Capella since the functionality wasn't ready yet on the beacon node side, and therefore there wasn't even a gRPC implementation on the validator side either. But since the Prysm team has been working hard on implementating withdrawals, it was time to revisit those 2 endpoints and add the missing functionality. Adding Capella support was pretty easy since I have already written all of the boilerplate code, heavylifting and tests a few weeks ago for Bellatrix, so all I needed to do was code the conversion of the withdrawals array between the gRPC and the REST API JSON structs, which looks like this: func jsonifyWithdrawals(withdrawals []*enginev1.Withdrawal) []*apimiddleware.WithdrawalJson { jsonWithdrawals := make([]*apimiddleware.WithdrawalJson, len(withdrawals)) for index, withdrawal := range withdrawals {

Jan 9, 2023
EPF Cohort 3 Dev Update 6 (pavignol)

Here are the Prysm validator features that I’ve been working on last week: Added a REST implementation for Validator's SubscribeCommitteeSubnets This endpoint was relatively straightforward to implement and can be summarized in 3 steps: For each slot in the input:If the slot is not found in the cached slot -> committees_at_slot map, query the attester duties corresponding to the slot's epoch by querying the /eth/v1/validator/duties/attester/{epoch} beacon node API; otherwise, retrieve the duties from the cache and go to step 2 For each duty:Retrieve its slot and committees_at_slot values and add them to the slot -> committees_at_slot map Return an object containing the following values: CommitteeIndex, CommitteesAtSlot, Slot, IsAggregator and ValidatorIndex. Except for CommitteesAtSlot, all other values are given as the input.

Dec 24, 2022
Read more from Patrice Vignola
Published on HackMD