Final dev update (pavignol)

Project summary

Before the Beacon REST API was standardized, the Prysmatic Labs team was strongly advocating for the use of a gRPC API instead of a REST API for communication between the validator client and the beacon chain. When it was decided that a REST API should be the standard, the Prysm validator client was already using a bunch of gRPC endpoints to communicate with the beacon chain. This resulted in the Prysm validator and beacon node clients not being standard-compliant and not being able to communicate with VCs and beacon nodes form other teams.

Eventually, as third parties wanted to be able to query the beacon chain, Prysmatic Labs started adding more REST endpoints to the beacon node until it became 100% compliant. But since the VC didn't have that chance and was stuck using the gRPC API, it meant that the beacon node also had to keep supporting both the REST and gRPC endpoints. This resulted in a huge maintenance burden.

Therefore, the goal of this project was to make the Prysm validator client compliant with the Beacon API standard in order to eventually remove this tech debt and being able to use the VC with beacon nodes made by other teams.

For a more details explanation of what this project entails, read the project description.

Status report

I'm happy to say that 100% of the Prysm validator gRPC endpoints have now been converted to REST endpoints, as shown by the following table:

Legend:

Symbol	State
	Not ready for PR
	PR opened
	Merged into develop

Done	Owner	gRPC	Beacon API	Description
	pavignol	GetDuties	/eth/v1/validator/duties/attester/{epoch} /eth/v1/validator/duties/proposer/{epoch}	Returns a list of duties which are the actions should be performed by validators for a given epoch.
	pavignol	GetBeaconBlock	/eth/v2/validator/blocks/{slot}	Retrieves the latest valid beacon block to be proposed on the beacon chain.
	pavignol	ProposeBeaconBlock	/eth/v1/beacon/blocks /eth/v1/beacon/blinded_blocks	Sends the newly signed beacon block to beacon node.
	pavignol	PrepareBeaconProposer	/eth/v1/validator/prepare_beacon_proposer	Submits fee recipient information to be used when preparing block.
	pavignol	GetAttestationData	/eth/v1/beacon/blocks/head/attestations (GET)	Retrieves the latest valid attestation data to be attested on the beacon chain.
	dhruv	ProposeAttestation	/eth/v1/beacon/blocks/head/attestations (POST)	Sends the newly signed attestation to beacon node.
	dhruv	SubmitAggregateSelectionProof	/eth/v1/validator/aggregate_attestation /eth/v1/validator/aggregate_and_proofs	Submit selection proof to the beacon node to aggregate all matching wire attestations with the same data root.
	dhruv	SubmitSignedAggregateSelectionProof
	pavignol	ProposeExit	/eth/v1/beacon/pool/voluntary_exits (POST)	Propose to leave the list of active validators.
	pavignol	SubscribeCommitteeSubnets	/eth/v1/validator/beacon_committee_subscriptions	Subscribe to particular committee ID subnets given validator's duty.
	manu	SubmitValidatorRegistrations	/eth/v1/validator/register_validator	Provide beacon node with registration for the given validators.
	manu	ValidatorIndex	/eth/v1/beacon/states/head/validators/{0x_prefixed_public_key}	Retrieves a validator's index location in the beacon state's validator registry.
	manu	ValidatorStatus	/eth/v1/beacon/states/head/validators/{0x_prefixed_public_key}	Returns a validator's status based on the current epoch.
	manu	MultipleValidatorStatus	/eth/v1/beacon/states/head/validators	Returns a list of validator statuses on the current epoch. After getting the list, filter by public keys.
	pavignol	SubmitSignedContributionAndProof	/eth/v1/validator/contribution_and_proofs	Submit a signed sync committee contribution and proof object.
	manu	SubmitSyncMessage	/eth/v1/beacon/pool/sync_committees	Submits a sync committee message to be broadcasted over network.
	dhruv	GetSyncSubcommitteeIndex	None	This can be computed locally compute_subnets_for_sync_committee
	dhruv	GetSyncMessageBlockRoot	/eth/v1/beacon/blocks/{block_id}/root
	pavignol	StreamDuties
	pavignol	WaitForChainStart	/eth/v1/beacon/genesis	/eth/v1/beacon/genesis returns the same set of information that eth/v1alpha1/validator/chainstart/stream is needed for.
	manu	WaitForActivation
	pavignol	StreamBlocksAltair
	pavignol	DomainData	None	Can be calculated refer with fork schedule and genesis information.
	manu	GetFeeRecipientByPubKey	None
	manu	CheckDoppelGanger
	dhruv	GetSyncCommitteeContribution	/eth/v1/validator/sync_committee_contribution?slot=%d&subcommittee_index=%d&beacon_block_root=%s

Future of the project

Although the main goal of the project has been achieved, there is still more work to do before it's production ready. Below are a few of the ideas I have to achieve this:

Onboard the current REST API validator endtoend tests to pull request test passes

The endtoend test pass for the REST API validator client currently only runs after pull requests have been merged. This decision was made in order to now slow down the Prysmatic Labs team in case we break something that doesn't affect any production code, but now that all endpoints are supported, we should add the tests to the pre-merge pass.

Onboard the REST API validator client to more endtoend tests

The REST API validator client are currently onboarded to a minimal endtoend test suite, but the gRPC path is exercised in more detailed endtoend tests. We should have the same coverage for the REST API as we do for the gRPC API.

Enable the Prysm REST API in Hive

Unlike other consensus clients, Hive currently uses the gRPC API when testing Prysm since the validator client wasn't able to send REST API requests. In order to increase our test coverage, we should now be able to remove this workaround in Hive and allow the Prysm client to be tested the same way as the other ones.

Add test passes that mix and match validator and beacon chain clients of different teams

In order to make sure that the Prysm VC is 100% compliant with the Beacon API spec, we should make sure that it works well when it queries beacon nodes from other teams.

Remove the gRPC tech debt from Prysm in favor of the REST API

One of the main reason for starting this project was to remove the maintenance burden and tech debt caused by having to keep a gRPC API that doesn't follow a standard. Therefore, this project wouldn't be totally complete if we didn't remove the gRPC legacy code.

Self-evaluation

Overally, I'm proud of what I achieved during the last 4 months. At the beginning, it wasn't obvious which steps would be needed to accomplish this project during the timeframe of the EPF while, at the same time, not disrupting the work of the EPF team or regressing existing features.

So I took it upon myself to design an abstraction that would make code that was previously calling into the gRPC validator client stay exactly the same, but instead the REST API implementation would be used. As long as the inputs and the outputs of each function of the REST API was the same as their corresponding function in the gRPC API, we were basically guaranteed that everything was working well. And, of course, I guarded everything behind a feature flag that was is disabled by default to make sure that we don't affect the work of the Prysmatic Labs team and don't cause any regressions.

I also created a design document that outlined the idea of this design and contained table that would make it easy to split endpoints between each member of the project. After the initial refactoring to add the abstraction layer, almost all the work was parallelizable between the 3 of us.