EPF Cohort 3 Dev Update 4 (pavignol)

Here are the Prysm validator features that I've been working on last week:

Onboarded the validators's Beacon REST API usage to the E2E tests

The end to end tests are an important part of the Prysm test collateral since they make many systems work together wihout any mocking and simulate how the validator and beacon node should work for about 10 epochs. Therefore, adding a version of those tests where the REST API is tested instead of the gRPC one seemed like a no-brainer. In addition to onboarding the tests to the E2E collateral, this change also merged the REST API build flavor with the gRPC one, which means that we don't need 2 build flavors anymore just to enable or disable the REST API. Everything is controlled through the --enable-beacon-rest-api command-line flag.

Added a REST API implementation for the Validator's DomainData endpoint

The DomainData endpoint is a relatively simple one since it only requires interacting with the /eth/v1/beacon/genesis endpoint, which had already been implemented during the WaitForChainStart work. After retrieving the genesis validator root, everything - including computing the signature domain - is done offline. The change can be found here.

Added a REST API implementation for the Validator's GetAttestationData endpoint

The goal of the GetAttestationData endpoint is to retrieve the following information from the attestations included into a slot:

• The root of the beacon block
• The committee index
• The slot
• The source checkpoint
• The target checkpoint

There was a small mixup while I was working on this PR, which is why it was closed with 0 files changed. Unfortunately, someone else rebased this PR on top of their own PR, which got merged first and therefore absorbed all of my changes. It's not a big deal besides messing up with the history (the other PR basically has 2 new features in it instead of a single one) and associating the commits with the wrong author during a git blame.

Added a REST API implementation for the Validator's ProposeBeaconBlock endpoint

This change was definitely the biggest one so far, although it was still not very complex. The complexity here was more related to the limitations of Go version 1.19 than anything else. At the time of writing, there are currently 5 types of blocks that Prysm supports:

• Phase0
• Altair
• Bellatrix
• Bellatrix (blinded)
• Capella (blinded)

Although all block are slightly different and mostly a superset of the previous version, most of their fields are identical. But because of these small differences, they all have different structs that make it hard to build the blocks in an elegant polymorphic way. For example, here is what the body of a Phase0 block looks like:

type BeaconBlockBodyJson struct {
	RandaoReveal      string                     `json:"randao_reveal" hex:"true"`
	Eth1Data          *Eth1DataJson              `json:"eth1_data"`
	Graffiti          string                     `json:"graffiti" hex:"true"`
	ProposerSlashings []*ProposerSlashingJson    `json:"proposer_slashings"`
	AttesterSlashings []*AttesterSlashingJson    `json:"attester_slashings"`
	Attestations      []*AttestationJson         `json:"attestations"`
	Deposits          []*DepositJson             `json:"deposits"`
	VoluntaryExits    []*SignedVoluntaryExitJson `json:"voluntary_exits"`
}

and here is what the body of an Altair block looks like:

type BeaconBlockBodyAltairJson struct {
	RandaoReveal      string                     `json:"randao_reveal" hex:"true"`
	Eth1Data          *Eth1DataJson              `json:"eth1_data"`
	Graffiti          string                     `json:"graffiti" hex:"true"`
	ProposerSlashings []*ProposerSlashingJson    `json:"proposer_slashings"`
	AttesterSlashings []*AttesterSlashingJson    `json:"attester_slashings"`
	Attestations      []*AttestationJson         `json:"attestations"`
	Deposits          []*DepositJson             `json:"deposits"`
	VoluntaryExits    []*SignedVoluntaryExitJson `json:"voluntary_exits"`
	SyncAggregate     *SyncAggregateJson         `json:"sync_aggregate"`
}

So although they are almost identical, if we were to refactor Prysm, BeaconBlockBodyJson could easily become a part of BeaconBlockBodyAltairJson, which would look like this:

type BeaconBlockBodyAltairJson struct {
	BeaconBlockBodyJson
	SyncAggregate     *SyncAggregateJson         `json:"sync_aggregate"`
}

But since this is not the case, we need to duplicate the logic of assigning each field to the corresponding gRPC value for each type of block, even though most of the fields are identical.

Another problem is that, even though the gRPC structs and the beacon API JSON structs are literally identical, there are no elegant ways to translate one into the other. For example, compare the structs between the beacon API version and the gRPC version of a Phase0 block:

type BeaconBlockBodyJson struct {
	RandaoReveal      string                     `json:"randao_reveal" hex:"true"`
	Eth1Data          *Eth1DataJson              `json:"eth1_data"`
	Graffiti          string                     `json:"graffiti" hex:"true"`
	ProposerSlashings []*ProposerSlashingJson    `json:"proposer_slashings"`
	AttesterSlashings []*AttesterSlashingJson    `json:"attester_slashings"`
	Attestations      []*AttestationJson         `json:"attestations"`
	Deposits          []*DepositJson             `json:"deposits"`
	VoluntaryExits    []*SignedVoluntaryExitJson `json:"voluntary_exits"`
}

type BeaconBlockBody struct {
	state         protoimpl.MessageState
	sizeCache     protoimpl.SizeCache
	unknownFields protoimpl.UnknownFields

	RandaoReveal      []byte                 `protobuf:"bytes,1,opt,name=randao_reveal,json=randaoReveal,proto3" json:"randao_reveal,omitempty" ssz-size:"96"`
	Eth1Data          *Eth1Data              `protobuf:"bytes,2,opt,name=eth1_data,json=eth1Data,proto3" json:"eth1_data,omitempty"`
	Graffiti          []byte                 `protobuf:"bytes,3,opt,name=graffiti,proto3" json:"graffiti,omitempty" ssz-size:"32"`
	ProposerSlashings []*ProposerSlashing    `protobuf:"bytes,4,rep,name=proposer_slashings,json=proposerSlashings,proto3" json:"proposer_slashings,omitempty" ssz-max:"16"`
	AttesterSlashings []*AttesterSlashing    `protobuf:"bytes,5,rep,name=attester_slashings,json=attesterSlashings,proto3" json:"attester_slashings,omitempty" ssz-max:"2"`
	Attestations      []*Attestation         `protobuf:"bytes,6,rep,name=attestations,proto3" json:"attestations,omitempty" ssz-max:"128"`
	Deposits          []*Deposit             `protobuf:"bytes,7,rep,name=deposits,proto3" json:"deposits,omitempty" ssz-max:"16"`
	VoluntaryExits    []*SignedVoluntaryExit `protobuf:"bytes,8,rep,name=voluntary_exits,json=voluntaryExits,proto3" json:"voluntary_exits,omitempty" ssz-max:"16"`
}

If we ignore the protobuf metadata at the top of the gRPC struct, they are identical and only their types differ. In theory, the conversion should also be fairly mechanical:

1. Iterate through all fields
2. If the field is a struct or an array, recursively iterate through all fields of the struct/array
3. If the field has the hex tag, convert it using the hexutil.Decode function
4. If the field is an integer, convert it to a string

But Golang doesn't yet have an elegant way to do this conversion a single time at compile time when dealing with very similar types. It has generics, but they are not powerful enough yet to be able to call common fields. It lacks a compile-type alternative to templates in C++. Even though reflection seems like a fairly elegant solution, I decided against it for a few reasons:

• Performance. Although reflection performance may not be the bottleneck for many applications, it seems like a waste of cycles when used just to convert types between one another.
• More error-prone. Reflection is relatively error-prone since it's dealing with an abstract representation of types and there are no compile-time checks.
• Compile-time errors become runtime errors. If the underlying structure of a struct changes (e.g. field name or field type), the program will compile just fine and the error will only be caught at runtime, which can make development and testing more difficult. Also, unless there is very good test coverage, some of the runtime errors could make their way into client releases.
• More complex to understand at a glance.

Overall, reflection is great when adding middlewares that connect existing APIs together, but when adding new features it feels cleaner and safer to go with the compile-time alternative of just assigning each struct field manually, although it may look less elegant at first.