EPF Cohort 3 Dev Update 6 (pavignol)

Here are the Prysm validator features that I’ve been working on last week:

Added a REST implementation for Validator's SubscribeCommitteeSubnets

This endpoint was relatively straightforward to implement and can be summarized in 3 steps:

1. For each slot in the input:
   • If the slot is not found in the cached slot -> committees_at_slot map, query the attester duties corresponding to the slot's epoch by querying the /eth/v1/validator/duties/attester/{epoch} beacon node API; otherwise, retrieve the duties from the cache and go to step 2
   • For each duty:
     • Retrieve its slot and committees_at_slot values and add them to the slot -> committees_at_slot map
2. Return an object containing the following values: CommitteeIndex, CommitteesAtSlot, Slot, IsAggregator and ValidatorIndex. Except for CommitteesAtSlot, all other values are given as the input.

We cache the committees_at_slot values in a map to reduce the number of requests sent to the beacon node since there are many slots per epoch. Since the SubscribeCommitteeSubnets function is only called for slots within 2 consecutive epochs, we have to send a maximum of 2 requests per function call.

While I was working on this feature, I figured out that a recent change broke the /eth/v1/validator/duties/attester/{epoch} endpoint for the Prysm beacon node because I was getting an error every time I requested the duties of the following epoch, which the spec allows. By digging a little bit, I found out that this is because the function StateBySlot used in the beacon node's REST API side is returning an error if the requested slot is higher than the current or head slot:

if target > p.GenesisTimeFetcher.CurrentSlot() {
    return nil, errors.New("requested slot is in the future")
}
if target > p.ChainInfoFetcher.HeadSlot() {
    return nil, errors.New("requested slot number is higher than head slot number")
}

For this reason I was haven't been able to merge this PR yet, but the team is aware of this issue and is working on it (although it will probably have to wait until everyone is back from vacation).

Added REST implementation for Validator's SubmitSignedContributionAndProof

There's not much to say about this endpoint since it was a very simple and straightforward conversion between the gRPC and REST API structs:

jsonContributionAndProofs := []apimiddleware.SignedContributionAndProofJson{
    {
        Message: &apimiddleware.ContributionAndProofJson{
            AggregatorIndex: strconv.FormatUint(uint64(in.Message.AggregatorIndex), 10),
            Contribution: &apimiddleware.SyncCommitteeContributionJson{
                Slot:              strconv.FormatUint(uint64(in.Message.Contribution.Slot), 10),
                BeaconBlockRoot:   hexutil.Encode(in.Message.Contribution.BlockRoot),
                SubcommitteeIndex: strconv.FormatUint(in.Message.Contribution.SubcommitteeIndex, 10),
                AggregationBits:   hexutil.Encode(in.Message.Contribution.AggregationBits),
                Signature:         hexutil.Encode(in.Message.Contribution.Signature),
            },
            SelectionProof: hexutil.Encode(in.Message.SelectionProof),
        },
        Signature: hexutil.Encode(in.Signature),
    },
}

You can find this PR over here.

Investigated a failure in the E2E tests

This one is a bit odd since it's not a feature request or a tangible contribution, but I had to use my debugging skills to figure it out.

One day before writing this update, Nishan from the Prysm team warned me that one of the E2E tests related to the REST API validator had been failing for a while in an easily reproducible, non-sporadic manner. I was surprised to hear that because we run the REST API part of the E2E tests before every PR to make sure that nothing is broken, and everything had been passing until that moment. But when I looked at the logs shared by Nishant, I realized that this is only reproducible when the entire test suite is run without filters. This was surprising to both he and I since our understanding was that none of the test cases share a state and shouldn't affect the results of other test cases.

Anyway, I started investigating using git bisect to figure out which commit caused the failures, and to my surprise I found out that the guilty commit had nothing to do with the work that we had been doing with the REST API as part of the EPF. The only reason it was failing only during the REST API tests was because the gRPC tests are ran in a separate test suite, and therefore don't have the same test side effects applied to them.

Even though the failure had nothing to do with the changes that we had been making as part of the EPF, it was nice to do some debugging with a larger scope than just the EPF and learn more about the system. Because of my findings, the team will be able to investigate this issue once they come back from vacation.