# Protobuf v2 transition notes

This document outlines the total context for the change of gogoproto to native v2 protobufs.

## Why change?
The change began as the standard eth2.0 APIs required optional fields, which gogoproto did not natively support. We saw this as a good opportunity to remove gogoproto from gRPC in order to continue accepting upstream v2 updates (which included optional fields), and to avoid the technical debt that relying on gogoproto solved.

## What changes in Prysm?
For Prysm as a client, nothing significant should change. It is as if we swapped out the underlying functionality from gogoproto to native v2 protobufs, which means there should hopefully be no differences for most parties involved. They should be entirely interoprable between eachother and the code difference is mainly using a different import of proto and the nuances involved in the different versions.

The different versions to be using now are:

 github.com/grpc-ecosystem/grpc-gateway/ ->  github.com/grpc-ecosystem/grpc-gateway/v2/

github.com/gogo/protobuf/types -> github.com/golang/protobuf/ptypes/empty

github.com/gogo/protobuf/proto -> google.golang.org/protobuf/proto

For example: we no longer use `ptypes.Empty`, but `emptypb.Empty` for empty definitions on APIs. And proto messages no longer have an embedded `Marshal()`. It is replaced with `proto.Marshal()`.

## What changes in ethereumapis?
Because we are no longer using gogoproto tags, and instead using native protobuf extensions, the syntax for adding extensions changes.
Before:

message BeaconBlock {
    uint64 slot = 1 [(gogoproto.casttype) = "github.com/prysmaticlabs/eth2-types.Slot"];
    uint64 proposer_index = 2 [(gogoproto.casttype) = "github.com/prysmaticlabs/eth2-types.ValidatorIndex"];
    bytes parent_root = 3 [(gogoproto.moretags) = "ssz-size:\"32\""];
    bytes state_root = 4 [(gogoproto.moretags) = "ssz-size:\"32\""];
}

After:

message BeaconBlock {
    uint64 slot = 1 [(ethereum.eth.ext.cast_type) = "github.com/prysmaticlabs/eth2-types.Slot"];
    uint64 proposer_index = 2 [(ethereum.eth.ext.cast_type) = "github.com/prysmaticlabs/eth2-types.ValidatorIndex"];
    bytes parent_root = 3 [(ethereum.eth.ext.ssz_size) = "32"];
    bytes state_root = 4 [(ethereum.eth.ext.ssz_size) = "32"];
}

You can view all possible options to pass in [here](https://github.com/prysmaticlabs/ethereumapis/blob/b69e962c600b020348daadd8ac18d786c1c368c2/eth/ext/options.proto).

The input to them stays the same as before, these simply append the tags to the protobuf structs for fastssz to read them.

## But how are we still supporting cast types?
Good question, now we have the [`protoc-gen-go-cast`](https://github.com/prysmaticlabs/protoc-gen-go-cast) protoc extension, which wraps around `protoc-gen-go` and casts any requested types into the compiled output. go-cast mainly relies on reading the protogen input (extensions) for information on what to cast, and then using AST utils on the compiled output to cast them all into the requested types manually.

On top of this, we are using our own [`grpc-gateway`](https://github.com/prysmaticlabs/grpc-gateway) plugin for generating gRPC gateway files which are needed for JSON endpoints. This plugin has had minor go-template modifications to allow passing on any requested type casts from the protos into the generated grpc-gateway files.

## How does maintenance in the future change?
For the future, the following maintenance will be required:
* Supporting any other extensions more complex than struct tags will require manual work protoc-gen-go-cast, unless another protoc plugin supports them.
* Eth2-Types added in the future will require modifications to the grpc-gateway plugin, I recommend we create a function in eth2-types that grpc-gateway uses to prevent having to modify the go templating in the future.
* If there are any desirable proto-gen-go or grpc-gateway updates, we will have to merge them from upstream, but cases like this are rare. gogoproto is usually decently slow to accept any new updates.

## Was this change worth it?
I think so! Now we're on the edge of protobufs newest APIs, no longer relying on gogoproto for our full network stack essentially. It might require some more maintenance but in the future I think avoiding gogoproto will pay off, as we are now using entirely native protobuf!