EventSource Support for gRPC Gateway

• EventSource Support for gRPC Gateway
  • Background
  • EventSource Browser Standard
  • Current Streaming Support in gRPC Gateway
  • Required Changes to gRPC Gateway
    • Solution
  • Required Changes in Prysm
    • Supported Events
  • Work Estimates
  • References

TL;DR: The eth2.0-apis standard requires all eth2 implementations to support an /eth/v1/events endpoint which uses the browser EventSource API for subscribing to server-side events. The grpc-ecosystem/grpc-gateway project does not convert stream endpoints into this standard, and will require modification for Prysm to fully comply.

Background

EventSource is a native browser API allowing for a standard way of reading server-sent events, supported by most major, modern browsers. The Mozilla developer docs describe EventSource as:

A web content's interface to server-sent events. An EventSource instance opens a persistent connection to an HTTP server, which sends events in text/event-stream format. The connection remains open until closed by calling EventSource.close().

Once the connection is opened, incoming messages from the server are delivered to your code in the form of events. If there is an event field in the incoming message, the triggered event is the same as the event field value. If no event field is present, then a generic message event is fired.

Unlike WebSockets, server-sent events are unidirectional; that is, data messages are delivered in one direction, from the server to the client (such as a user's web browser). That makes them an excellent choice when there's no need to send data from the client to the server in message form. For example, EventSource is a useful approach for handling things like social media status updates, news feeds, or delivering data into a client-side storage mechanism like IndexedDB or web storage.

The eth2.0-apis standard requires implementations to support an /eth/v1/events endpoint that meets the requirements of an EventSource backend. This endpoint allows for a very straightforward way of subscribing to relevant events from eth2 nodes, without needing to send data to the server. The endpoint needs to support the following event types

• head
• block
• attestation
• voluntary_exit
• finalized_checkpoint
• chain_reorg

From the client side, subscribing to events from the backend is as easy as:

const source = new EventSource('http://localhost:3500/eth/v1/events');
source.addEventListener('head', headHandler, false);

EventSource Browser Standard

The formal specification for the EventSource standard is described here. The official specification states the behavior as follows:

On the server-side, the server sends messages in the following form, with the text/event-stream MIME type:

data: This is the first message.

data: This is the second message, it
data: has two lines.

data: This is the third message.

Authors can separate events by using different event types. Here is a stream that has two event types, "add" and "remove":

event: add
data: 73857293

event: remove
data: 2153

event: add
data: 113411

That is, the content-type of data sent needs to be text/event-stream. and we can either include an event: specifier or not, but data: should always be part of the response. The specification is fairly straightforward, and we should allow for both closing and error handling of data coming from the server.

Current Streaming Support in gRPC Gateway

Currently, gRPC gateway converts streams into new-line delimited JSON streams, known as NDJSON. This format is essentially a JSON response payload that we normally get from gRPC, with a new-line delimiter \n at the end of each payload, received via TCP streams. NDJSON does not use websockets, and can be used anywhere as long as the specification is met. An example of NDJSON payload is as follows:

{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/milk-and-wood-san-jose?osq=Desserts","text":"Milk & Wood"}]},{"Business":[{"href":"https://www.yelp.com/biz/dzuis-cakes-and-desserts-san-jose?osq=Desserts","text":"Dzui’s Cakes & Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/recess-italian-ice-and-desserts-san-jose?osq=Desserts","text":"Recess Italian Ice and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/icicles-san-jose-7?osq=Desserts","text":"ICICLES"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-rendezvous-san-jose?osq=Desserts","text":"Sweet Rendezvous"}]},{"Business":[{"href":"https://www.yelp.com/biz/passion-t-snacks-and-desserts-san-jose?osq=Desserts","text":"Passion-T Snacks and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/vampire-penguin-featuring-jastea-san-jose?osq=Desserts","text":"Vampire Penguin featuring Jastea"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-gelato-tea-lounge-san-jose?osq=Desserts","text":"Sweet Gelato Tea Lounge"}]},{"Business":[{"href":"https://www.yelp.com/biz/matcha-love-san-jose-6?osq=Desserts","text":"Matcha Love"}]},{"Business":[{"href":"https://www.yelp.com/biz/anton-sv-p%C3%A2tisserie-san-jose-2?osq=Desserts","text":"Anton SV Pâtisserie"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286383006},"timestamp":1513286383006,"sequenceNumber":0}}
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=10","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=10","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/oooh-san-jose-4?osq=Desserts","text":"Oooh"}]},{"Business":[{"href":"https://www.yelp.com/biz/hannah-san-jose?osq=Desserts","text":"Hannah"}]},{"Business":[{"href":"https://www.yelp.com/biz/chocatoo-san-jose?osq=Desserts","text":"Chocatoo"}]},{"Business":[{"href":"https://www.yelp.com/biz/nox-cookie-bar-san-jose?osq=Desserts","text":"Nox Cookie Bar"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-fix-creamery-san-jose?osq=Desserts","text":"Sweet Fix Creamery"}]},{"Business":[{"href":"https://www.yelp.com/biz/my-milkshake-san-jose?osq=Desserts","text":"My Milkshake"}]},{"Business":[{"href":"https://www.yelp.com/biz/matcha-love-san-jose-6?osq=Desserts","text":"Matcha Love"}]},{"Business":[{"href":"https://www.yelp.com/biz/banana-cr%C3%AApe-san-jose-2?osq=Desserts","text":"Banana Crêpe"}]},{"Business":[{"href":"https://www.yelp.com/biz/marco-polo-italian-ice-cream-san-jose-4?osq=Desserts","text":"Marco Polo Italian Ice Cream"}]},{"Business":[{"href":"https://www.yelp.com/biz/blackball-desserts-san-jose-san-jose?osq=Desserts","text":"BlackBall Desserts San Jose"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286384917},"timestamp":1513286384917,"sequenceNumber":1}}
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=20","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=20","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/anton-sv-p%C3%A2tisserie-san-jose-2?osq=Desserts","text":"Anton SV Pâtisserie"}]},{"Business":[{"href":"https://www.yelp.com/biz/soyful-desserts-san-jose-8?osq=Desserts","text":"Soyful Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/cocola-bakery-san-jose?osq=Desserts","text":"Cocola Bakery"}]},{"Business":[{"href":"https://www.yelp.com/biz/charlies-cheesecake-works-san-jose?osq=Desserts","text":"Charlie’s Cheesecake Works"}]},{"Business":[{"href":"https://www.yelp.com/biz/jt-express-san-jose-2?osq=Desserts","text":"JT Express"}]},{"Business":[{"href":"https://www.yelp.com/biz/nox-cookie-bar-san-jose?osq=Desserts","text":"Nox Cookie Bar"}]},{"Business":[{"href":"https://www.yelp.com/biz/shuei-do-manju-shop-san-jose?osq=Desserts","text":"Shuei-Do Manju Shop"}]},{"Business":[{"href":"https://www.yelp.com/biz/churros-el-guero-san-jose?osq=Desserts","text":"Churros El Guero"}]},{"Business":[{"href":"https://www.yelp.com/biz/sno-crave-tea-house-san-jose-4?osq=Desserts","text":"Sno-Crave Tea House"}]},{"Business":[{"href":"https://www.yelp.com/biz/j-sweets-san-jose-3?osq=Desserts","text":"J.Sweets"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286395948},"timestamp":1513286395948,"sequenceNumber":2}}

There is an open issue in the grpc-gateway repository asking to replace the default behavior with the text/event-stream standard to satisfy EventSource, however, no one has implemented this yet. That is, the way grpc-gateway handles streams does not fit our needs to align with the eth2.0-apis standard. If we want the grpc-gateway authors to accept our changes, however, we should ideally implement this feature as a pluggable marshaler to the gateway instead of replacing NDJSON support entirely.

Required Changes to gRPC Gateway

The main invariant we need to preserve when implementing changes in gRPC gateway is we cannot force developers to think about the gateway when writing their applications. Developers should be able to define a regular gRPC service stream and not care about how the gateway will interpret it, i.e., it should just work. For example, we can define a basic stream endpoint as follows:

service BeaconChain {
  rpc StreamChainHead(google.protobuf.Empty) returns (stream ChainHead) {
    option (google.api.http) = {
        get: "/eth/v1beta/chainhead/stream"
    };
  }
}

message ChainHead {
    uint64 slot = 1;
    bytes beacon_block_root = 2;
}

and when using a client to call this endpoint using the EventSource browser API, we should see something like:

data: "{\"slot\": \"10\", \"beacon_block_root\": \"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf\"}"\n

The /eth/v1/events endpoint from eth2.0-apis requires us to specify the event kind we wish to subscribe to via query-parameters. If not specified, we should return an error. The EventSource specification states that responses can include an event field which denotes the kind of event we are receiving. For example, attempting to subscribe to eth/v1/events?topics=head should return a response which looks as follows:

event: head\n
data: "{\"slot\": \"10\", \"block\": \"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf\", \"state\":\"0x600e852a08c1200654ddf11025f1ceacb3c2e74bdd5c630cde0838b2591b69f9\", \"epoch_transition\": false, \"previous_duty_dependent_root\": \"0x5e0043f107cb57913498fbf2f99ff55e730bf1e151f02f221e977c91a90a0e91\", \"current_duty_dependent_root\": \"0x5e0043f107cb57913498fbf2f99ff55e730bf1e151f02f221e977c91a90a0e91\"}"\n
\n

Solution

In order to support event streams, gRPC implementations of streaming endpoints can use a common protobuf message defined in the grpc-gateway package as:

message EventSource {
  string event = 1;
  google.protobuf.Any data = 2;
}

We maintain a fork here which checks if a stream endpoint is returning that special proto message, and then performs custom marshaling to send the proper response data over HTTP of type text/event-stream.

An example ping-pong endpoint implementation would look as follows in Go:

func (s *server) StreamEvents(
    _ *pb.EventRequest, stream pb.Events_StreamEventsServer,
) error {
    ticker := time.NewTicker(time.Millisecond * 500)
    defer ticker.Stop()
    for {
        select {
        case <-ticker.C:
            data, err := anypb.New(req)
            if err != nil {
                return err
            }
            if err := stream.Send(&gwpb.EventSource{
                Event: "pong",
                Data:  data,
            }); err != nil {
                return err
            }
        case <-stream.Context().Done():
            return errors.New("context canceled")
        }
    }
}

Required Changes in Prysm

We predict most changes required to support this feature lie in grpc-gateway rather than Prysm. Prysm currently uses a broadcast pattern for events at runtime which sends all events via feeds available as globals of a running beacon node. Any runtime service that needs these feeds can subscribe to them as needed.

A potential schema for the /eth/v1/events endpoint using grpc and protobuf could work as follows:

service Events {
  rpc Events(EventTopicsRequest) returns (stream gateway.EventSource) {
    option (google.api.http) = {
        get: "/eth/v1/events"
    };
  }
}

message EventTopicsRequest {
    repeated string topics = 1;
}

message ChainHead {
    uint64 slot = 1;
    bytes beacon_block_root = 2;
}

message Attestation {
    bytes aggregation_bits = 1;
}
...

A potential server implementation of the /eth/v1/events endpoint in Prysm could work as follows:

1. Parse the requested topics in the request query parameters, such as head,attestation,voluntary_exit
2. Conditionally subscribe to the corresponding event feeds based on the topic parameters
3. The response payload is a protobuf message including the event name and the actual data payload using the oneof pattern. The gateway then transcodes this data as needed into the text/event-stream MIME type
4. Gracefully handle disconnects and errors

Supported Events

Prysm currently uses a broadcast pattern for events at runtime which sends all events via feeds available as globals of a running beacon node. Any runtime service that needs these feeds can subscribe to them as needed. These event feeds are available in the beacon-chain/node package.

The following are the required event types and whether or not Prysm already has an event feed for them which works today:

Event Name	Supported?
head
block
attestation
voluntary_exit
finalized_checkpoint
chain_reorg

Work Estimates

1. Fork and clone grpc-gateway
2. Create a sample Go application with a tiny protobuf service definition, a minimal stream gRPC server, and a gateway exposed on localhost using our fork
3. Create a simple client which requests the stream endpoint from the gateway in step (2)
4. Fully understand the codepath which runs in the grpc-gateway when transcoding a stream response into NDJSON
5. Identify where marshaling logic for the EventSource standard can included instead of NDJSON, or in-addition to it
6. Identify the requirements of protobuf response message needed to support grpc-gateway transcoding into the text/event-stream format
7. Include the changes in Prysm's beacon-chain/rpc package which allow for basic subscription of a single event using text/event-stream, such as head
8. Support all other required event types in Prysm

References

• https://ethereum.github.io/eth2.0-APIs/#/Events/eventstream
• https://github.com/grpc-ecosystem/grpc-gateway/issues/26
• https://developer.mozilla.org/en-US/docs/Web/API/EventSource
• https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events
• http://ndjson.org/