# EventSource Support for gRPC Gateway

[toc]

:::info
:bulb: **TL;DR**: The [eth2.0-apis](https://github.com/ethereum/eth2.0-APIs) standard requires all eth2 implementations to support an [/eth/v1/events](https://ethereum.github.io/eth2.0-APIs/#/Events/eventstream) endpoint which uses the browser [`EventSource`](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) API for subscribing to server-side events. The [grpc-ecosystem/grpc-gateway project](https://github.com/grpc-ecosystem/grpc-gateway) :skull: **does not** :skull: convert stream endpoints into this standard, and will require modification for Prysm to fully comply.
:::

## Background

`EventSource` is a native browser API allowing for a standard way of reading server-sent events, supported by [most major, modern browsers](https://developer.mozilla.org/en-US/docs/Web/API/EventSource#browser_compatibility). The Mozilla [developer docs](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) describe `EventSource` as:

> A web content's interface to server-sent events. An EventSource instance opens a persistent connection to an HTTP server, which sends events in text/event-stream format. The connection remains open until closed by calling EventSource.close().

> Once the connection is opened, incoming messages from the server are delivered to your code in the form of events. If there is an event field in the incoming message, the triggered event is the same as the event field value. If no event field is present, then a generic message event is fired.

> Unlike WebSockets, server-sent events are unidirectional; that is, data messages are delivered in one direction, from the server to the client (such as a user's web browser). That makes them an excellent choice when there's no need to send data from the client to the server in message form. For example, EventSource is a useful approach for handling things like social media status updates, news feeds, or delivering data into a client-side storage mechanism like IndexedDB or web storage.

The [eth2.0-apis](https://github.com/ethereum/eth2.0-APIs) standard requires implementations to support an [/eth/v1/events](https://ethereum.github.io/eth2.0-APIs/#/Events/eventstream) endpoint that meets the requirements of an `EventSource` backend. This endpoint allows for a very straightforward way of subscribing to relevant events from eth2 nodes, without needing to send data to the server. The endpoint needs to support the following event types

- head
- block
- attestation
- voluntary_exit
- finalized_checkpoint
- chain_reorg

From the client side, subscribing to events from the backend is as easy as:

const source = new EventSource('http://localhost:3500/eth/v1/events');
source.addEventListener('head', headHandler, false);

## EventSource Browser Standard

The formal specification for the `EventSource` standard is described [here](https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events). The official specification states the behavior as follows:

> On the server-side, the server sends messages in the following form, with the `text/event-stream` MIME type:
>
> `data: This is the first message.`
>
> `data: This is the second message, it`
> `data: has two lines.`

> `data: This is the third message.`
>
> Authors can separate events by using different event types. Here is a stream that has two event types, "add" and "remove":

> `event: add`
> `data: 73857293`

> `event: remove`
> `data: 2153`

> `event: add`
> `data: 113411`

That is, the content-type of data sent needs to be `text/event-stream`. and we can either include an `event:` specifier or not, but `data:` should **always** be part of the response. The specification is fairly straightforward, and we should allow for both closing and error handling of data coming from the server.

## Current Streaming Support in gRPC Gateway

Currently, gRPC gateway converts streams into [new-line delimited JSON](http://ndjson.org/) streams, known as NDJSON. This format is essentially a JSON response payload that we normally get from gRPC, with a new-line delimiter `\n` at the end of each payload, received via TCP streams. NDJSON does not use websockets, and can be used anywhere as long as the [specification](https://github.com/ndjson/ndjson-spec) is met. An example of NDJSON payload is as follows:

{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/milk-and-wood-san-jose?osq=Desserts","text":"Milk & Wood"}]},{"Business":[{"href":"https://www.yelp.com/biz/dzuis-cakes-and-desserts-san-jose?osq=Desserts","text":"Dzui’s Cakes & Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/recess-italian-ice-and-desserts-san-jose?osq=Desserts","text":"Recess Italian Ice and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/icicles-san-jose-7?osq=Desserts","text":"ICICLES"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-rendezvous-san-jose?osq=Desserts","text":"Sweet Rendezvous"}]},{"Business":[{"href":"https://www.yelp.com/biz/passion-t-snacks-and-desserts-san-jose?osq=Desserts","text":"Passion-T Snacks and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/vampire-penguin-featuring-jastea-san-jose?osq=Desserts","text":"Vampire Penguin featuring Jastea"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-gelato-tea-lounge-san-jose?osq=Desserts","text":"Sweet Gelato Tea Lounge"}]},{"Business":[{"href":"https://www.yelp.com/biz/matcha-love-san-jose-6?osq=Desserts","text":"Matcha Love"}]},{"Business":[{"href":"https://www.yelp.com/biz/anton-sv-p%C3%A2tisserie-san-jose-2?osq=Desserts","text":"Anton SV Pâtisserie"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286383006},"timestamp":1513286383006,"sequenceNumber":0}}
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=10","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=10","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/oooh-san-jose-4?osq=Desserts","text":"Oooh"}]},{"Business":[{"href":"https://www.yelp.com/biz/hannah-san-jose?osq=Desserts","text":"Hannah"}]},{"Business":[{"href":"https://www.yelp.com/biz/chocatoo-san-jose?osq=Desserts","text":"Chocatoo"}]},{"Business":[{"href":"https://www.yelp.com/biz/nox-cookie-bar-san-jose?osq=Desserts","text":"Nox Cookie Bar"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-fix-creamery-san-jose?osq=Desserts","text":"Sweet Fix Creamery"}]},{"Business":[{"href":"https://www.yelp.com/biz/my-milkshake-san-jose?osq=Desserts","text":"My Milkshake"}]},{"Business":[{"href":"https://www.yelp.com/biz/matcha-love-san-jose-6?osq=Desserts","text":"Matcha Love"}]},{"Business":[{"href":"https://www.yelp.com/biz/banana-cr%C3%AApe-san-jose-2?osq=Desserts","text":"Banana Crêpe"}]},{"Business":[{"href":"https://www.yelp.com/biz/marco-polo-italian-ice-cream-san-jose-4?osq=Desserts","text":"Marco Polo Italian Ice Cream"}]},{"Business":[{"href":"https://www.yelp.com/biz/blackball-desserts-san-jose-san-jose?osq=Desserts","text":"BlackBall Desserts San Jose"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286384917},"timestamp":1513286384917,"sequenceNumber":1}}
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=20","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=20","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/anton-sv-p%C3%A2tisserie-san-jose-2?osq=Desserts","text":"Anton SV Pâtisserie"}]},{"Business":[{"href":"https://www.yelp.com/biz/soyful-desserts-san-jose-8?osq=Desserts","text":"Soyful Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/cocola-bakery-san-jose?osq=Desserts","text":"Cocola Bakery"}]},{"Business":[{"href":"https://www.yelp.com/biz/charlies-cheesecake-works-san-jose?osq=Desserts","text":"Charlie’s Cheesecake Works"}]},{"Business":[{"href":"https://www.yelp.com/biz/jt-express-san-jose-2?osq=Desserts","text":"JT Express"}]},{"Business":[{"href":"https://www.yelp.com/biz/nox-cookie-bar-san-jose?osq=Desserts","text":"Nox Cookie Bar"}]},{"Business":[{"href":"https://www.yelp.com/biz/shuei-do-manju-shop-san-jose?osq=Desserts","text":"Shuei-Do Manju Shop"}]},{"Business":[{"href":"https://www.yelp.com/biz/churros-el-guero-san-jose?osq=Desserts","text":"Churros El Guero"}]},{"Business":[{"href":"https://www.yelp.com/biz/sno-crave-tea-house-san-jose-4?osq=Desserts","text":"Sno-Crave Tea House"}]},{"Business":[{"href":"https://www.yelp.com/biz/j-sweets-san-jose-3?osq=Desserts","text":"J.Sweets"}]}]}]},"pageData":{"statusCode":200,"timestamp":1513286395948},"timestamp":1513286395948,"sequenceNumber":2}}

There is an [open issue](https://github.com/grpc-ecosystem/grpc-gateway/issues/26) in the grpc-gateway repository asking to replace the default behavior with the `text/event-stream` standard to satisfy `EventSource`, however, **no one has implemented this yet**. That is, the way grpc-gateway handles streams **does not** fit our needs to align with the eth2.0-apis standard. If we want the grpc-gateway authors to [accept our changes](https://github.com/grpc-ecosystem/grpc-gateway/issues/26#issuecomment-202184776), however, we should ideally implement this feature as a pluggable marshaler to the gateway instead of replacing NDJSON support entirely.

## Required Changes to gRPC Gateway

The main invariant we need to preserve when implementing changes in gRPC gateway is we cannot force developers to _think_ about the gateway when writing their applications. Developers should be able to define a regular gRPC service stream and not care about how the gateway will interpret it, i.e., it should _just work_. For example, we can define a basic stream endpoint as follows:

service BeaconChain {
  rpc StreamChainHead(google.protobuf.Empty) returns (stream ChainHead) {
    option (google.api.http) = {
        get: "/eth/v1beta/chainhead/stream"
    };
  }
}

message ChainHead {
    uint64 slot = 1;
    bytes beacon_block_root = 2;
}

and when using a client to call this endpoint using the `EventSource` browser API, we should see something like:

data: "{\"slot\": \"10\", \"beacon_block_root\": \"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf\"}"\n

The [/eth/v1/events](https://ethereum.github.io/eth2.0-APIs/#/Events/eventstream) endpoint from eth2.0-apis **requires** us to specify the event kind we wish to subscribe to via query-parameters. If not specified, we should return an error. The `EventSource` [specification](https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events) states that responses can include an `event` field which denotes the kind of event we are receiving. For example, attempting to subscribe to `eth/v1/events?topics=head` should return a response which looks as follows:

event: head\n
data: "{\"slot\": \"10\", \"block\": \"0x9a2fefd2fdb57f74993c7780ea5b9030d2897b615b89f808011ca5aebed54eaf\", \"state\":\"0x600e852a08c1200654ddf11025f1ceacb3c2e74bdd5c630cde0838b2591b69f9\", \"epoch_transition\": false, \"previous_duty_dependent_root\": \"0x5e0043f107cb57913498fbf2f99ff55e730bf1e151f02f221e977c91a90a0e91\", \"current_duty_dependent_root\": \"0x5e0043f107cb57913498fbf2f99ff55e730bf1e151f02f221e977c91a90a0e91\"}"\n
\n

### Solution

In order to support event streams, gRPC implementations of streaming endpoints can use a common protobuf message defined in the grpc-gateway package as:

message EventSource {
  string event = 1;
  google.protobuf.Any data = 2;
}

We maintain a [fork](https://github.com/prysmaticlabs/grpc-gateway) here which checks if a stream endpoint is returning that special proto message, and then performs custom marshaling to send the proper response data over HTTP of type `text/event-stream`.

An example ping-pong endpoint implementation would look as follows in Go:

func (s *server) StreamEvents(
    _ *pb.EventRequest, stream pb.Events_StreamEventsServer,
) error {
    ticker := time.NewTicker(time.Millisecond * 500)
    defer ticker.Stop()
    for {
        select {
        case <-ticker.C:
            data, err := anypb.New(req)
            if err != nil {
                return err
            }
            if err := stream.Send(&gwpb.EventSource{
                Event: "pong",
                Data:  data,
            }); err != nil {
                return err
            }
        case <-stream.Context().Done():
            return errors.New("context canceled")
        }
    }
}

## Required Changes in Prysm

We predict most changes required to support this feature lie in grpc-gateway rather than Prysm. Prysm currently uses a broadcast pattern for events at runtime which sends all events via feeds available as globals of a running beacon node. Any runtime service that needs these feeds can subscribe to them as needed.

A potential schema for the `/eth/v1/events` endpoint using grpc and protobuf could work as follows:

service Events {
  rpc Events(EventTopicsRequest) returns (stream gateway.EventSource) {
    option (google.api.http) = {
        get: "/eth/v1/events"
    };
  }
}

message EventTopicsRequest {
    repeated string topics = 1;
}

message ChainHead {
    uint64 slot = 1;
    bytes beacon_block_root = 2;
}

message Attestation {
    bytes aggregation_bits = 1;
}
...

A potential server implementation of the `/eth/v1/events` endpoint in Prysm could work as follows:

1. Parse the requested topics in the request query parameters, such as `head,attestation,voluntary_exit`
2. Conditionally subscribe to the corresponding event feeds based on the topic parameters
3. The response payload is a protobuf message including the event name and the actual data payload using the `oneof` pattern. The gateway then transcodes this data as needed into the `text/event-stream` MIME type
4. Gracefully handle disconnects and errors

### Supported Events

Prysm currently uses a broadcast pattern for events at runtime which sends all events via feeds available as globals of a running beacon node. Any runtime service that needs these feeds can subscribe to them as needed. These event feeds are available in the [beacon-chain/node](https://github.com/prysmaticlabs/prysm/blob/f2fe1f7683fb80272bd168e6a4a0a10be878e604/beacon-chain/node/node.go#L71) package.

The following are the required event types and whether or not Prysm already has an event feed for them which works today:

| Event Name | Supported? |
|:------ |:----------- |
| head | :100: |
| block | :100: |
| attestation | :100: |
| voluntary_exit | :cry: |
| finalized_checkpoint | :100: |
| chain_reorg | :100: |

## Work Estimates

1. Fork and clone grpc-gateway
2. Create a sample Go application with a tiny protobuf service definition, a minimal stream gRPC server, and a gateway exposed on localhost using our fork
3. Create a simple client which requests the stream endpoint from the gateway in step (2)
4. Fully understand the codepath which runs in the grpc-gateway when transcoding a stream response into NDJSON
5. Identify where marshaling logic for the `EventSource` standard can included instead of NDJSON, or _in-addition_ to it
6. Identify the requirements of protobuf response message needed to support grpc-gateway transcoding into the `text/event-stream` format
7. Include the changes in Prysm's `beacon-chain/rpc` package which allow for basic subscription of a single event using `text/event-stream`, such as `head`
8. Support all other required event types in Prysm

## References

- https://ethereum.github.io/eth2.0-APIs/#/Events/eventstream
- https://github.com/grpc-ecosystem/grpc-gateway/issues/26
- https://developer.mozilla.org/en-US/docs/Web/API/EventSource
- https://html.spec.whatwg.org/multipage/server-sent-events.html#server-sent-events
- http://ndjson.org/