changed 3 years ago  
Linked with GitHub

Substreams in subgraphs

GIP: XXX
Title: Substreams as a Subgraph
Authors:
Created: <The date the GIP was created.>
Updated: <The date the GIP was last updated.>
Stage: Draft

Abstract

Substreams are a new streaming-first blockchain indexing technology developed for The Graph Network, focused on performance and composability.

A substreams package is a collection of interdependent modules, which combine to extract and transform blockchain events in a highly configurable and parallelisable way, producing a streaming output.

This GIP describes the simplest possible integration of Substreams with Subgraphs, to take that final streaming output and make it persistent and queryable.

Motivation

Adding Substream support to Graph Node is the fastest way to bring the performance and composability benefits of substreams to The Graph Network.

High Level Description

specVersion: 0.0.X
schema:
  file: ./src/schema.graphql
dataSources:
  - kind: substreams
    name: Uniswap
    network: mainnet
    source:
      package:
        moduleName: output
        file:
          /: /ipfs/QmbHnhUFZa6qqqRyubUYhXntox1TCBxqryaBM1iNGqVJzT
          # This IPFs path would be generated from a local path at deploy time
    mapping:
      kind: substreams/graph-entities
      apiVersion: 0.0.X

This introduces a new substreams "kind" of dataSource. This dataSource is identified on the subgraph.yaml by a name and a filepath. This filepath will point locally to a Substreams package, which will be uploaded to IPFS on deployment.

Subgraphs with a substreams dataSource can only have that single dataSource.

Graph Node configuration

In order to support substream-based subgraphs, Graph Node will need to be Substream aware. An additional substream endpoint will be required for each network which Graph Node will want to support.

This can be an extension of the existing provider configuration in Graph Node:

[chains.mainnet]
shard = "main"
protocol = "ethereum"
provider = [
  { label = "substreams-provider-mainnet",
    details = { type = "substreams",
    url = "https://mainnet-substreams-url.grpc.substreams.io/",
    token = "exampletokenhere" }},
]

Substreams may require dedicated error handling / retry logic, though a lot of the requirements should be covered by the Firehose cursor-based integration.

Mapping entities

Substreams have their own protobuf schemas (documentation). However to directly copy the raw substream schema would be to lose the ability to link and traverse entities in the resulting GraphQL schema.

Utilities should certainly be provided to auto-generate a .graphql file from a substream protobuf schema, with intelligent type mapping. However once generated, developers should be able to update the .graphql schema, to identify connections between entities, and to add derived fields. Therefore there is still value in having a schema file as part of the subgraph definition.

This proposal does assume a tight coupling between entity names in the substream, and entity names in the subgraph

Subgraph Entities should be able to specify a subset of the overall Substream protobuf schema (i.e. it doesn't need to be 1:1)

Updating entities

In order to support performant querying, Graph Node stores full entities at every block where those entities change. It supports "upsert" type behaviour, where saved values are merged with existing entities.

Substreams currently provide entity "deltas", which just capture the changed values. Therefore there is a requirement to fetch any prior entity values to create the full entity to store. This could be done on either the Substream or Graph Node side of the integration, so we should work through the trade offs & alternatives.

Graph Node will still need to "close" the block range for any existing entities, which could negatively impact indexing performance.

Deterministic indexing

A fundamental requirement is deterministic indexing, specifically the generation of a Proof of Indexing, which can then be cross-checked across indexers.

Graph Node makes the assumption that all upstream datasources are deterministic, and will make the same assumption about Substreams

Depending on the implementation in Graph Node, this could leverage the existing Proof of Indexing setup, or it may need a dedicated Proof of Indexing in Graph Node.

Handling re-orgs

Graph Node will still need to be re-org aware. The substreams connection should provide the relevant information (which blocks to remove). Graph Node will need to remove entities from those blocks, and re-open the block range for older entities. This functionality already exists in Graph Node, but might need to be updated for the new data source.

Linear process & batch back-filling

Graph node will need to support linear processing, e.g. when indexing at the chain head. There is a possibility that linear processing historical blocks becomes a significant performance bottleneck. In that case, a dedicated "batch" insert implementation might be required to load historical data into the subgraph.

Monitoring

Graph Node's instrumentation & monitoring may need to be updated for the new type of indexing (e.g. tracking indexing time per block etc).

Graph CLI

Graph CLI will need to be updated for the new type of data source. Graph CLI could also include the helpers to auto-generate .graphql files for a given substream.

Backwards Compatibility

This functionality is purely additive. Graph Node may need a new specVersion given the new structure of the manifest, but this could also be handled in the same way as the introduction of new protocols, where Graph Node versioning provides the necessary guarantees.

Dependencies

  • Substream support for a given network requires a Firehose implementation for that network.
  • Requires determinism from the upstream Firehose & Substreams
  • Substreams introduce additional infrastructural requirements for indexers
  • This integration may require some changes as part of Substreams

Validation

Determinism - are indexing results consistent?
Performance - are the performance gains as expected?
Scaleability - can Graph Node support 1-2K subgraphs powered by substreams?

Rationale and Alternatives

  • Alternative Postgres sink
  • Less opaque integration

Copyright Waiver

Copyright and related rights waived via CC0.

Notes

2022-07-18

  • Julien & Filipe to meet to specify the to-do list
  • Use a "hello world" substream as an initial basic substream
  • Considerations - how best to create the proof-of-concept
  • Challenge - taking protobuf model - stream of changes, merge them into the Postgres entities
Select a repo