owned this note
owned this note
Published
Linked with GitHub
# graph-cli: update & add
## Context
We want to improve the developer experience for boostrapping & updating subgraphs. A range of tools have been developed by the community to streamline this process (token-terminal, graphgen, scaffold-eth, Jolly Roger, Sushiswap, Synthetix, OpenZeppelin, CirclesUBI).
This document proposes two features which will make it easier for developers to update and add to their subgraphs.
This functionality will then be usable in a dedicated Graph Protocol [hardhat plugin](https://hardhat.org/plugins/) for building subgraphs alongside smart contract development.
## User Stories
*As a subgraph developer...*
### I want to update my subgraph configuration for a new network or deployment
While I am working on a subgraph, I may want to update its contracts or networks:
- During local development, if I deploy a new version of the contract, and want the subgraph to index that newly deployed version
- If the protocol is deployed on multiple chains, I may want to redeploy the same subgraph on a different chain, with the addresses and startBlock for the contracts on that chain
*Currently they have to update the `subgraph.yaml` by hand, or develop their own solutions - indeed they are [directed to do so in The Graph's documentation](https://thegraph.com/docs/en/hosted-service/deploy-subgraph-hosted/#deploying-the-subgraph-to-multiple-ethereum-networks)*
**We should allow subgraph developers to easily update their subgraph configuration for a network during development using `graph-cli`**
**Proposal**: Create a simple `networks.json` standard, which can be referenced during `graph build`
`networks.json` specification:
```json
{
"mainnet": { // the network name
"dataSource1": { // the dataSource name
"address": "0xabc...", // the contract address
"startBlock": 123456 // the startBlock
},
"dataSource2": {
"address": "0x123...",
"startBlock": 123444
}
},
"optimism": {
"dataSource1": {
"address": "0x987...",
"startBlock": 123
},
"dataSource2": {
"address": "0xxyz..",
"startBlock": 123
}
}
}
```
We can then add `--network` as an option in `graph build`
```bash
graph build [options] [<subgraph-manifest>]
New options:
--network <name> Network to use from the networkFile
--networkFile <path> Network file (default: "./networks.json")
```
When running `graph build`:
- If a `--network` is specified:
- `graph-cli` should iterate through the dataSources in the specified `<subgraph-manifest>`, updating the `network`, `address` and `startBlock` based on the specified `network` from the `networkFile`
- There will need to be slightly different treatment for different `protocols` (e.g. NEAR has `accounts` rather than `addresses`)
- Then the existing `graph build` function should be called
- If any dataSources in the subgraph manifest are not updated, this should be logged by `graph-cli`
- If any dataSources in the configuration file are not used, this should be logged by `graph-cli`
When running `graph init`, a `networks.json` should be auto-generated based on the provided information. *Users will then be able to easily update or add additional networks.*
**Implementation alternatives:**
- Use templating, or environment variables ([as described here](https://thegraph.com/docs/en/hosted-service/deploy-subgraph-hosted/#deploying-the-subgraph-to-multiple-ethereum-networks)), rather than a typed JSON configuration file
- This is a less robust interface for third party tools (e.g. a Hardhat plugin) to integrate with
- Create a dedicated command for updating the configuration
- This alters the developer workflow
### I want to add a new data source to an existing subgraph
When I start building a subgraph, I can use `graph init` to scaffold a simple subgraph with one data source. However if I want to add a new dataSource, I must do that manually.
**We should make it easy for developers to add new data sources during development**
**Proposal**: add a new `graph add` command, which adds a new data source.
```bash
graph add <address> [<subgraph-manifest default: "./subgraph.yaml">]
Options:
--abi <path> Path to the contract ABI (default: download from Etherscan)
--index-events Index contract events as entities
--contract-name Name of the contract (default: Contract)
--merge-entities Whether to merge entities with the same name (default: false)
```
- This should fetch the ABI, and create a dataSource in the same way that `graph init` creates a dataSource `--from-contract`, updating the schema and mappings accordingly.
- The `merge-entities` setting identifies how the developer would like to handle entity and event name conflicts:
- If true: the new dataSource should use existing handlers & entities
- If false: a new entity & event handler should be created with `${dataSourceName}{EventName}`
- The relevant data (address & startBlock) should be written to the `networks.json` for the relevant network
> There are some failure modes where the existing subgraph has custom mappings, but those are out of scope, and should generally be avoided in the default
Changes to the existing `graph init` flow:
- At the end of adding `--from-contract`, ask if the developer wants to add another contract (and if so, initiate `graph add` for that contract)
## References
- [Hardhat plugin forum post](https://forum.thegraph.com/t/hardhat-graph-plugin/2863)
- Graphgen: subgraph generation based on annotations
- Tokenterminal have a sophisticated setup for initialising subgraphs
- [Synthetix subgraphs](https://github.com/Synthetixio/synthetix-subgraph)
- [OpenZeppelin subgraphs](https://github.com/OpenZeppelin/openzeppelin-subgraphs#readme)
- [Amxx utils](https://github.com/Amxx/graphprotocol-utils)
- [OpenZeppelin generate](https://docs.openzeppelin.com/subgraphs/0.1.x/generate)
- [Synthetix PR to support generated manifests](https://github.com/graphprotocol/graph-cli/pull/642)
- [No code subgraph builder](https://www.notion.so/edgeandnode/Low-code-Helper-to-Create-Subgraphs-8af0394667904f47bbf63deac89fdf51)
- [CirclesUBI](https://github.com/CirclesUBI/circles-subgraph)
Sign in with Wallet
