The Graph Migration

--- tags: v3 --- # The Graph Migration July 29, 2022 Call with @kyleArojas#8414 and @schmidsi#9465 The Graph grant program ends in September. They are willing to give us 50,000 GRT to migrate the Mainnet subgraphs. They will have additional grants for other networks soon. They can provide a detailed estimate of how much our query fees would be. And are open to feedback if we feel they are too expensive. Right now they are thinking quality service will be around 5000-7000 GRT (600-900 USD) per month. This may only be for one subgraph though. Max fee setting is related to quality of service. And we can tune the fee and optimize the subgraphs to reduce it. We may also earn a portion of the query fees. ## Process They are willing to walk us through the migration process on a 30 minute call. We will need someone for KYC and DocuSign and to make sure the subgraph is compliling and APIs keys are set-up. - KYC ourselves - can we use some sort of warcamp entity here? - Create API keys - Load up GRT on Polygon - Deploy in Subgraph Studio - Sam has done this for the 1 main subgraph - https://thegraph.com/studio/subgraph/daohaus/ - Publish and Signal They can then help us with testing until we feel comfortable. ## Other Options - Set-up our own node - Hosting our own indexer (requires DevOps) ## Notes from 8/9/22 Sync - Talking to the Graph about migrating Mainnet subgraphs and the price is pretty prohibitive for even 1 - What do we need to do to combine into one subgraph? - Do we have other options for running our own node and indexer? - is this desirable? - Could we monetize? Would it be worth it? ### Mainnet - Sam got main graph deployed to the Studio and started the KYC process to get the grant - This is in progress - Likely wont' be needed to combine the subgraphs but could take this on if we need - Other ones are pretty low traffic so we may be able to get some GRT onto those once we figure out how it works - If GRT runs out and we don't use the sg much anymore we could let it go - What about v3? We'll be adding at least double (or thereabouts) for the period when we'd have v2 and v3 up and running - Based on current traffic it was around $600 - $900 estimated costs for the Mainnet graph - We'd only need to migrate Mainnet since they don't have a plan for the other graphs -- in the new year they're turning off **all** hosted support - Why is our v2 Mainnet graph getting so much action? - Landing on the site -> going to the app -> queries up front - Explore section uses it, background scripts are using it such as Discord bots, third parties use it like DeepDAO and MetaGame - What is the plan to stop people from spamming to increase costs for teams that are hosting the specific Subgraph? - We want to encourage people to build on top of our tech, but this may increase our costs as we get adoption - We'd be bearing the costs of a project's API requests (or we restrict it) but we want folks to use it and build - Is there an option for people to pay for requests through the Graph? - Would we be able to do metered billing for third party services? - Could set up API keys but this stifles development #### Options - **Some sort of billing / rates for devs** - Would we have an estimate of a per query basis - .0003 per transaction query - Could leverage Lit to have a subscription or rate limiting service - Free tier to have folks get started - Developer programs -> stake HAUS to get access to the second tiers - **Do we want to put barriers in front of people using our tools?** - If we show value in our tools and folks using them we may be able to grants to help support costs - **Even if we don't use the service, we're still going to need to do work such as rate limiting** - As folks increase usage, there will be costs that'll increase as our adoption gains - Servers, dev ops time, etc. - What would our own service cost? - **We become an indexer on their network** - What would this take? - Would this help with our overall costs - We are going to push forward and getting the Mainnet Subgraphs into the Studio - Sam going through this process and can push forward - Will buy us 6 months - How much would our costs increase as we scale up usage? - Costs for servers, traffic, dev time, etc. - There are also several other graphs that we use that others have done -- what do we need to plan for if something happens to these? - How many DAOs are using our Graph on Mainnet? - Only support L2s? - This could be an option but we're only delaying the decision until they add the Studio to the L2s, which is likely ### Existential Crisis and Strategy - So dependent on the Graph, what would we do if we can't afford this? - It could sink us - Indexer requirements: 100K GRT (~$14000) - 2x the amount we'd get for the grant - GRT price will likely head up in the coming months - This is a 'cost' but we'd also be staking it, so we could always unstake and pull it out - We'd need to all coordinate to get the 100K GRT and *stake it* to even have the option - There is also hardware requirements and an IPFS node - Cost would be in *running* the full archive node -> lots of storage and needs to be able to handle the indexing - How would we handle this in a decentralized way? -> Kubernetes / Digital Ocean setup - 4-core 8gb CPU is $48/month on DO droplet - 1tb disk space, 30gb ram - Is the IPFS node on the server (it's separate on their list of key requirements) - Based on the requirements, we'd be paying around **$200/month** - Are there optimizations we could do to decrease our Graph queries? - Could we index it onto a cheaper form on our own servers and host it? - We may also be making unnecessary queries - Does this need to be SSD or could it be S3 - How much time would this take per month and how much would we need to pay? ### Questions - What tools does The Graph have to allow for *other users* of our Subgraphs to pay for usage? - What is our volume of queries? - Are there any gains for us for being an indexer vs. paying for indexing? - If you're an indexer you end up paying *yourself* the fees and may be able to index other SGs that you could get paid for - This seems like a good option for us as it could at least be revenue neutral - What are the costs for this path? - DevOps team on call to fix any issues or we'd lose some of our staked GRT (we'd get slashed for being down) - Node management - Costs won't be too bad since we won't be making code updates -- they'd restart the container - Ongoing maintenance could be related to upgrading nodes (graph and ipfs) and making sure the cluster is stable - Infrastructure and hosting costs (DO droplets) - What flexibility do we have on price of the query fees? - If we have a price point that's good for us, could we provide that feedback and see how we could get there: - Optimizing our queries -> does this change the price? - Per query fee and it's a market so you're offering that you put up and the indexers then accept it -> bid for quality of service - Is the price in $ or based on current GRT price? - Scary part of this is that things get out of control - There should be a cap -- advising on how to get back to a normal range when there is overages - Could result in runaway budget and costs - How does this work for community funded projects that aren't VC backed startups or DeFi projects with massive treasuries? - Marketplace - Query fees are split between the indexer and the curator -- in setting ours up, we could be both entities - Could we offer this as a service to DAOs on our platform? - Would this be a viable source of revenue for us? - If there is a way for folks to pay for their own hosted services easily this is likely where we'd want to move - We want folks to *use our tools to make our software* but we don't want to have to pay as folks adopt our services - If we go this route, we'd have to pay to charge for services we provide - Over next 6 months we'd want to push for as much v2 -> v3 adoption as possible and provide a white label solution to migrate to v3 - Push the fees off to each DAO - We want folks to be able to start up DAOs easily and then evolve to the space where you can no longer use without paying - We probably need a similar angle - Where are our queries coming from? - Is it from people scraping our data or is it from our dailiy use? - A lot of it is likely from our own caches and services - If we're running an update every 5 minutes then other services may run it every second - **How can we investigate the source of our traffic?** - Are our queries inefficient? Could we optimize? - If a lot is coming from outside sources this would be potentially a revenue stream (split between indexer and curator) - We pay it and then it's up to us to figure out how to pass the costs on to the people querying