---
# System prepended metadata

title: Infrastructure and Data/Indexooors Sync (5/18/22)
tags: [' ops', ' data', v3]

---

---
tags: v3, ops, data
---

# Infrastructure and Data/Indexooors Sync (5/18/22)

- Thinking through from the Hub perspective, mainly
- Hub table: Working to provide cross-chain queries to get all of a user's DAOs and certain data points, but certain fields like *Vaults* is trickier
    - Making call to Gnosis to get balance info and totals it and returns this to frontend
    - Fine with a low number of DAOs, but this wouldn't scale with many DAOs (or in an Explore view, or Core UI) so we'd likely need to integrate some form of caching 
- Thought through this in context of a Lambda or Ceramic
    - Have jobs that run on crons:
    - 1) Pulling safe addresses and puts into a queue
    - 2) Pulls from the queue and processes (hit Gnosis for token data, other sources for NFT data, etc) and roll up into a job that would make this available as a stream that is pushed into Ceramic
    - Frontend or SDK would be able to hydrate all of the DAO data when it's needed
    - Can we derive document identifier and then hit Ceramic
    - This would eliminate the S3 layer, but there's questions about *where* we want to do this processing?
    - Lambdas vs. Kubernetes clusters?
- Concerned about adding more centralization to the stack given the management overhead we ran into with v2
- Don't see much of a way around this with the Hub and other views
    - Likely will need to manage some infrastructure (do we run this on a decentralized compute protocol or do we run it on something like AWS or Digital Ocean)
    - If we're doing this, limiting the scope and making it easy for other devs to replicate is crucial
    - Ideally, we'd be able to add this to a 'recipe book' for other devs to use
    - Lambdas and Kubernetes would be open source and folks could follow our instructions and set this up themselves -- would need to change the metadata definitions
    - Recreate the same docs with different stream identifiers
    - Replicate keys that we're using as well
        - Ceramic definitions could live in a npm package in the monorepo
        - Since using a different wallet address there would be a different stream id
- Ceramic: Would need a DH controlled wallet that controls the streams
    - No funds in the wallet
    - Hide the private key, but no real damage if compromised
        - If compromised, change the seed phrase and get the DID and update the frontend to use this DID to derive the stream id

### Kubernetes vs. Lambdas

- See the [pros/cons doc](https://github.com/HausDAO/daohaus-monorepo/issues/126) for more detail
- Even if using Lambdas it would be different than we currenlty use them
- Set up a queue system within Kubernetes
- Jobs should be automatic -- would need to set a cron schedule
- Overhead is making sure that the queue system is up (Redis instance) -- troubleshooting overhead if something goes down
- Lambdas are less portable -- we'd be in the AWS box
    - Is there a way to do more generic functions that could be deployed to whatever cloud provider?
- Kubernetes would be a Docker cluster that could be put onto any Linux server (or use a managed service)
- [Helm Charts (Playbook)](https://helm.sh/docs/topics/charts/)
    - We could define the app needs in this Playbook where folks set in values
    - Similar to Terraform where folks define config and it would spin up the clusters needed to support the app infrastructure
- Configue Lit to handle permissioning around the accounts
    - What does Lit support?
- AWS is so fully featured and would require someone who is dev ops focused to permission properly or there is a concern around tech debt, whereas Kubernetes would be 1 dashboard
- Making infrastructure easy and direct to set up for folks would be critical
- **Possible future state** 
    - Would be able to share the job queue and central *write* queue among nodes of clusters? This way it could be round-robined and share the load of who is querying
    - Would make the deterministic DID a bit tougher, but this would be really cool to explore
    - Ceramic is working on multi-account DIDs, so we could potentially have a trusted network of infrastructure folks that we'd give write control to the write DID
    - All share the DID but multiple wallets control the DID
    -  Leave the door open to fully decentralize process
- Being able to stand up the API anywhere would open doors for more decentralized frontends
- Check out Golem, Akash, and possibly Spheron
    - Avoiding AWS lock-in could be a good idea for helping us transition to potential web3 compute platforms 
    - What are the security implications of these? What are the parameters?
        - Are these less secure?
- Trueblocks.io
    - Look to this for inspiration for docs and process

## Ops Infrastructure (Ceramic)

- Kubernetes infrastructure needs:
    - Ceramic schema and Redis instance for managing queues
    - Start with token prices
- Rage on the infrastructure design, and then move into Sage
    - ERC20 token balances and totals in Eth and USD
    - Teaches us about how to set up the infrastructure (bull, and the jobs)
        - Chance to spin up our own DAO Digital Ocean instance and demo the admin dashboard
- If we like this approach we could potentially transition v2 to this and use v2 as a proving ground
- Ticket for setting up the cluster, ticket for Ceramic node, and ticket for the jobs 
    - Tokens
    - Deploying
    - Set up cluster and Ceramic node
        - First set of docs / demo once we get this locked in
        - Dashboard overview
        - Ceramic node running in cluster
    - Include the Helms Charts in our repo

## Approaching Data Processing

- How do we want to approach data processing? Do we want to use Lambdas to do this?

## DAO Metadata

- Are we storing this in Poster or would we want to use something else?
- Moving more into Poster -- test some of this out within v2 
    - Data model for this and how it could play with the subgraph
- Hub DAO metadata:
    - DAO avatar
        - Not in here yet since it'll come with Core UI
        - Name is added through Poster and indexed in subgraph
    - Keeps our data sources in 1 call
    - Name + avatar part of initial subgraph call (IPFS hash dropped in)
    - DAO metadata would live in our main subgraph
- Explore the new process for infra reimbursement/payments