# Acala Stats Dune Migration
## Legends
- ✅ ok
- ❌ blocker
- ⭕ solvable blocker
- 🤔 question/TBD

## Potential New Features with Dune
- query data using dune api
- more comprehensive data and new graphs/tables
- tx/user/tvl growth chart
- ldot/dot exchange rate history chart
- dex pool composition pie chart

## Dune Limits
### ✅ api limit
Api upload flow already tested.

Api upload size limit is 200mb, should be enough.

### ✅ storage limit
Storage limit for current plan is 15GB, should be enough.

Euphrates dataset is only 2mb, I doubt stats data will be 10000x bigger.

### ⭕ credit limit
Current credit usage: ~70% (16000/25000) each month. So if we add many new queries/dashboards, the credit might not be enough.

There are a couple solutions:
- reduce dashboard auto refresh interval (current: 12h)
- reduce/remove materialized views
- pay for for extra quota used (Price per extra credit: $0.025, or upgrade plan to have 100000 credit)

### ✅ functionality limit
Dune has all components on the stats page: line chart, volumn chart, card and table. So should be able to replicate everything on dune dashboards, as long as we have the data ready.

## Data Limits
### ⭕ data fetch and calculation
On top of the subql data, stats pages have some extra calculations. Dune SQL has limited calculation ability, so might be hard to replicate these extra calculation.

However, we can make subql do these calculation directly, and index the result. Subql exposes `api` for the indexer, so in theory should be able to calculate everything that is calculated by stats page.

### ⭕ data porter flow
Currently subqls are deployed on physical machines, and the db is running via docker compose, so it is not directly acceesible remotely.

**solutions**
- deploy new subqls to AWS. Old subql doesn't have enough data anyways. In this way data porter flow is already tested and running (preferred)
- research on how to setup db remote access to physical machines

## Migration Plan
### POC
- extend current subquries to index **minimum** required extra data on top on old data
- (optional) upgrade to latest subql version
- deploy to aws instead of physical machine, and only index latest `X` days/weeks
- implement data porter and run it **manually** to migration data to dune
- implement **POC** dune queries
- setup **POC** dune graphs, tables and dashboard

### Produciton
- (optional) index an acala dictionary, which will speed on all subquery indexing
- determine any new graphs or table that we need
- extend current subquries to index **all** required data for new features
- deploy to aws and index all historical data
- setup **auto** data porter
- implement **all** dune queries
- setup **all** dune graphs, tables and dashboard

## Questions
Technical:
- Do we need to upgrade subql version for all old subquries? They are 2 major verisons behind
- Do we need to index an acala dictionary to speed up indexing?

Features:
- What new graphs/tables do we need