# Gateway spec

## Defining new features and verifying them

### Definition of Ready
- clear gloals/output defined for the feature

### Definition of Done
- when applies, version with changes deployed in our infra
- tests written or manually tested
- instructions defined for how to test/verify
- PRs merged in release branch

## Features

### V1
- [x] Chain indexing
- [x] Upserts
- [x] Upserts for blocks
- [x] Upserts for Arweave transactions
- [x] Upserts for Bundlr transactions
TODO: review the code, this might be done
- [x] Upserst for Bundlr transaction receipts
Is this really needed?
- [x] Store all transaction tags in plain text
Currently Arweave transaction tags are stored in `base64url_nopad` encoded while tags for Bundlr transactions are stored in plain text. All transaction tags should be stored in plain text so that we can index and search them properly. For Arweave tags, assume UTF-8 character encoding when decoding and storing the data.
- [x] review peer/node praise/punnish logic
Looks like the current code is too agressive and does not praise
nodes enough. Currently almost all nodes have minimum rank.
- [x] Fork removal
- [x] Recheck working with indexing & sync
- [x] Automated peer crawling
- [x] Regularly launch a job to crawl the network to find new nodes
- [x] Require peers to use software release 57 or newer
- [x] Bundle indexing
- [x] Fix Rust-SDK
- [x] Implement EIP-712 verification
- [x] Include bundler information in Bundlr transactions
- [x] Add information of top level transaction
- [x] GraphQL (see if test query works)
- [ ] `GET /tx/:txid/data` -> identical to `GET /:txid` with size limit
NOTE: implemented, but not tested
- [x] `GET /:txid`
Get transaction data as a single download.
- [x] Getting data
- [x] Get data from cache
- [x] S3 data service
- [x] Get data from Bundlr optimistic cache
NOTE: implemented, but not tested
- [x] Get data from Arweave
- [x] Cache data into S3 while returning to user
- [x] Download data chunk by chunk
- [x] Extract Bundlr transaction data from an Arweave tx data
- [x] Manifest parsing
See: https://github.com/ArweaveTeam/arweave/blob/master/doc/path-manifest-schema.md
- [x] `GET /:txid/:path`
NOTE: implemented, but not tested
- [x] Read into memory and parse
NOTE: implemented, but not tested
- [x] `GET /tx/:txid`
- [x] Get from DB
- [x] `GET /tx/:txid/status`
- [x] Get from DB
- [ ] `GET /tx/:txid/offset`
Can this be cached? In what circumstances can the offset change
after it's been fetched once from miners? Can we only cache offsets
after 50 blocks? Yes after 50
- [ ] `GET /chunk/:offset`
- [ ] Get from miners
- [x] `GET /peers`
- [x] Get from DB (probably get all peers with reputation > X)
- [x] Peers data, should match with arweave.net: `["host1:port1", "host2:port2", ...]`
- [x] Receipt retrieval
- [x] Get receipts for each Bundlr transaction
Receipts are received over the websocket connection and saved in the database
- [x] Bundlr optimistic GraphQL
- [x] WebSocket listener for new transactions
- [x] Get receipt for each transaction
- [x] Auto reconnect upon failure
- [x] Get block data
- [x] `GET /block/hash/{hash}`
- [x] `GET /block/height/{height}`
- [x] API does not proxy requests to arweave.net

### V2 (due: 2023-08-24)
- [x] apalis upgrade
Move to latest version. The current one is pretty unstable and job scheudling does not return job ID for the new job that makes logging more painful. Also, using redis as the backend makes it pretty painful to debug scheduling issues. Also all examples and docs are using postgres backend so it's probably more stable.
- [x] upgrade to latest stable
this require some changes in our abstractions too, because newer apalis versions use ULIDs for job IDs instead of UUIDs
- [x] Nested bundle indexing
- [x] arweave-indexer to process nested bundles
- [x] create shared indexer lib from arweave-indexer that can be used with bundler-indexer for scheduling nested bundle processing (1md)
- [x] create separate app for indexer workers (<1md)
- [x] use indexer lib for scheduling nested bundle processing in bundler-indexer (<1md)
- [x] extract transaction data download code from indexer lib and move it to common lib so that api code can use it (1md)
- [x] implement nested bundle support in api (1md)
- [x] Improved data dowloading performance (see [node.js implementation](https://github.com/Bundlr-Network/Whistleblower/blob/master/src/utils/txDownloader.ts)) (2md)
- [x] Improved `GET /<txid>`
- uses same code as the indexing part, so partly done
- [x] Improved bundle downloading during indexing
- this is partly done, but requires review of the code
- [x] Cache-based endpoint retrieval. Get price from N miners and choose the "most common" price that is returned. i.e. consensus on price (1md)
- [x] Price cron
- Get the price from N nodes
- price is per chunk (ceil)
- req path is /price/{bytes}
- Calculate average price from responses
- Store price in DB (or redis)
- [x] Price data API route
- read average price from db
- [x] Fallback checks (<1md)
- [x] Bundler check as a fallback
If transaction is not found, check bundlers if they have it.
When websocket connection is lost, we will miss transactions and currently the bundler indexer does not handle this correctly. So we need to have a fallback for checking transactions directly from all bundlers if not found in out db.
- [x] Range request support (HTTP ranges) (<1md)
- [x] `GET /raw/:txid` returns tx data without parsing manifest (<1md)

### V3 (due: 2023-11-03)

- [x] Bundle index filtering so we can choose to not index certain data (1md)
- [x] DB indexing of manifests (3md)
- [x] WebSocket support for txs and blocks (1md)
- Use redis for websocket connections and push events from the backend to redis
- [x] GraphQL subscriptions (3md)
Implemented now, but schema and filtering is not quite there. See the separate item for reviewing websocket and GraphQL events.
- [x] ANS-106 support (2md)
Support *Do-Not-Store* transactions: https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-106.md
- [x] when received *Do-Not-Store* TX, flag TX in the DB accordingly
- [x] when data is requested for a *Do-Not-Store* TX, then return no data (which status code to use? 404, 451) - USE 451
- [x] only the owner of a tx can ask data not to be stored, if *Do-Not-Store* tx is requesting another tx to be blocked but owners do not match, then ignore
- [x] Return raw manifest if index is NOT set (<1md)
- [x] Set HTTP caching related headers correctly (2md)
To improve client or CDN level caching we need to set different headers right that affect this behaviour.
- [x] Set Date header correctly in metadata and data download paths (set to max for final (>=50 confs) and set to 2 mins for non-final (<50 confs))
- [x] TX data cache cleanup (<1md)
TX data should be removed from the cache after 1 month since the last access (or when not accessed, 1 month since indexing)
- it seems this is more complicated than I thought. Defining rule for expiring object 30 days after creation is very simple thing to do. Cleaning up objects 30 days after last access, not so. There is an AWS article about how to do this, but it's not very simple: https://aws.amazon.com/blogs/architecture/expiring-amazon-s3-objects-based-on-last-accessed-date-to-decrease-costs/
- Q: should we still go for 30 days after last access or can we do 30 days after creation and later have the more complicated approach implemented?
- A: as agreed in Discord, implementing now 30 days expiration since upload and added item in the backlog for implemeting 30 days expiration since last access
- [x] Remove `common::arweave::BlockInfo::indep_hash`
Parts of the code that uses this function do **not** handle old transactions correctly. Once we start processing **all** historic data, this must be fixed.
- [x] Deprecate `common::arweave::BlockInfo::indep_hash` to find uses
- [x] Refactor code to use `common::arweave::BlockInfo::id` instead or add new functions if this does not work

### V4
- [x] Full GraphQL support (2md)
- [x] Support 3 query types
- [x] [transactions](https://gql-guide.vercel.app/#transaction-data-structures)
- [x] [blocks](https://gql-guide.vercel.app/#block-data-structures)
- [x] GraphQL pagination (2md)
- NOTE: Cursor = base64url(tx_id)
- [x] GraphQL support for filter by timestamps (<1md)
NOTE: implemented, but disabled, because enabling indexes are removed from the database to make database smaller.
- [x] Test and optimize GraphQL queries
Main queries that need to be fast:
- by id
- by list of ids
- by owner
- by list of owners
- by list tag names
- by list of tags
- all combinations of the above
- [x] Redesign database schema for smaller storage needs and cost
- [x] Use `postgres` as `apalis` storage
The `redis` storage seems to have quite a lot problems with it while SQL based storage implementations might work better. Also, switching to `postgres` storage allows easier queries for failed jobs so that we can rerun them or trigger new jobs for trying to process those failed jobs
- [x] Fix `previous_block` always to use `indep_hash`
Code has a mistake that expects V1 blocks (and first V2 block) to use `hash` value when linking to previous block. All blocks should use `indep_hash` of the previous block and all blocks should have valid `indep_hash` value.
- [x] Owner to Owner address derivation logic
Implement logic for resolving owner address for Arbundles. Expect all bundles to be arbundles and if the this fails, leave owner address as null.
- [x] Database improvements
- [x] store whole parent chain with the transaction data
Include all parents as a list with the transaction data so that we can cheaply fetch this information from the DB. To consider: maybe store this only with bundles so that we avoid copying this data with every single row for bundled txs.
- [x] Improve schema (use tx and tx_filter approach)
- [x] return tx status information when querying tx information (block.height, block.indep_hash, block.hash, confirmations) from the db. This is needed so that we can apply caching related headers correctly without separate database query. We probably need to separate TxRecord (inserting data) and TxQueryRecord (getting data)
- something like this would work as a single query: `SELECT t.*, b.indep_hash AS block_indep_hash, b.hash AS block_hash, b.height AS block_height, head.height - b.height + 1 AS confirmations FROM (SELECT MAX(height) AS height FROM blocks WHERE trunk = true) AS head, transactions AS t LEFT JOIN blocks AS b ON t.id = ANY(b.txs) WHERE t.id = 'NUNGPtYcy-ZCk4gtfbQC0Ne3oh_bLRAcHezoUADRQTQ' AND b.trunk = true;` though this needs to be analyzed properly if this could be optimized to be faster.
- [x] Define block version heights and use them for deserialization
- v1: block height < 269510
- v2: block height >= 269510, < 422250
- v3: block height >= 422250, < 633720
- v4: block height >= 633720, < 812970
- v5: block height >= 812970, < 1132210
- v6: block height >= 1132210
- [x] Update BlockInfo to cover new block versions
Currently we support block structures up to V3 (all where block.height > 422250). More recent block data from arweave.net shows new fields that we are not aware of. Old docs describing these structures seem to be gone and we need to find new source to track for changes in block data structures.
- API docs seem to be available again in https://docs.arweave.org/developers/arweave-node-server/http-api
- [x] Indexing historic data by defining block range
- [x] Chunk database upserts for ANS104 TXs when procesing bundles
- [x] Skip node rank checks (and updates) when downloading TX data
- [x] Use EFS as local TX data cache in indexer
- [x] Redesign indexer worker to allow multiple small workers in separate EC2 instances
Run multiple indexer works in small EC2 instances so that we can avoid getting rate limited by miners (this needs TX data to be cached in EFS)
- [x] Add support for running VACUUM between chunks of processed blocks

### After V4, parking the project

After V4 project was parked. V4 main focus was on indexing historic data and during the release, big changes were made for the indexer. Because of these changes live indexing and gateway API were left with lower focus so those are not current well tested.

Also the infra got destroyed and creating complete infra from scratch was never tested so there might be things in the Terraform code that do not work out of the box when trying to bring complete infra up.

### V5 (due: ???)
- [ ] Handle duplicate TXs
- [ ] Handle situation when bundle has same TX multiple times. For example bundle `LPdwhCNkV8aoJ20KnzVwdyA_-tR0SX5qqWyFH-DcUfM` has twice the TX with ID `a6bz7G7KTmhtou0j54EG_j4LyJsapvAcT7G62tcrN1Q`. Open question: are these two entries in the bundle identical? Right now the code fails on this situation.
- [ ] Handle case of TX ID collisions. For example when two bundles have different TXs or bundle has TX with same ID as one of the L1 TXs? Right now our upsert logic simply updates the existing TX with new data, but this code requires review to find out if this update is complete or partial (leaving inconsistent state).
- [ ] Route for pushing transaction metadata (2md)
Webhooks for receiving updates from bundler nodes
- [ ] register webhook to each bundler node
- [ ] webhook endpoint for receiving transaction messages
- [ ] webhook endpoint for receiving transaction receipt messages
- [ ] Bundler indexer, on websocket reconnect check for missed transactions (3md)
When doing initial websocket connection building or in case of reconnect, check for missed transactions. The connection logic should be as follows:
1. check when did we receive the last transaction (as `last_tx_moment`)
2. create websocket connection and start processing transactions
3. wait to receive first transaction (as `first_tx_moment`)
4. get all transactions that are not yet bundled between `last_tx_moment` and `first_tx_moment`
- [ ] filter out nodes that do not define `network = "arweave.N.1"` (< 1md)
- [ ] Split GraphQL, WebSocket and REST API in separate apps (1md)
These parts should be split in separate apps so that we can do better load balancing. GraphQL (because subscriptions) and WebSocket need more resources, because they have long living connection with the server while REST API can be significantly more light weight.
- [ ] Refactor API components to need only DB read access (2md)
API should not need write access to the DB, instead it should only need read access to allow better DB performance. Only indexer workers should need write access. Currently API side needs write access to the database so that it can write ANS106 (Do-Not-Store) and manifest data when indexer has not yet written it (probably quite rare use case). Also it needs DB write access to update node ranking when downloading transaction data (this needs better mechanism).
- [ ] Separate miner ranking for metadata requests and tx data requests (2md)
Some miners don't serve unpacked tx data and when trying to download tx data from them, we end up punishing them leading rank to fall to zero. These nodes could be used for metadata requests taking some load off from nodes that serve unpacked tx data. By separating rank between these two use cases, we should have better working ranking and less failure situation where all or most of the nodes get ranked to zero and we have no usable nodes.
- [ ] Fix how database connection is passed to indexer workers (1md)
Currently database connection is passed to workers by taking the connection from the pool when worker is started. This is pretty wrong, because then the connection is never reset between requests that workers do. Also this causes long living connections that might not be used and eventually timeout and leads worker to die. Instead, worker should get pool passed when initialized and then acquire connection when running a job and at the end, release the connection back to the pool.
- [ ] Mutable pointer resolver route (REMINDER: add IIP spec)
- [ ] PagerDuty integration
COMMENT: I would separate monitoring topics in to a own milestone target. At this point it might be a bit too early to do this.
- [ ] Internal monitoring (e.g. calling upon a 500 status code)

### V6
- [ ] IPFS support
- [ ] Bulk GET support
- [ ] Bundlers downtime contigency plan
- [ ] Full user side monitoring
- [ ] Custom allow lists (OPEN QQS Graphql filter??)

### V7
- [ ] Vector + SigNoz + ClickHouse for request metrics logging
- use OpenTelemetry for sending data
- We want to be able to sell analytics on what the most popular data is

### Unscheduled items
- [ ] Download TX data using unpacked chunks
To increase pool of available miners, download unpacked chunks and unpack a the indexer.
- [ ] Expire tx data cache items in S3 after 30 days from last access
See AWS article for more details about how to implement: https://aws.amazon.com/blogs/architecture/expiring-amazon-s3-objects-based-on-last-accessed-date-to-decrease-costs/
- [ ] When HTTP streaming fails, return HTTP trailer indicating failure
When streaming data fails after initial respose is sent and streaming is started, we should use HTTP trailer headers to indicate that it failed. See more: https://www.rfc-editor.org/rfc/rfc2616#section-14.40
- [ ] Upgrade `apalis` to see if job status is updated correct after worker finishes with it. Current version, `0.4.3`, seems to leave job status to `Pending` even when it cleanly finishes with the job.
- [ ] Store small objects more efficiently (3md)
It's not cost effective to use S3 as lots of PUTs. IMO: we just store all objects <256KiB in the DB as we have PBs of space to use up!)
- [ ] Review and refactor websocket and GraphQL subscription events
- define what to do with tx coming multiple times, deduplicate or not?
- this happens when we get tx from bundler and then later through block processing or when tx is processed as a part of new block after fork
- what are the fields should we report for txs and blocks, we probably should drop some of the data to make those messages smaller and also drop fields that might change after forks or bundle being pushed to Arweave.
- [ ] rebranding `rust-sdk`
Currently `rust-sdk` is published as `bundlr-sdk` that should be rebranded as `irys-sdk`.
- [ ] improvements for selective bundle processing filters
- [ ] query parents and make them available in a tx filter
- [ ] support length of parents in the filters (for checking depth of nesting)
We need to support for filter like:

(
  tx.owner EQ <bundler-1>
  OR tx.owner EQ <bundler-2>
)
OR (
  tx.top_level_parent.owner EQ <bundler-1>
  OR tx.top_level_parent.owner EQ <bundler-2>
  AND (
    NOT tx.owner EQ <blacklisted address>
    AND NOT tx.owner EQ <another blacklisted address>
    AND tx.parents.length LE 5
  )
)
OR (
  block.height LE <height cut-point>
)

### Deferred items (DO NOT ADDRESS THESE TILL WE HAVE A WORKING PRODUCT)
- [ ] gateway capabilities endpoint
See: https://github.com/ArweaveTeam/arweave-standards/blob/master/ans/ANS-101.md
- [ ] arweave-id processing
Support transactions for providing `arweave-id` information as described in: https://github.com/ArweaveTeam/arweave-standards/blob/master/2nd-layer/arweave-id-custom-gateway.md
- [ ] implement clean shutdown of `apalis` workers. This is needed so that we can stop the contrainer for the workers, upgrade the container and then continue processing the stack. Currently we need to stop other containers, wait for last job to be finished and then stop workers, run updates and bring whole system up with upgraded containers. Currently there can be lost jobs if workers die (because docker kills the container) and they hold a job. When restarting, it seems apalis workers do not rerun the job for some reason.
- [ ] create test for `Do-Not-Store` transaction being processed before we process the actual transaction (there are few scenarios where this can happen: when processing historic data or executing fallback checks from bundlers)
- [ ] stop using `ed25519-dalek` crate
This crate is renamed, but we might want to use an alternative one as
the new renamed one is not that actively developed.
- [ ] stop using `dotenv` crate, it's abandoned
`dotenv` seems to be abandoned. Replaced with `dotenvy` after `dotenv` maintainers did not respond to any communication attempts.
- [ ] stop using `web3` create, it's abandoned
This should be replaced with something, probably with `ethers-rs`
- [ ] TX data validation
- [ ] Block validation
- [ ] Redirect on getting data (see how arweave.net response to a `GET /<txid>` request)
- [ ] Can we reconstruct txs from index + data cache
- [ ] Clean FS upon indexing
Currently we have temporary solution that removes all files that are not accessed for the past two days. We need a logic to remove files when bundle is processed, but this gets a bit more complicated when we add nested bundles as we need job dependencies support and apalis does not have it. After implementing support for huge tx data, we probably won't even need this.
- [ ] Bundlr info extractor per tx
- [ ] Fee
- [ ] Currency
- [ ] Revenue
- [ ] Bundle info (tx id)
- [ ] Handle country specific *Do-Not-Store*
Do not store can only be limited to certain country/countries.

## Priorities

- Getting data
- Getting GraphQL working FULLY
- All Bundlr related components working

## Testing

- Compare results between arweave.net and our gateway
- Check if pact.io could be used for defining test suite
This would allow us to drive contract based testing and publish first gateway contract that can be used for testing clients/apps and also for testing gateway implementations.