# Indexer missing events report

The teritori indexer has various chances to miss events since it's inception, here's the explaination of what it does, what happens and what are the ideas for fixing it

## The goal

The teritori indexer is an event sourced system that sequentially reads all transactions of the teritori chain from genesis in order to populate a database with aggregated data that can't be properly queried from the chain and smart contracts directly

## The naive flow

- 1) Choose a `chunk_size` (max number of blocks queried and processed in one database transaction)
- 2) Read or initialize a field in the database that keeps track of the height of the next block to process (`height_cursor`)
- 3) Declare `chunk_end = height_cursor + chunk_size`
- 3) Query the `latest_height` from a teritori node, if this height is lower than `chunk_end` set `chunk_end = latest_height + 1`
- 4) Start a database transaction
- 5) Use the tendermint `tx_search` rpc to query a page of transactions contained in blocks in the range `[height_cursor, chunk_end[`. See https://docs.tendermint.com/v0.34/rpc/#/Info/tx_search and https://docs.tendermint.com/v0.34/app-dev/indexing-transactions.html
- 6) Process transactions in the page
- 7) Go back to 5) until there is no transaction left in the chunk
- 8) Upate the height cursor
- 8) Commit the database transaction
- 10) Go back to 3) until killed

We use tendermint `tx_search` because cosmos' `GetTxsEvent` (`tx_search` wrapper) is broken in our version of cosmos-sdk. See https://github.com/cosmos/cosmos-sdk/issues/11538

## Problems

### Too much txs

If we were to query all transactions in the chain the indexer would be too slow to replay the whole chain

Fortunately, the `tx_search` rpc allows to filter on events so we can request only cosmwasm related transactions and thus greatly reduce the number of transaction to process

### Limited queries

While the `tx_search` call allows some filterting of transaction, it's query language is very poor and does not allow to set multiple filter, so we can't ask for cosmwasm AND aidrop messages, thus we could not implemented the `Claim aidrop` quest

### Load balanced nodes desync

With the naive implementation, if the teritori endpoint is load balanced, nodes behind the load balancer might not all be at the same height
Due to this, there is a race condition between the height query and the tx_search queries when:
- the indexer queries a first node that is ahead for the latest_height
- then queries another node for the txs until that height but this node doesn't have these txs yet and will return an empty chunk
- the height_cursor will be updated to the latest_height and txs will be lost

We fixed this by using a websocket client that ensures we connect to the same node for all queries. A side benefit of using a websocket client is that we reuse the connection

### Tendermint tx indexer race condition

There is a race condition if we query the height and txs between the moment that the node has processed a block and the moment that the tendermint tx indexer has finished to index transactions. See https://docs.tendermint.com/v0.34/app-dev/indexing-transactions.html for details about the tendermint tx indexer.

We did not found a proper way to fix this for now, so we decided to wait one block until we query transactions to decrease the chances to fall in the race condition

## Ideas

### Use the psql tx indexer

Tendermint provides another implementation of the tx indexer that populate a postgresql database so we can do advanced SQL queries instead of using `tx_search`

This would be the optimal solution as it would grant us a lot more freedom on how we query transactions (and thus implement the `Claim airdrop` quest for example)

We started a node that uses the psql indexer but it's very slow and is estimated to finish replaying the chain in more than a month

### Patch tendermint to be able to query the tx indexer height

This would fix the tendermint tx indexer race condition but not the limited filter problem

### Keep a tx hash cursor

We could
- keep the height and hash of the last transaction processed by the teritori indexer
- query transactions from this height
- ignore transactions until the one after the last processed transaction hash

This would fix the tendermint tx indexer race condition but not the limited filter problem