# Rain Arb bot responsibilites

These are the operational responsibilites for overseeing the bot performance, this is different from building and maintaining the bot software.

## Provisioning

The arb bot consists of several components:

- The arb bot client that submits transactions
- An indexer that makes orders discoverable
- A full node that exposes chain state to the bot

All three are in scope for bot operations.

We DO NOT operate bare metal instances of any of these. All services are either vendored (e.g. some paid RPC) or dockerized.

Further, all "our" infrastructure is open sourced, explicitly with the intent that others can run their own instances in a permissionless manner.

Therefore, the scope of provisioning "our" instances is limited to what can be simply and efficiently reproduced via. containerization (i.e. docker compose).

### Bot instances

- Provisioning new bots
- Updating bots to the latest versions of the software
- Making sure changes are staged, in case of regression or unpredictable issues with new versions
- Ensuring bot wallets have enough gas
- Sweeping hot wallets for accrued tokens to cold wallets

### Subgraphs

- Goldsky subgraph provisioning for supported subgraphs
- Local nodes for subgraphs Goldsky does not support
- Subgraph data size reporting
- Subgraph availability reporting
- Backup / redunancy planning for subgraphs e.g. hosted graph service

### Nodes

@todo

## Monitoring

The golden rule of monitoring is that we know about issues before our users do. The first time we discover an infrastructure issue cannot be because someone else reported it to us.

### Metrics

We need simple and effective metrics that encompass as many possible failure modes as possible. This forms a definition of "uptime" that we use to objectively define "better" and "worse".

- how many blocks elapsed between opportunity and execution?
- how often does our execution succeed?

The first tells us whether we are effectively clearing user's strategies. The best possible outcome is that we clear a transaction in the first block that it is clearable, a score of `0`. The worst possible outcome is complete failure to clear, a score of "infinity".

Many strategies are time sensitive, so delays literally equate to funds loss. Consider a dutch auction that is reducing its price every block. Failing to clear the auction when valid liquidity exists is literally draining the owner's vault.

The second metric tells us how well we are submitting transactions in the face of flakey RPCs, MEV front running, etc. High failure rates can cause opp/exec delays, and even cost gas for the bot due to reverts.

Once we are quantifying the above with real data, we can establish baselines, SLA expectations and iteratively improve infrastructure against objective measurements.

### Alerts

Once we have established baseline expectations of our infrastructure, any exceptions that fall outside expectations need to alert us.

Alerts need to be timely, high signal/noise and actionable.

Setting up a TG channel that spams all activity is not helpful as it quickly becomes a firehose that nobody pays attention to.

Alerts need to have very low false positive and false positive rates, so that if one is ever raised we immediately and reliably triage it.

Examples of potential alerts:

- Bot down alert over telegram and email
- Subgraph down alert over telegram and email
- Bot price alert where account is overrunning
- Subgraph price alert where account is overrunning
- Bot gas balance alert

### Dashboards

Creating and maintaining dashboards for
- Network health
- Orders cleared e.g. number and volume of transactions by time period, network, token
- Gas balances of all bot wallets

## Forensics

Once an issue is discovered by our monitoring systems we need to be able to reliably reproduce it.

Reproduction is always the first step towards diagnosing and solving any issue. Failure to reproduce means failure to solve.

As we work with blockchain, we have a super power in the form of fork testing. As long as we log every attempted interaction with the chain, we can always reproduce the chain state in that moment.

Offchain reproductions are more difficult, with perhaps more subjective and onerous logging requirements. Therefore, as much as possible should be lifted onchain for all bot operations. Where this is not possible, additional logs will need to be included for future forensics.

Log retention/rotation, scrubbing, efficiency, storage policy, etc. all fall under forensics responsibilities.

Example forensics tasks include:

- Make sure logs are capturing right information adjusting as required
- Reduce log size as much as possible without compromising accurate information
- Monitor logs to performance issues
- Manage hyperdx account and payment plan
- Ensure logs contain enough information to investigate any potential issues that oculd occur

## Triage

🚨 **Issues related to our subgraph or bot network could cause real funds loss for our users, therefore they take priority over all other work** 🚨

Because of this, even if an issue is merely _suspected_, it must be taken seriously and steps taken to prove or disprove the hypothesis.

If there is a suspected issue that cannot be resolved by inspecting logs, a fork test will be required.

Potential issues could be:
- An order that should have cleared because the market appeared to allow for it
- Reverts
- Orders not being picked up at all
- Vault balances disappearing quickly

### Scoring and prioritising

We need an objective way to score issues as they arise, based on high level criteria.

An example is the CVSS https://en.wikipedia.org/wiki/Common_Vulnerability_Scoring_System system for security vulnerabilities which has been widely applied to many IT systems.

The CVSS isn't exactly what we need for this infrastructure, but we can adapt it relatively easily to suit our needs.

All issues need to be immediately documented in our project management system (e.g. on kanban), scored and prioritised according to the score.

All issues that hit a high/critical score threshold MUST be treated as "tools down" priority and take precedence over other responsibilities such as BAU development work.