# Avail performance monitoring bot

For this monitoring bot we will use substrate sidecar api to query the network as making storage calls on substrate is very complicated. Sidecar connects to a node on the network and exposes rest endpoints which gives out json responses. I have set up a sidecar for the current testnet of avail which will be used for testing this tool while it gets developed. Setup of sidecar is outside the scope of this project. I will add the set up instructions of the sidecar in the readme.

Uptime and performance of a validator on avail is measured using 2 metrics:- blocks produced and Imonline heartbeat message. A validator produces blocks in a session or if it does not get a chance to produce a block then it sends a heartbeat tx to the network.

This bot should send alerts to telegram and pagerduty and will need a database influxdb/postgres to store data.

There should be these fields in the config file of the bot

sidecar_url = ""
validator_address = ""
valdiator_name = ""
telegram_api_key = ""
telegram_chat_id = 
pagerduty_api_key = ""
scrape_interval = "600s"

For devloping and testing this bot this can be used as sidecar url: http://168.119.108.59:8080 and use `5E2c4Wj7aeE4ENFGr8RBLz81yQzNyuzF8ZvnyWY9Y6Ps7BMk` this as validator address and name as Vitwit.

An ideal row in the db should look something like this:-

SessionIndex | HeartbeatBlock | BlocksProduced | Scrape1 | Scrape2 | Scrape3 | AlertSent

int | int | int | bool | bool | bool | bool

Current session index can be queried using this endpoint: http://168.119.108.59:8080/pallets/session/storage/currentIndex. Value field of the response gives the current ongoing session. If this value is not already present in the db then it should be pushed. If it is already present then move to the next step.

Each session has it's own heartbeat after block which can be queried using this endpoint: http://168.119.108.59:8080/pallets/imOnline/storage/heartbeatAfter. The `value` field gives the block number after which any validator can send a heartbeat tx. If this height does not already exist for a session then push to db. If it's already present then move to the next step.

Current block height of the network can be queried using this endpoint: http://168.119.108.59:8080/blocks/head/header. `number` field gives you the current block height of the network. If this block height is less than the heartbeat block height for the current session then do nothing and sleep till the next scrape time. If the current block height is greater than the heartbeat block height for the current session then move to the next step.

Blocks produced by a validator for a session can be queried using thi endpoint: http://168.119.108.59:8080/pallets/imOnline/storage/authoredBlocks?keys[]=<session-index>&keys[]=<val-addr>. This is an example of querying the authored blocks of validator `5E2c4Wj7aeE4ENFGr8RBLz81yQzNyuzF8ZvnyWY9Y6Ps7BMk` of session `210`:- http://168.119.108.59:8080/pallets/imOnline/storage/authoredBlocks?keys[]=210&keys[]=5E2c4Wj7aeE4ENFGr8RBLz81yQzNyuzF8ZvnyWY9Y6Ps7BMk. `value` field is the number of blocks produced by the validator. If the value is greater than 0 then push the value in the db then turn all the bool values to true and sleep till the next scrape time. Validator performance for this session has been measured, nothing has to be queried till the next session starts. If the blocks produced value is 0 then push to db and move to the next step.

To check if a validator has sent out a heartbeat or not we first need to find the authority id of the validator. You can get the list of the validators in a session using this endpoint:- http://168.119.108.59:8080/pallets/session/storage/validators. `value` field will be a an array of all the active validators. The index of our validator in the array is the authority id of our validator. Ex:- if our validator is the first object in the array ie. array[0] then auhority id of the validator is 0, if our validator is the tenth object in the array ie. array[9] then the authority id of the validator is 9. To check if the validator sent a heartbeat we can use this endpoint:- http://168.119.108.59:8080/pallets/imOnline/storage/receivedHeartbeats?keys[]=<session-index>&keys[]=<authority-id>. If the `value` is true then turn all the bool values as true and sleep till the next scrape time. Validator performance for this session has been measured, nothing has to be queried till the next session starts. If the `value` is `null` then move to the next step.

Make the Scrape1 value as true and keep all the other values as false and sleep till next scrape time. Once the next scrape starts, as the blocks produced was 0 for the previous scrape, query the blocks produced for the val again. If this time the blocks produced value is greater than 0 then again make everything true and do nothing till next session. If the value is still 0 then check for heartbeat. If heartbeat is present then all bool true and do nothing till next session. If heartbeat is still not present then turn just scrape2 as true. At this point scrape1 and scrape2 are true and scrape3 and alertsent are false. Do the above process again for the third time. If blocks not produced, heartbeat not sent then turn scrape3 true and send out an alert. After sending the alert turn the alert sent to true and do nothing till next scrape time. We are keeping a field for alertsent as we want to send an alert only per session once if the validator is not performing to avoid overwhelming the user.