EPF Update #4

TLDR / Main Updates

• Postgres DB backend store is integrated and eveything is saved into a DB, analysis stats are also being saved into a table, but this patch is still partially a WIP
• Refactored parts of the relay monitor "collector" to support re-tries. This can be customized to specify a delay and max number of retries. The idea is to re-use this in any code path that might benefit from requesting data per-slot

Dev Update for 01/16/2023

This update is shorter, as last week led to a few WIP patches that are still in progress and will be ready by next update: rewritten bid analysis functions (rules of "validity" of bid received via getHeader()), completed flow of pushing all analysis + faults into a DB, and report generation from the DB (and even turning this into a stream).

I also got started playing around with some initial work on refactoring handling validator registrations by the relay monitor, but I'm focusing on the existing WIPs first.

Retry logic

I rewrote pieces of the "collector" to support retries and kick-off retrying to collect the bid from the relay if the initial requests fails or if the relay claims that there is no bid for that slot. This work is done and has been committed into the repo. The individual processes in the relay monitor work together in parallel and utilize a "clock" in order to keep track of slots and make calls to fetch data per-slot when the time comes. (all data is pretty much in slot granularity). Of course unforceen errors or discrepencies with this clock might case some requests to fail, hence the work to easy support re-tries.

The first part where this has been currently added is the GetBid function that is a wrapper around the getHeader() endpoint that a Relay exposes (the endpoint itself is actually part of the Builder API – link to getHeader in Builder API and the Relay implements/supports this). Now if the GetBid code path fails for whatever reason, the relay-monitor will retry for a few times, waiting a few seconds in-between (configurable). At the same time, the new getHeader() call for the new slot can be kicked off too. So the relay monitor can be retrying in the background while the "main" collector tries to fetch the bid (if any) for the next slot. Once the bid is found via the initial request or any of the re-tries, it follows the same path and is saved to the DB.

Work for upcoming week

I've already briefly mentioned this, but this week the WIP patches that are getting wrapped up prior to next week's updates are

1. Re-write of bid analysis functions + more comments + logging
2. Support for queries of bid analysis, e.g. "count faults from slot n to n + 10" or just "how many bids have been analyzed / inserted in the last hour"
3. Utitlizing above, generate report by reading from the DB directly