# Attestation Inclusion Tracking

[sheet]: https://docs.google.com/spreadsheets/d/1SNFf4LsDOK91SWuQZm9DYBoX9JNQNMKHw66Rv0l5EGo/edit#gid=553688981

## Introduction

This document describes a method for tracking the performance of Eth2 validators. It tracks the following parameters for each validator:

- `attestation_hits`: The validator had an attestation included on-chain.
- `attestation_misses`: There was no attestation included on-chain for an epoch.
- `head_attestation_hits`: The validator had an attestation included on-chain with a *correct* `beacon_block_root`.
- `head_attestation_misses`: The validator had an attestation included on-chain with an *incorrect* `beacon_block_root`.
- `target_attestation_hits`: The validator had an attestation included on-chain with a *correct* `target.root`.
- `target_attestation_misses`: The validator had an attestation included on-chain with an *incorrect* `target.root`.
- `delay_avg`: The average of all the attestation delays where an attestation was included on-chain.

It collects CSV data using the `lcli` tool, then displays that data in Google Sheets.

There's an example spreadsheet where you can view this data for some random validators. You can also copy the spreadsheet, add your own validators and compare them against the global average:

- [[PUBLIC] Validator Attestation Inclusion Peformance: Epochs 27112-32300][sheet]

## Example Time Span

We provide the tools to run this method across any span of epochs, but we provide an [example spreadsheet][sheet] across the following range:

- Start epoch: `27112`
- 2021-04-01 00:00:00 UTC
- Start of Q2.
- End epoch: `32300`
- 2021-04-24 UTC (not sure exact time)
- A couple of epochs prior to the mainnet block production issues (started epoch 23202).

This example timespan starts immediately *after* a prior Q1 2021 validator rewards survey by Ben Edgington and stops just before there were issues with block production on mainnet. This timespan feels recent, unstudied and represents typical operating conditions of the chain.

## Method for Generating Raw Data

### Method: Rust

https://github.com/paulhauner/lighthouse/blob/etl/lcli/src/etl/validator_performance.rs

### Method: English (with some Rust 😉)

Define the `ValidatorPerformance` struct:

struct ValidatorPerformance {
    /// The validator had an attestation included on-chain.
    pub attestation_hits: usize,
    /// Inverse of `attestation_hits`.
    pub attestation_misses: usize,
    /// The validator had an attestation included on-chain which matched the "head" vote.
    pub head_attestation_hits: usize,
    /// Inverse of `head_attestation_hits`.
    pub head_attestation_misses: usize,
    /// The validator had an attestation included on-chain which matched the "target" vote.
    pub target_attestation_hits: usize,
    /// Inverse of `target_attestation_hits`.
    pub target_attestation_misses: usize,
    /// The validator achieved a `head_attestation_hits` point when the first slot of the epoch was
    /// a skip-slot.
    ///
    /// This is generally *not useful*, it was used to try and debug late blocks on mainnet.
    pub first_slot_head_attester_when_first_slot_empty: usize,
    /// The validator achieved a `head_attestation_hits` point when the first slot of the epoch was
    /// *not* a skip-slot.
    ///
    /// This is generally *not useful*, it was used to try and debug late blocks on mainnet.
    pub first_slot_head_attester_when_first_slot_not_empty: usize,
    /// A map of `inclusion_distance -> count`, indicating how many times the validator achieved
    /// each inclusion distance.
    pub delays: HashMap<u64, u64>,

}

The, allocate a list of `[ValidatorPerformance; num_validators]`, where all values in `ValidatorPerformance` are either `usize::zero` or `HashMap::empty`.

Then,

1. Collect the all blocks between the start/end epoch.
2. Load the pre-state of the first (lowest-slot) block.
3. Start sequentially applying each block to the state.

Each time an epoch boundary is reached, use [`ValidatorStatus`](https://github.com/sigp/lighthouse/blob/3a24ca5f14c6e6d6612fd43eca82aa0c1e6aba16/consensus/state_processing/src/per_epoch_processing/validator_statuses.rs#L50-L82) to detemine the participation from the *previous* epoch (i.e., the epoch for which we have a complete set of attestations) and increment any relevant `ValidatorPerformance` value for each validator.

Once we have finished applying the blocks to the state, produce a CSV which is a 1:1 representation of the `[ValidatorPerformance; num_validators]` list *with one exception*: we produce a `delay_avg` which is simply the average of all attestation delays.

## Generating the CSV and Importing to Google Sheets

1. Install Lighthouse.
2. Run a Lighthouse node until it's synced.
3. After the Lighthouse node syncs, terminate the process so we can read from its database.
1. Clone [this branch](https://github.com/paulhauner/lighthouse/tree/etl) and go into the `lcli` directory.
4. Run `cargo run --release etl-validator-performance --output ./valperf.csv --start-epoch 27112 --end-epoch 32300`
5. Go to the "datasource" tab of the spreadsheet.
6. File -> Import -> Upload -> upload `valperf.csv`
7. Set "Import Location" to "Replace Current Sheet".
8. Import Data.

Sheets will take a while to load it, your browser might complain.

## Notes

- I should modify this to use the standard HTTP API instead of the Lighthouse DB.