# Doppelganger Detection Testing Results

## Setup
Two AWS instances both running a Besu/Teku setup have been used to conduct the doppelganger detection feature tests
1. `dev-teku-besu-ohio-goerli-dev-dg`
2. `dev-teku-besu-ohio-goerli-dev-dg-2`

The tests have been performed on the Goerli testnet

## DG detection at validator stratup
### 1. Starting the nodes
Besu and Teku have been started in both instances and finished syncing after few days.
The BN/VC have been started in a single process and without loading any validator keys for both Teku instances (`dev-teku-besu-ohio-goerli-dev-dg` and `dev-teku-besu-ohio-goerli-dev-dg-2`).
The purpose of this step is to give the nodes time to sync.

### 2. Running the first Teku node on `dev-teku-besu-ohio-goerli-dev-dg`
A validator key have been imported into the first Besu/Teku node (`dev-teku-besu-ohio-goerli-dev-dg` instance) after making the 32 eth deposit.
Both BN and VC instances are still running in a single process.
The validator is up and running and has the index 397892 assigned to it: [Validator 397892](https://goerli.beaconcha.in/validator/397892)

### 3. Running the second Teku node `dev-teku-besu-ohio-goerli-dev-dg-2`
The second BN/VC node have been started on the `dev-teku-besu-ohio-goerli-dev-dg-2` with the option `Xbeacon-liveness-tracking-enabled` in order to start caching the active validators. No keys have been imported to this node at this stage.
The logs show that the [Validator 397892](https://goerli.beaconcha.in/validator/397892) running on the `dev-teku-besu-ohio-goerli-dev-dg` have been discovered and its index cached:

{"@timestamp":"2022-12-16T11:43:08,580","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142403","throwable":""}
{"@timestamp":"2022-12-16T11:48:56,363","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142404","throwable":""}
{"@timestamp":"2022-12-16T11:48:56,492","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142404","throwable":""}
{"@timestamp":"2022-12-16T11:52:20,487","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142405","throwable":""}
{"@timestamp":"2022-12-16T11:52:20,687","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142405","throwable":""}
{"@timestamp":"2022-12-16T11:52:20,750","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142405","throwable":""}
{"@timestamp":"2022-12-16T11:52:21,659","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142405","throwable":""}
{"@timestamp":"2022-12-16T11:52:22,568","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142405","throwable":""}
{"@timestamp":"2022-12-16T11:58:44,881","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142406","throwable":""}
{"@timestamp":"2022-12-16T11:58:44,898","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142406","throwable":""}
{"@timestamp":"2022-12-16T11:58:46,515","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142406","throwable":""}
{"@timestamp":"2022-12-16T12:09:56,667","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142407","throwable":""}
{"@timestamp":"2022-12-16T12:09:56,797","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142407","throwable":""}
{"@timestamp":"2022-12-16T12:09:56,964","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142407","throwable":""}
{"@timestamp":"2022-12-16T12:09:57,019","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142407","throwable":""}
{"@timestamp":"2022-12-16T12:16:20,202","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142408","throwable":""}
{"@timestamp":"2022-12-16T12:16:21,398","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142408","throwable":""}
{"@timestamp":"2022-12-16T12:16:23,495","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142408","throwable":""}
{"@timestamp":"2022-12-16T12:20:32,355","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142409","throwable":""}
{"@timestamp":"2022-12-16T12:25:44,464","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142410","throwable":""}
{"@timestamp":"2022-12-16T12:25:45,525","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142410","throwable":""}
{"@timestamp":"2022-12-16T12:34:08,268","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142411","throwable":""}
{"@timestamp":"2022-12-16T12:34:10,288","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142411","throwable":""}
{"@timestamp":"2022-12-16T12:38:32,568","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142412","throwable":""}
{"@timestamp":"2022-12-16T12:38:33,598","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142412","throwable":""}
{"@timestamp":"2022-12-16T12:38:52,082","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142412","throwable":""}
{"@timestamp":"2022-12-16T12:49:20,690","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142413","throwable":""}
{"@timestamp":"2022-12-16T12:55:20,381","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142414","throwable":""}
{"@timestamp":"2022-12-16T12:57:08,517","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142415","throwable":""}
{"@timestamp":"2022-12-16T13:06:44,217","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142416","throwable":""}
{"@timestamp":"2022-12-16T13:14:32,197","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142417","throwable":""}
{"@timestamp":"2022-12-16T13:21:32,916","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142418","throwable":""}
{"@timestamp":"2022-12-16T13:26:20,076","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142419","throwable":""}
{"@timestamp":"2022-12-16T13:30:56,272","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142420","throwable":""}
{"@timestamp":"2022-12-16T13:39:20,588","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142421","throwable":""}
{"@timestamp":"2022-12-16T13:39:22,659","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142421","throwable":""}
{"@timestamp":"2022-12-16T13:45:08,054","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142422","throwable":""}
{"@timestamp":"2022-12-16T13:45:09,556","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142422","throwable":""}
{"@timestamp":"2022-12-16T13:45:09,755","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142422","throwable":""}
{"@timestamp":"2022-12-16T13:48:08,291","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142423","throwable":""}
{"@timestamp":"2022-12-16T13:48:08,920","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142423","throwable":""}
{"@timestamp":"2022-12-16T13:48:10,748","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142423","throwable":""}

**Observation:** The [Validator 397892](https://goerli.beaconcha.in/validator/397892) is seen 2 to 4 times in each epoch which seems to be a good result for an efficient doppelganger detection.

### 4. Importing the same key to the second Teku node `dev-teku-besu-ohio-goerli-dev-dg-2` (Copying the key and loading the validator at startup)
The same private key have been copied to the second Teku node. The BN/VC have been restarted in order to load the key at startup. The following options have been used to enable the validators indices caching and the DG detection feature:
- `Xbeacon-liveness-tracking-enabled` to enable active validators caching
- `Xdoppelganger-detection-enabled` to enable the DG detection feature

PS: Since the node (hence the BN) have restarted in order to load the imported key. The validators cache is erased (In memory cache).

## Results

**On 8 attempts, the validator doppelganger have been detected 7 times**.
**The detection time ranges from 53 seconds to 10:01 minutes**.

For all the 8 attempts, the BN/VC were restarted each time in order to load the DG key and perform a DG detection. This is the worst case scenario since the BN will start caching the active validators at the same time as the DG detection process starts (Would be more efficient to give the BN few minutes to scan the network first).

1. Started at `15:58:42`, DG detected at `16:01:30`: **Took 2:06 minutes**

{"@timestamp":"2022-12-16T15:58:42,380","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:58:54,392","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142442, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:59:06,405","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:59:18,406","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:59:30,413","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:59:42,418","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T15:59:54,419","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:00:06,420","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:00:18,425","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:00:30,426","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:00:42,428","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:00:54,429","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:01:06,430","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:01:18,431","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:01:30,432","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:01:30,433","level":"FATAL","thread":"validator-async-3","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:01:30,434","level":"FATAL","thread":"validator-async-3","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:01:30,435","level":"INFO","thread":"validator-async-3","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

2. Started at `16:01:54`, DG detected at `16:02:47`: **Took 53 seconds **

{"@timestamp":"2022-12-16T16:01:54,781","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:02:06,812","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:02:23,728","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:02:35,744","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:02:47,744","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:02:47,745","level":"FATAL","thread":"validator-async-0","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:02:47,746","level":"FATAL","thread":"validator-async-0","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:02:47,746","level":"INFO","thread":"validator-async-0","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

3. Started at `16:03:21`, DG detected at `16:10:47`: **Took 6:26 minutes **

{"@timestamp":"2022-12-16T16:03:21,343","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:03:33,394","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142443, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:03:47,889","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:03:59,890","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:04:11,893","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:04:23,894","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:04:35,895","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:04:47,895","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:04:59,896","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:05:11,897","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:05:23,897","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:05:35,898","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:05:47,899","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:05:59,899","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:06:11,900","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:06:23,901","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:06:35,902","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:06:47,903","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:06:59,904","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:07:11,904","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:07:23,905","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:07:35,906","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:07:47,906","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:07:59,907","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:08:11,908","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:08:23,909","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:08:35,910","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:08:47,910","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:08:59,911","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:09:11,912","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:09:23,912","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:09:35,913","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:09:47,914","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:09:59,914","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:10:11,915","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:10:23,915","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:10:35,916","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:10:47,917","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:10:47,917","level":"FATAL","thread":"validator-async-4","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:10:47,918","level":"FATAL","thread":"validator-async-4","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:10:47,919","level":"INFO","thread":"validator-async-4","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

4. Started at `16:11:23`, DG detected at `16:20:11`: **Took 8:48 minutes**

{"@timestamp":"2022-12-16T16:11:23,867","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:11:35,879","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142444, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:11:47,891","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:11:59,892","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:12:11,893","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:12:23,894","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:12:35,895","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:12:47,896","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:12:59,897","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:13:11,898","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:13:23,899","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:13:35,900","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:13:47,900","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:13:59,901","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:14:11,902","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:14:23,903","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142445, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:14:35,904","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:14:47,905","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:14:59,906","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:15:11,906","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:15:23,907","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:15:35,908","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:15:47,908","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:15:59,909","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:16:11,918","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:16:23,918","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:16:35,919","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:16:47,920","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:16:59,921","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:17:11,925","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:17:23,926","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:17:35,927","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:17:47,928","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:17:59,928","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:18:11,932","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:18:23,933","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:18:35,933","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:18:47,934","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:18:59,935","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:19:11,936","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:19:23,937","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:19:35,937","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:19:47,938","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:19:59,939","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:20:11,939","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:20:11,941","level":"FATAL","thread":"validator-async-1","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:20:11,942","level":"FATAL","thread":"validator-async-1","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:20:11,942","level":"INFO","thread":"validator-async-1","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

5. Started at `16:24:40`, DG detected at `16:23:58`: **Took 3:07 minutes**

{"@timestamp":"2022-12-16T16:20:51,080","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:21:03,114","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:21:22,040","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:21:34,041","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:21:46,045","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:21:58,046","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:22:10,047","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:22:22,048","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:22:34,049","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:22:46,055","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:22:58,163","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:10,267","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:22,274","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:34,274","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:46,275","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:58,276","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:23:58,306","level":"FATAL","thread":"validator-async-3","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:23:58,306","level":"FATAL","thread":"validator-async-3","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:23:58,307","level":"INFO","thread":"validator-async-3","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

5. Started at `16:20:51`, DG detected at `16:31:52`: **Took 10:01 minutes**

{"@timestamp":"2022-12-16T16:24:40,585","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:24:52,594","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142446, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:25:04,618","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:25:16,625","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:25:28,629","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:25:40,632","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:25:52,633","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:26:04,634","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:26:16,635","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:26:28,636","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:26:40,637","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:26:52,638","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:27:04,639","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:27:16,641","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:27:28,642","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:27:40,643","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:27:52,644","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:28:04,645","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:28:16,646","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:28:28,647","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:28:40,647","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:28:52,648","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:29:04,649","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:29:16,649","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:29:28,650","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:29:40,651","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:29:52,651","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:30:04,652","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:30:16,652","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:30:28,653","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:30:40,654","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:30:52,654","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:04,655","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:16,656","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:28,657","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:40,657","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:52,658","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:31:52,659","level":"FATAL","thread":"validator-async-1","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:31:52,660","level":"FATAL","thread":"validator-async-1","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:31:52,660","level":"INFO","thread":"validator-async-1","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

6. Started at `16:32:31`, DG detected at `16:36:55`: **Took 4:24 minutes**

{"@timestamp":"2022-12-16T16:32:31,562","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:32:43,572","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142447, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:32:55,634","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:33:07,636","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:33:19,637","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:33:31,638","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:33:43,639","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:33:55,639","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:34:07,640","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:34:19,641","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:34:31,642","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:34:43,642","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:34:55,643","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:35:07,644","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:35:19,645","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:35:31,645","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:35:43,646","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:35:55,647","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:07,648","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:19,649","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:31,650","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:43,650","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:55,651","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:36:55,652","level":"FATAL","thread":"validator-async-0","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:36:55,652","level":"FATAL","thread":"validator-async-0","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:36:55,653","level":"INFO","thread":"validator-async-0","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

7. Started at `16:37:34`, DG detected at `16:40:34`: **Took 3:00 minutes**

{"@timestamp":"2022-12-16T16:37:34,457","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:37:46,485","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142448, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:37:58,533","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:38:10,538","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:38:22,539","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:38:34,539","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:38:46,540","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:38:58,541","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:39:10,542","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:39:22,564","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:39:34,565","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:39:46,566","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:39:58,567","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142449, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:40:10,567","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142450, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:40:22,568","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142450, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:40:34,569","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142450, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:40:34,570","level":"FATAL","thread":"validator-async-4","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2022-12-16T16:40:34,571","level":"FATAL","thread":"validator-async-4","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2022-12-16T16:40:34,571","level":"INFO","thread":"validator-async-4","class":"DoppelgangerDetectionShutDown","message":"Validator doppelganger detected. Shutting down...","throwable":""}

8. **Failed attempt**, started at 16:48:23 finished at 16:59:12. **Took 10:49 minutes** and didn't detect the doppelganger.

{"@timestamp":"2022-12-16T16:48:23,968","level":"INFO","thread":"beaconchain-async-1","class":"teku-status-log","message":"Starting doppelganger detection for public keys: a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:48:35,980","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142450, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:48:47,998","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:48:59,999","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:49:12,000","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:49:24,002","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:49:36,004","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:49:48,004","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:50:00,005","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:50:12,006","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:50:24,007","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:50:36,008","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:50:48,009","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:51:00,011","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:51:12,012","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:51:24,013","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:51:36,014","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:51:48,016","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:52:00,017","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:52:12,018","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:52:24,024","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:52:36,025","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142451, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:52:48,037","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:53:00,038","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:53:12,039","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:53:24,040","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:53:36,040","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:53:48,041","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:54:00,042","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:54:12,043","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:54:24,043","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:54:36,044","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:54:48,045","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:55:00,046","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:55:12,047","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:55:24,047","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:55:36,048","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:55:48,049","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:56:00,049","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:56:12,050","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:56:24,051","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:56:36,051","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:56:48,052","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:57:00,053","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:57:12,053","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:57:24,054","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:57:36,055","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:57:48,055","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:58:00,056","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:58:12,057","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:58:24,057","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:58:36,058","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:58:48,059","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:59:00,059","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Performing doppelganger check. Epoch 142452, Public keys a2fbe8a","throwable":""}
{"@timestamp":"2022-12-16T16:59:12,060","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"No validators doppelganger detected after 2 epochs. Stopping doppelganger detection for public keys a2fbe8a.","throwable":""}

### Failed attempt analysis
During the 8th attempt, the DG wasn't detected even though it was seen by the BN and its index cached during the same epochs (50, 51, 52). The DG checks have also been performed after the BN have cached the DG index.

{"@timestamp":"2022-12-16T16:40:32,171","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142450","throwable":""}
{"@timestamp":"2022-12-16T16:40:32,412","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142450","throwable":""}
{"@timestamp":"2022-12-16T16:47:44,557","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142451","throwable":""}
{"@timestamp":"2022-12-16T16:59:09,359","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142452","throwable":""}
{"@timestamp":"2022-12-16T16:59:26,853","level":"TRACE","thread":"ActiveValidatorChannel-0","class":"ActiveValidatorCache","message":"Touch validator 397892 at epoch 142452","throwable":""}

Possible causes:
- VC/BC communication issue
- Validators liveness API issue (the active validators cache not checking the validator index correctly)
- Doppelganger Detector not interpreting the validators liveness reponse correctly (Many object collections mappings/filtering are performed)

Update:
After tuning the DG detection (Checking last and current epoch at the first run, debugging the active validator cache, leaving some time to the running validator so it could be seen in the network...) the issue seems to be solved.

## DG detection via the key manager API
### 1. Starting the nodes
Besu and Teku have been started in both instances and are already in sync.
The first node `dev-teku-besu-ohio-goerli-dev-dg` is running two validators since few days.
The second node `dev-teku-besu-ohio-goerli-dev-dg-2` is running but without any validators.
Tha validator rest api is enabled.

### 2. Loading the same keys
A validator key (already used in the first instace `dev-teku-besu-ohio-goerli-dev-dg`) is imported through the key manager API.

### Results
The DG is detected within the first round, almost instantly

1. Started at `08:16:19`, DG detected at `08:16:31`: **Took 12 seconds**

{"@timestamp":"2023-01-11T08:16:19,279","level":"INFO","thread":"qtp48042118-184","class":"teku-status-log","message":"Starting doppelganger detection for public keys: b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:16:31,281","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Performing doppelganger check. Epoch 148220, Public keys b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:16:31,299","level":"FATAL","thread":"validator-async-0","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2023-01-11T08:16:31,299","level":"FATAL","thread":"validator-async-0","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2023-01-11T08:16:43,300","level":"INFO","thread":"validator-async-4","class":"teku-status-log","message":"Doppelganger detection check finished. Stopping doppelganger detection for public keys b27ab22.","throwable":""}
{"@timestamp":"2023-01-11T08:16:43,301","level":"FATAL","thread":"validator-async-4","class":"teku-status-log","message":"Detected 1 validators doppelganger

2. Started at `08:17:05`, DG detected at `08:17:17`: **Took 12 seconds**

{"@timestamp":"2023-01-11T08:17:05,176","level":"INFO","thread":"qtp48042118-119","class":"teku-status-log","message":"Starting doppelganger detection for public keys: b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:17:17,177","level":"INFO","thread":"validator-async-0","class":"teku-status-log","message":"Performing doppelganger check. Epoch 148220, Public keys b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:17:17,178","level":"FATAL","thread":"validator-async-3","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2023-01-11T08:17:17,178","level":"FATAL","thread":"validator-async-3","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2023-01-11T08:17:29,179","level":"INFO","thread":"validator-async-1","class":"teku-status-log","message":"Doppelganger detection check finished. Stopping doppelganger detection for public keys b27ab22.","throwable":""}
{"@timestamp":"2023-01-11T08:17:29,179","level":"FATAL","thread":"validator-async-1","class":"teku-status-log","message":"Detected 1 validators doppelganger

3. Started at `08:25:13`, DG detected at `08:25:25`: **Took 12 seconds**

{"@timestamp":"2023-01-11T08:25:13,147","level":"INFO","thread":"qtp48042118-189","class":"teku-status-log","message":"Starting doppelganger detection for public keys: b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:25:25,148","level":"INFO","thread":"validator-async-3","class":"teku-status-log","message":"Performing doppelganger check. Epoch 148221, Public keys b27ab22","throwable":""}
{"@timestamp":"2023-01-11T08:25:25,149","level":"FATAL","thread":"validator-async-4","class":"DoppelgangerDetector","message":"Validator doppelganger detected...","throwable":""}
{"@timestamp":"2023-01-11T08:25:25,149","level":"FATAL","thread":"validator-async-4","class":"teku-status-log","message":"Detected 1 validators doppelganger: 
{"@timestamp":"2023-01-11T08:25:37,149","level":"INFO","thread":"validator-async-2","class":"teku-status-log","message":"Doppelganger detection check finished. Stopping doppelganger detection for public keys b27ab22.","throwable":""}
{"@timestamp":"2023-01-11T08:25:37,150","level":"FATAL","thread":"validator-async-2","class":"teku-status-log","message":"Detected 1 validators doppelganger