# Eth2 Validator Key management ## Description One important part of the Ethereum and whole blockchain ecosystem is client diversity. As we have explained before, in order to maintain a safe and robust distributed system, we must ensure the network is not only spread evenly accross world-wide locations, but also software spread, so as to avoid a malicious attacker taking control of the whole ecosystem after finding a software bug. As of this, we have performed a study on how to switch Eth2 validators from one CL client to another, so as to show to the community how the process looks and encourage users to switch between the most used clients to the less used ones. Together with our last experiment, we aim to demonstrate that the 6 main Eth2 CL clients are suitable to run validators and they are all interoperable among them. ## Hardware / Environment For this experiment we have used a Raspberry PI 4b with 8GB RAM, connected via wired Internet connection. The study was performed in the Kiln network with 250 validators. As storage, we have used a 256 GB class A Samsung microSD card and a 248 GB SSD. ### Versions ## Methodology The approach to perform the study: 1. Sync the client 2. Import 250 validators 3. Run for some time, let them attest accordingly 4. Stop the client, export the slashing protection db 5. Sync the next client to try 6. Import the same 250 validators 7. Import the recently exported slashing protection db from the previous client 8. Back to point 2 As we have used a Raspberry PI, we have followed this guide to deploy the clients: https://ethereum-on-arm-documentation.readthedocs.io/en/latest/kiln/kiln-testnet.html# As the EL client, we have always used Geth. [Lodestar guide](https://hackmd.io/@philknows/By5qahdZc) [EthereumOnArm guide: Prysm, Lighthouse, Teku and Nimbus](https://ethereum-on-arm-documentation.readthedocs.io/en/latest/kiln/kiln-testnet.html#) [Automated Launch custom script](https://github.com/migalabs/eth2-client-analyzer/tree/feature/validator_management/validator_management) [Postman requests](https://web.postman.co/workspace/My-Workspace~888f987a-8e6c-4be7-886a-eb9031148c67/request/5618734-14680f42-dffc-4d58-a67b-85a55e0562ae) Under the ethereum user home directory we have created one folder for each of the clients we will be testing out: ``` /home/ethereum/<client>_logs ``` ### Key generation In order to generate the validator keys we have used [eth2-val-tools](https://github.com/protolambda/eth2-val-tools.git), using the mnemonics as a source, but this can also be done through the [Staking Deposit Cli](https://github.com/ethereum/staking-deposit-cli) ``` eth2-val-tools keystores --source-mnemonic="<mnemonics>" --source-min=0 --source-max=250 ``` The resulting folder will be the source for all clients when importing the validators. ### First approach As part of testing and understanding the process, we have done a first run to learn the best possible way of switching Eth2 validators between clients, checking if there was any bug or error we would not understand and, if so, asking the corresponding teams to solve it accordingly. #### Prysm The first client we have tried out is Prysm. After syncing and running the validators, we have stopped the client and exported the slashing protection db. In order to import the validators: ``` for d in /home/ethereum/kiln_validators/assigned_data_0_249/secrets/*; do final_d=$(basename $d) validator-kl accounts import --keys-dir=/home/ethereum/kiln_validators/assigned_data_0_249/keys/$final_d --account-password-file=$d --wallet-dir /home/ethereum/.pry-geth/kiln/testnet-pry --wallet-password-file=./wallet_password.txt | tee -a import_log.txt done ``` or if you are using staking-deposit-cli: ``` bazel run //validator -- accounts import --keys-dir=/home/blockchain/validator_keys --account-password-file=/home/blockchain/password.txt --wallet-password-file=/home/blockchain/password.txt ``` In order to import the slashing protection db: ``` validator-kl slashing-protection-history export --datadir=/home/ethereum/.pry-geth/kiln/testnet-pry --slashing-protection-export-dir=/home/ethereum/prysm_logs/prysm_slashing_protection/ | tee -a slashing_protection_export.log ``` ### Lighthouse First of all we have synced the beacon node together with the EL client. We have imported the same 250 validators and we have imported the salshing protection JSON file we just exported from Prysm: ``` lighthouse-kl --network kiln account validator slashing-protection import ../prysm_logs/prysm_slashing_protection/slashing_protection.json --datadir=/home/ethereum/.lh-geth/kiln/testnet-lh | tee -a import.log ``` In order to import the validators: ``` for d in /home/ethereum/kiln_validators/assigned_data_0_249/secrets/*; do final_d=$(basename $d) lighthouse-kl --network kiln account validator import --directory /home/ethereum/kiln_validators/assigned_data_0_249/keys/$final_d --datadir=/home/ethereum/.lh-geth/kiln/testnet-lh --password-file $d --reuse-password done ``` or if you are using staking-deposit-cli: ``` ./lighthouse --network kiln account validator import --directory /home/blockchain/validator_keys/ --password-file /home/blockchain/password.txt --reuse-password ``` After running for some hours, we have stopped the client and exported the slashing protection db: ``` lighthouse-kl account validator slashing-protection export /home/ethereum/lighthouse_logs/lighthouse_slashing_protection.json --datadir=/home/ethereum/.lh-geth/kiln/testnet-lh --network kiln ``` ### Nimbus Again, we have synced the beacon node together with the EL client. After this, we have stopped Nimbus (this time the BN and VN are together) and we have imported the 250 validators, as well as the slashing protection db recently exported from Lighthouse: ``` nimbus_beacon_node-kl slashingdb import /home/ethereum/lighthouse_logs/lighthouse_slashing_protection.json --data-dir=/home/ethereum/.nim-geth/kiln/testnet-nim | tee -a import.log ``` It is important to mention that the slashing db import process was succesful, but the terminal output was a bit confusing: ``` INF 2022-06-14 13:06:44.113+00:00 No slashing protection data for validator - initiating topics="antislash" validator=b9f19a99f1217fe0dfef8ee95f23302f0bee9ae7373be6608df1b54e3ebfa0300d2791da212fbbd618bf8c289306d4ea ``` In order to import the validators: ``` for d in /home/ethereum/kiln_validators/assigned_data_0_249/secrets/*; do final_d=$(basename $d) # password=`cat $d` # echo "$password" echo "$password" | tee | nimbus_beacon_node-kl deposits import --data-dir=/home/ethereum/.nim-geth/kiln/testnet-nim "/home/ethereum/kiln_validators/assigned_data_0_249/keys/$final_d" done ``` or if you are using staking-deposit-cli: ``` ./build/nimbus_beacon_node deposits import "/home/blockchain/validator_keys/" Please enter the password for decrypting '/home/blockchain/validator_keys/keystore-m_12381_3600_408_0_0-1657633858.json' ``` After running for some hours, we have stopped the client and exported the slashing protection db: ``` nimbus_beacon_node-kl slashingdb export /home/ethereum/nimbus_logs/slashing_database.json --data-dir=/home/ethereum/.nim-geth/kiln/testnet-nim | tee -a ./nimbus_logs/slashing_logs.log ``` ### Teku We have synced the beacon node together with the EL client. After this we have imported the 250 validators, as well as the slashing protection db: ``` teku-kl slashing-protection import --from /home/ethereum/nimbus_logs/slashing_database.json | tee -a teku_logs/slashing_import.log ``` In order to import the validators: ``` cp -R kiln_validators/assigned_data_0_249/teku-keys/ validator_keys/ cp -R kiln_validators/assigned_data_0_249/teku-secrets/ validator_keys/ ``` We have run the client for some hours. ``` teku-kl slashing-protection export --data-path /home/ethereum/.teku-geth/kiln/datadir-teku --to /home/ethereum/teku_logs/slashing_db.json | tee -a slashing.log ``` ### Grandine We have synced the beacon node together with Geth. After this, we have imported the slashing protection db through: ``` ./grandine-0.2.0_beta3_arm64 --network kiln interchange import /home/ethereum/teku_logs/slashing_db.json | tee -a slashing_import.log ``` We have imported the validators through the following script: ``` for single_key in /home/ethereum/second_1000/assigned_data_0_249/keys/*; do final_pubkey_folder=$(basename $single_key) echo "Importing $final_pubkey_folder" cp -R $single_key/*.json /home/ethereum/.grandine/keys/$final_pubkey_folder.json cp -R /home/ethereum/second_1000/assigned_data_0_249/secrets/$final_pubkey_folder /home/ethereum/.grandine/secrets/$final_pubkey_folder.txt done ``` After this, the client was running correctly and validators were attesting. We have finally exported the slashing protection db through: ``` ./grandine-0.2.0_beta3_arm64 --network kiln interchange export /home/ethereum/grandine_logs/slashing_db.json | tee -a slashing_export.log ``` The process finished successfully ### Lodestar We have first tried the Docker setup but it stopped working, so we had to manually compile the client and execute it. To import validators: ``` for d in /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/second_1000/assigned_data_0_249/lodestar-secrets/*; do final_d=$(basename $d) echo $final_d sudo docker run --rm --name kiln-lodestar_val --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 validator import --network kiln --rootDir /data/lodestar --directory /data/second_1000/assigned_data_0_249/keys/0x$final_d --passphraseFile /data/second_1000/assigned_data_0_249/lodestar-secrets/$final_d done ``` or if you are using staking-deposit-cli: ``` ./lodestar validator import --network kiln --directory /home/blockchain/validator_keys/ --passphraseFile /home/blockchain/password.txt ``` To import / export slashing protection: ``` /home/ethereum/lodestar_logs/lodestar/lodestar validator slashing-protection import --network kiln --file /home/ethereum/grandine_logs/slashing_protection.json --rootDir /home/ethereum/lodestar_logs/lodestar-beacondata/ ``` ``` /home/ethereum/lodestar_logs/lodestar/lodestar validator slashing-protection export --network kiln --file /home/ethereum/lodestar_logs/slashing_protection.json --rootDir /home/ethereum/lodestar_logs/lodestar-beacondata/ ``` ## Automated process After trying out all import/export we have now created a script automating this process. The result is that all clients were executed during the same day, with the same 250 validators, one after the other. No collisions, slashings or attestation issues happened. ### With 2 epochs wait between clients: ![](https://i.imgur.com/NvQb1bn.png) ### Without waiting in between client swap: ![](https://i.imgur.com/2eQJ72n.png) As you may see in the figure above, every 10 epochs there is a cut in rewards, as we have stopped a client and started the next one, sync to the head (which should take minutes) and continue attesting. All of them obtain more or less the same amount of rewards. ## Eth2 Key Manager API For all clients we have first tried with one validator and then with 250 validators. The request used for the experiment is: ``` curl --location --request POST 'http://localhost:5062/eth/v1/keystores' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer api-token-0xd4ae5f49b2a0e991b2cc1a970ef80ccf3d1b5a6c68bd41d8a1d5c47882ab5a19' \ --data-raw '{ "keystores": [ "{\"crypto\"...}" ], "passwords": ["xxx"], "slashing-protection": [] } ``` ### Prysm #### Import ``` { "data": [ { "status": "IMPORTED", "message": "" } ] } ``` ### Delete ``` { "data": [ { "status": "DELETED", "message": "" } ], "slashing_protection": "{\"metadata\":{\"interchange_format_version\":\"5\",\"genesis_validators_root\":\"0x99b09fcd43e5905236c370f184056bec6e6638cfc31a323b304fc4aa789cb4ad\"},\"data\":[{\"pubkey\":\"0xa1330ecf3f213e85c446047544816b841c57417180f6b06fa1b77c9a45e756f0fc425879d595b5dd9a9126cf92120ee7\",\"signed_blocks\":[],\"signed_attestations\":[{\"source_epoch\":\"27635\",\"target_epoch\":\"27636\"},{\"source_epoch\":\"27636\",\"target_epoch\":\"27637\",\"signing_root\":\"0x6ce31098bf46b894d6069e0a0ae88cb4af53cd143fa5a5774197613979822185\"}]}]}" } ``` ### Lighthouse It imports one by one Response: ``` { "data": [ { "status": "imported" } ] } ``` ``` Jul 11 14:10:16.553 INFO Importing keystores via standard HTTP API, count: 1 Jul 11 14:10:16.553 WARN No slashing protection data provided with keystores Jul 11 14:10:18.001 INFO Connected to beacon node(s) synced: 1, available: 1, total: 1, service: notifier Jul 11 14:10:20.207 INFO Enabled validator voting_pubkey: 0xa1330ecf3f213e85c446047544816b841c57417180f6b06fa1b77c9a45e756f0fc425879d595b5dd9a9126cf92120ee7, signing_method: local_keystore Jul 11 14:10:21.445 INFO Modified key_cache saved successfully Jul 11 14:10:21.446 INFO Awaiting activation slot: 878426, epoch: 27450, validators: 1, service: notifier Jul 11 14:10:24.003 INFO Validator exists in beacon chain validator_index: 108207, pubkey: 0xa1330ecf3f213e85c446047544816b841c57417180f6b06fa1b77c9a45e756f0fc425879d595b5dd9a9126cf92120ee7, service: duties ``` ``` { "data": [ { "status": "deleted" }, } ``` ## Nimbus ``` { "data": [ { "status": "imported", "message": "" } ] } ``` ``` { "data": [ { "status": "deleted", "message": "" } ], "slashing_protection": { "metadata": { "interchange_format_version": "5", "genesis_validators_root": "0x99b09fcd43e5905236c370f184056bec6e6638cfc31a323b304fc4aa789cb4ad" }, "data": [ { "pubkey": "0xa1330ecf3f213e85c446047544816b841c57417180f6b06fa1b77c9a45e756f0fc425879d595b5dd9a9126cf92120ee7", "signed_blocks": [], "signed_attestations": [ { "source_epoch": "27636", "target_epoch": "27637", "signing_root": "0x0000000000000000000000000000000000000000000000000000000000000000" } ] } ] } } ``` # Errors ## Prysm Key Manager API When importing 250 keys the client simply gives a timeout. Importing one key works well. ## Lighthouse Eth2 Key Manager API Once validators are imported, the client asks for a fee recipient, which cannot be provided through the API (feature not merged into master yet). We had to use branch unstable to do so. ## Lodestar importing slashing protection When importing the slashing protection, we are having the folowing issues, the parameter to indicate which json file to import does is not recognized by the client: ``` ethereum@ethereumonarm-179983307:~/lodestar_logs/lodestar/kiln/devnets$ sudo docker run --rm --name kiln-lodestar --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-da ta:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 account validator slashing-protection import --network kiln --file /home/ethere um/teku_logs/slashing_db.json | tee -a slashing_import.log [sudo] password for ethereum: ✖ Unknown argument: file ethereum@ethereumonarm-179983307:~/lodestar_logs/lodestar/kiln/devnets$ sudo docker run --rm --name kiln-lodestar --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-da ta:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 account validator slashing-protection import --help | tee -a slashing_import.lo g account [subcommands..] DEPRECATED Options: --rootDir Lodestar root directory [string] --network Name of the Ethereum Consensus chain network to join [string] [choices: "mainnet", "gnosis", "prater", "kiln", "ropsten", "sepolia" , "dev"] [default: "mainnet"] --paramsFile Network configuration file [string] --terminal-total-difficulty-override Terminal PoW block TTD override [string] --terminal-block-hash-override Terminal PoW block hash override [string] --terminal-block-hash-epoch-override Terminal PoW block hash override act ivation epoch [string] --rcConfig RC file to supplement command line a rgs, accepted formats: .yml, .yaml, .json -h, --help Show help [boolean] -v, --version Show version number [boolean] ``` After speaking to the Lodestar team, the command to import the slashing protection is (documentation outdated, "account" command is deprecated): ``` sudo docker run --rm --name kiln-lodestar_val --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 validator slashing-protection import --network kiln --file /data/grandine_slashing_db.json --rootDir /data/lodestar | tee -a slashing_import.log ``` It is worth mentioning that this process does not output any logging but the final message of success or failure. It would be good that they output some logs with the current import process so the user can understand the current performance. The Lodestar team has opened a Github issue to improve this feature. We have now imported the validators through the following script: ``` for d in /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/second_1000/assigned_data_0_249/lodestar-secrets/*; do final_d=$(basename $d) echo $final_d sudo docker run --rm --name kiln-lodestar_val --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 validator import --network kiln --rootDir /data/lodestar --directory /data/second_1000/assigned_data_0_249/keys/0x$final_d --passphraseFile /data/second_1000/assigned_data_0_249/lodestar-secrets/$final_d done ``` We now run the client and after this, we export the slashing protection: ``` sudo docker run --rm --name kiln-lodestar_val --network host -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data:/data -v /home/ethereum/lodestar_logs/lodestar/kiln/devnets/kiln-data/kiln:/config chainsafe/lodestar:arm64 validator slashing-protection export --network kiln --file /data/lodestar_slashing_dd.js on --rootDir /data/lodestar | tee -a slashing_export.log ``` It is important to mention that this process ended successfully, but when importing this json file into Prysm we get the following error: ``` FATAL Could not import slashing protection cli: slashing protection JSON metadata was incorrect: slashing protection JSON version '4' is not supported, wanted '5' ``` We have reported to both teams to understand the issue and try to solve it. ## Nimbus when importing 250 vals through the API ``` /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(454) main /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(447) NimMain /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2167) main /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2035) handleStartUpCmd /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1857) doRunBeaconNode /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1647) start /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1591) run /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(279) poll /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(365) futureContinue /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(94) processRestRequest /home/blockchain/nimbus-eth2/vendor/nim-presto/presto/route.nim(369) restApiEthV1KeystoresPostHandler /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(365) futureContinue /home/blockchain/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/gc.nim(196) restApiEthV1KeystoresPostHandler /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(148) restApiEthV1KeystoresPostHandler /home/blockchain/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/chcks.nim(23) raiseIndexError2 /home/blockchain/nimbus-eth2/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(49) sysFatal [[reraised from: /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(454) main /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(447) NimMain /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2167) main /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2035) handleStartUpCmd /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1857) doRunBeaconNode /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1647) start /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(1591) run /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncmacro2.nim(279) poll /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(365) futureContinue /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(94) processRestRequest /home/blockchain/nimbus-eth2/vendor/nim-presto/presto/route.nim(369) restApiEthV1KeystoresPostHandler /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(387) futureContinue ]] [[reraised from: /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(454) main /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(447) NimMain /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2167) main /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2035) handleStartUpCmd /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1857) doRunBeaconNode /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1647) start /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1591) run /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(279) poll /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(365) futureContinue /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncmacro2.nim(131) processRestRequest [[reraised from: /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(454) main /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(447) NimMain /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2167) main /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2035) handleStartUpCmd /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1857) doRunBeaconNode /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1647) start /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1591) run /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(279) poll /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(365) futureContinue /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncmacro2.nim(131) processRestRequest ]] [[reraised from: /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(454) main /home/blockchain/nimbus-eth2/vendor/nim-libp2p/libp2p/stream/bufferstream.nim(447) NimMain /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2167) main /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(2035) handleStartUpCmd /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1857) doRunBeaconNode /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1647) start /home/blockchain/nimbus-eth2/beacon_chain/nimbus_beacon_node.nim(1591) run /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncloop.nim(279) poll /home/blockchain/nimbus-eth2/vendor/nim-chronos/chronos/asyncfutures2.nim(387) futureContinue ]] Error: unhandled exception: index 1 not in 0 .. 0 [IndexError] ```