# Creating a new multi-client, multi-party testnet This runbook explains the steps involved in running a multi-client testnet involving external parties. The basic steps are: - [ ] Decide date for genesis + delay + fork epoch - [ ] Decide on number of validators - [ ] Contract - [ ] Setup config - [ ] Generate mnemonics - [ ] Generate genesis state - [ ] Send mnemonics - [ ] Deploy 1st client or bootnode - [ ] Get ENR - [ ] PR with the information - [ ] Provision nodes - [ ] Deploy other clients with ENR to get them peering - [ ] Check if genesis root + genesis data matches with client API. It should match what the python script (genesis testnet tool repo) outputs for our genesis.ssz - [ ] Publish list of ENR address of the deployed clients, add as file to the PR - [ ] Setup monitoring and dashboards for the clients - [ ] Setup forkmon - [ ] Setup beacon explorer using a prysm node ## Decide config data Talk to the other clients/participants as to a comfortable time for everyone. 11/12 UTC is a good timezone to attempt genesis, everyone would be awake at that timezone. Plan in at least a few days of delay to allow client teams to bake in genesis and test in their own internal systems + timezone lag. Things that need to be decided from an `altair hf` perspective: min genesis, genesis delay, fork epoch general formula: ``` - --timestamp = when nodes can start peering (in eth2-testnet-genesis tool, for generating genesis state)(This is the same as the MIN_GENESIS_TIME in the eth2 config) - GENESIS_DELAY = added to timestamp option, to get the resulting genesis_time in the state. - genesis_time = The actual moment the network is considered live and epoch 0 starts - genesis_time + ALTAIR_FORK_EPOCH * SLOTS_PER_EPOCH = time of the altair hardfork ``` ## Decide on number of validators Depending on the importance of the testnet, decide the number of participating validators. An internal testnet could have just 1000 vals. A multi-client multi-party should ideally start at ~20,000 vals to allow for an accurate division. ## Deposit contract We will re-use the deposit contract and its associated tools found in this repository: `https://github.com/ethereum/eth2.0-specs` - Clone the repo - Run `make compile_deposit_contract`, this should generate a `build` folder with the - Get your metamask/eth1 address and private key, export it as variables called `ETH1_FROM_ADDR` and `ETH1_FROM_PRIV`(google how to get private key from metamask) - Move into the build folder with `cd build` - Deploy the contract with `ethereal contract deploy --network=goerli --name=DepositContract --json=combined.json --from=$ETH1_FROM_ADDR --privatekey=$ETH1_FROM_PRIV` You should now see a tx hash, enter that in a block explorer for goerli and you should see the deployed contract. At this stage, the following are important to note for the future: - Tx hash of the deployment - Contract address - Deposit block number - Deposit block hash ## Setup config We are now ready to generate the testnet config. Find the latest version of the config here: `https://github.com/eth2-clients/eth2-networks`. - Create a branch in the repo, create a new folder in the appropriate location - Copy over the base config and make edits to the following fields: - `CONFIG_NAME` - `MIN_GENESIS_ACTIVE_VALIDATOR_COUNT` - `MIN_GENESIS_TIME` - `GENESIS_FORK_VERSION` - `GENESIS_DELAY` - `ALTAIR_FORK_EPOCH` - `DEPOSIT_CONTRACT_ADDRESS` ## Generate mnemonics Now that the deposit contract and config are set, we know how many validators we need and where to make the deposits. Typically, for a testnet, all the mnemonics are generated by 1 party and then sent to the teams running the validators. - Mnemonics can be generated by using the `eth2-val-tools` cli tool - Create a `mnemonics.yaml` file (DO NOT COMMIT!) - Enter the mnemonics as a list: ```yaml # Who it's being assigned to - mnemonic: "" # a 24 word BIP 39 mnemonic count: 1234 # amount of validators # Who it's being assigned to - mnemonic: "" # a 24 word BIP 39 mnemonic count: 1234 # amount of validators ``` - The count numbers should add up to the `MIN_GENESIS_ACTIVE_VALIDATOR_COUNT` field in the config ## Generate genesis state We now have all the component parts to generate the genesis state. Since we want to avoid making deposits for all the validators individually, we can just bake the deposits into the genesis state. This can be done with the `eth2-testnet-genesis` tool and it requires the following parameters: - `phase0`/`altair`/`merge`: Specify which genesis state to create, for `altair hf` we use `phase0` since we want to simulate the fork - `config`: The config file created earlier that specifies the chain spec - `eth1-block`: The deposit block hash, block hash of the block in which the contract was deposited - `mnemonics`: The mnemonic file created earlier, specifying the mnemonic and associated validator count - `timestamp`: When the clients can start peering (NOT GENESIS TIME). The `timestamp` value + `genesis delay` = genesis time You should now see a `genesis.ssz` file that contains the genesis state. Confirm the genesis state data is correct with: `zcli pretty phase0 BeaconState genesis.ssz` ## Send mnemonics Reach out to the client teams and send their mnemonic + validator counts. ## Deploy 1st client or bootnode Now that we have a setup network, we need to setup the bootnode so peers can find each other. To do this: - Create a AWS instance to host the bootnode, ideally in the `cluster` repo - Create an inventory, ideally in `eth2.0-devops/testnets/<testnet-name>-bootnodes` - Export the private key for the bootnode with `export <testnetname>_BOOTNODE_PRIV_101="xxxxxxx"` - Add the variables needed for the bootnode, update the `GENESIS_FORK_VERSION` to reflect the new testnet - Run the `setup_network_bootnodes.yml` playbook in `ansible_eth2/playbooks` to setup the bootnode with the config - SSH into the bootnode and check the logs with `docker logs bootnode` - Grab the field called `ENR` - Confirm the fields using `https://enr-viewer.com/` ## PR with the information We should have all the information required by the client teams for their individual setups. So we create a PR in the `eth2-clients/eth2-networks` repo. The following files need to be added in the respective folder for the testnet: - `README.md`: One liner for the testnet - `bootstrap_nodes.txt`: Containing the ENR of the bootnodes or ready clients - `config.yaml`: The config for the testnet - `deploy_block.txt`: The block at which the deposit contract was deployed - `deposit_contract.txt`: The deposit contract address - `genesis.ssz`: The genesis file created previously ## Provision nodes The planning side of the testnet is now done. Provision the nodes as usual, Terraform on `cluster` repo + Ansible in `eth2.0-devops`. ## Deploy other clients with ENR to get them peering Use the config to create keys + deploy clients in the wished % division. ## Check if genesis data is correct on clients - SSH into the AWS instances and check the client genesis data with an API request to the beacon node. This should work: `curl localhost:4000/eth/v1/beacon/genesis` - Use the `compute_genesis_details.py` file in the `testnet_config folder to cross-check the data ## Publish list of ENR address Update the `bootstrap_nodes.txt` file in the PR with some more client ENRs, if they are stable. ## Setup monitoring and dashboards for the clients Set the monitoring variables (`push_metrics_enabled: true`) and check the data in `grafana`. Copy the existing dashboards and change the datasource to match the new filter.