gogo's oracle - HackMD

# gogo's oracle ## production links - [aggregated logs](https://marvin.thgttg.com/grafana/goto/OhUWMgWNR?orgId=1) (log in with github, metacraft-labs gh org members) - [lighthouse beacon api](https://lighthouse.dendreth.io/eth/v1/node/version) - [nimbus beacon api](https://nimbus.dendreth.io/eth/v1/node/version) ## setup dev environment ### docker - [install docker](https://docs.docker.com/engine/install/fedora) ```bash= sudo dnf install -y dnf-plugins-core sudo dnf config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo sudo dnf install -y \ docker-ce \ docker-ce-cli \ containerd.io \ docker-buildx-plugin \ docker-compose-plugin sudo usermod -a -G docker $(whoami) sudo systemctl enable --now docker.service systemctl status docker.service ``` ### nix - install nix *[zero-to-nix](https://zero-to-nix.com/start/install)* ```bash= curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ``` - get rid of `untrusted substituter` warnings in nix command output. there are better ways to handle this, but this is quick. ```bash= grep trusted-users /etc/nix/nix.conf &> /dev/null \ || echo "trusted-users = $(whoami)" | sudo tee --append /etc/nix/nix.conf sudo systemctl restart nix-daemon.service ``` ### redis - install redis ```bash= sudo dnf install -y redis sudo systemctl enable --now redis.service ``` ### repository - fetch dendreth, checkout branch, fetch submodules, fetch dependencies (optional, cm build has no need for local source code) ```bash= repo_url=https://github.com/metacraft-labs/DendrETH.git repo_path=~/git/metacraft-labs/DendrETH repo_branch=diva-deployment git clone ${repo_url} ${repo_path} cd ${repo_path} git checkout ${repo_branch} git submodule update --init --recursive yarn install ``` ### starship - (optional) get rid of starship git timeout warnings (only needed if you use [starship](https://starship.rs)). because the dendreth repo contains many submodules, starship needs more time (than the default of 500ms) to compute git metadata ```bash= grep command_timeout ~/.config/starship.toml &> /dev/null \ || echo 'command_timeout = 3000' >> ~/.config/starship.toml ``` ## build - build commitment-mapper (builds directly from github, not the local repo) ```bash= mkdir -p /tmp/dendreth-build cd /tmp/dendreth-build nix \ --extra-experimental-features 'nix-command flakes' \ build 'github:metacraft-labs/DendrETH/diva-deployment#circuit-executables.commitment-mapper.levels.all.pkg' ``` - build from local source ```bash= repo=${HOME}/git/metacraft-labs/DendrETH cargo +nightly build \ --release \ --manifest-path ${repo}/beacon-light-client/plonky2/crates/circuit_executables/Cargo.toml declare -a binaries=() binaries+=( balance_verification ) binaries+=( balance_verification_circuit_data_generation ) binaries+=( bls12_381 ) binaries+=( bls12_381_circuit_data_generation ) binaries+=( bls12_381_components_proofs ) binaries+=( calc_pairing_precomp ) binaries+=( calc_pairing_precomp_circuit_data_generation ) binaries+=( cloud_healthcheck ) binaries+=( commitment_mapper ) binaries+=( commitment_mapper_circuit_data_generation ) binaries+=( deposit_accumulator_balance_aggregator_diva ) binaries+=( deposit_accumulator_balance_aggregator_diva_circuit_data_generation ) binaries+=( deposit_accumulator_balance_aggregator_final_layer ) binaries+=( final_exponentiate ) binaries+=( final_exponentiate_circuit_data_generation ) binaries+=( final_layer ) binaries+=( fp12_mul ) binaries+=( fp12_mul_circuit_data_generation ) binaries+=( miller_loop ) binaries+=( miller_loop_circuit_data_generation ) binaries+=( pubkey_commitment_mapper ) for binary in ${binaries[@]}; do binary_local_path=${repo}/beacon-light-client/plonky2/crates/target/release/${binary} binary_remote_path=/var/lib/dendreth/release/${binary//_/-} echo ${binary//_/-} sha256sum ${binary_local_path} | cut -d ' ' -f 1 | tr -d '\n' | tee ${binary_local_path}.sha256 echo echo rsync \ --rsh 'ssh -o ConnectTimeout=1 -i ~/.ssh/id_quadbrat -p 52201' \ --rsync-path='sudo rsync' \ --archive \ ${binary_local_path} \ mitko.thgttg.com:${binary_remote_path} rsync \ --rsh 'ssh -o ConnectTimeout=1 -i ~/.ssh/id_quadbrat -p 52201' \ --rsync-path='sudo rsync' \ --archive \ ${binary_local_path}.sha256 \ mitko.thgttg.com:${binary_remote_path}.sha256 done ``` ## publish release - put the commitment mapper binary somewhere public ```bash= binary_path=$(tail -n 1 /tmp/dendreth-build/result/bin/commitment_mapper | cut -d ' ' -f 2 | tr -d '\n') rsync \ --rsh 'ssh -o ConnectTimeout=1 -i ~/.ssh/id_quadbrat -p 52201' \ --rsync-path='sudo rsync' \ --archive \ --verbose \ ${binary_path} \ mitko.thgttg.com:/var/lib/dendreth/release/commitment-mapper sha256sum ${binary_path} | cut -d ' ' -f 1 | tr -d '\n' | tee /tmp/dendreth-build/commitment-mapper.sha256 rsync \ --rsh 'ssh -o ConnectTimeout=1 -i ~/.ssh/id_quadbrat -p 52201' \ --rsync-path='sudo rsync' \ --archive \ --verbose \ /tmp/dendreth-build/commitment-mapper.sha256 \ mitko.thgttg.com:/var/lib/dendreth/release/commitment-mapper.sha256 ``` - deploy commitment mapper circuits to production ```bash= circuits_path=$(tail -n 1 /tmp/dendreth-build/result/bin/commitment_mapper | cut -d ' ' -f 4 | tr -d '\n') rsync \ --rsh 'ssh -o ConnectTimeout=1 -i ~/.ssh/id_quadbrat -p 52213' \ --rsync-path='sudo -u dendreth rsync' \ --archive \ --verbose \ --copy-links \ --delete \ ${circuits_path}/ \ gramathea.thgttg.com:/var/lib/dendreth/data/circuits ``` ## install/deploy - install commitment-mapper (to the temp folder here, but we'll orchestrate remote deployment) ```bash= # set up folders mkdir -p /tmp/dendreth/{bin,data/{proof,serialized_circuits}} # install the binary binary_path=$(grep commitment_mapper /tmp/dendreth-build/result/bin/commitment_mapper | cut -d ' ' -f 2) cp \ ${binary_path} \ /tmp/dendreth/bin/commitment-mapper # install the serialized circuits cp \ /tmp/dendreth-build/result/data/serialized_circuits/* \ /tmp/dendreth/data/serialized_circuits/ ``` ## run commitment mapper - run commitment mapper with **aws** proof storage ```bash= /tmp/dendreth/bin/commitment-mapper \ --serialized-circuits-dir /tmp/dendreth/data/serialized_circuits \ --proof-storage-type aws \ --aws-region us-east-1 \ --aws-bucket-name dendreth-validators-commitment-mapper \ --redis redis://127.0.0.1:6379 ``` - run commitment mapper with **filesystem** proof storage ```bash= /tmp/dendreth/bin/commitment-mapper \ --serialized-circuits-dir /tmp/dendreth/data/serialized_circuits \ --proof-storage-type file \ --folder-name /tmp/dendreth/data/proof \ --redis redis://127.0.0.1:6379 ``` ## run input fetchers - init ```bash= cd ~/git/metacraft-labs/DendrETH yarn cd ~/git/metacraft-labs/DendrETH/beacon-light-client/plonky2/input_fetchers ``` - populate tasks for commitment mapper proof generation. this process runs continuously, monitoring changes in the validators tree and queuing update tasks. ```bash= GIT_ROOT=${HOME}/git/metacraft-labs/DendrETH yarn ts \ validators_commitment_mapper/runnable/run_scheduler.ts \ --redis-host 127.0.0.1 \ --redis-port 6379 \ --take 10 \ --beacon-node https://caveman.thgttg.com ``` - run the commitment mapper for validators depositing through the Diva contract ```bash= GIT_ROOT=${HOME}/git/metacraft-labs/DendrETH yarn ts \ balance_verification/deposits_accumulator/runnable/run_pubkey_commitment_mapper_scheduler.ts \ --protocol diva \ --contract-address 0xb4B46bdAA835F8E4b4d8e208B6559cD267851051 \ --json-rpc http://caveman.thgttg.com:8545 ``` ## deployment ### planned #### servers - [ ] beacon.dendreth.io - [x] geth (mainnet) - [x] nimbus beacon (mainnet) - [x] redis.dendreth.io - [x] redis - [ ] ???.dendreth.io - [ ] commitment-mapper (systemd) - [ ] ???.dendreth.io - [ ] input fetchers (pm2 - maybe) ### current ## monitoring the only grafana dashboard i could find with a host selector was: [14053](https://grafana.com/grafana/dashboards/14053-geth-overview)