gnoyager 1 - HackMD

![](https://rob.tn/gnoyager1/drifting-through-the-cosmos.png "drifting through the cosmos" =720x) # gnoyager 1 ## 1) start a gnoland node :::success **completed** - deployment of (and updates to) gnoland binary and systemd service is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1 - gnoland node runs as a systemd service on gnoyager1.v8r.io ::: ### notes - there is currently no publicly visible/downloadable gnoland node binary. this means it is difficult to ascertain the causes of issues or problems experienced by the wider gnoland community when things go wrong. ie: - what version of the source code was used for the binary build when the issue occured? - what os or environment configuration was used in the deployment environment that may have contributed to the issue? we must fix this first, since we want to create repeatable environments for debugging issues in order to be able to use consistent language and instructions for solving problems. ie: > if you experience *x* when running *y* version `1.0.0`, upgrade to version `1.0.1` where issue *x* was resolved by commit *z*. in order to reduce friction for the node-running community, we must provide static binary downloads for at least the x86_64 linux os/arch target which is what the majority of node runners will be familiar with. while it is perfectly acceptable to expect the developer/source-code-contributor community to maintain a local golang development environment to facilitate a binary build, this is not the case for the node-runner community who are better served by access to pre-built, versioned, binary releases. node-runners maintain nodes for multiple projects and expecting them to maintain a development environment for the idiosyncracies of each project they run nodes for, creates an unnecessary barrier to entry and complicates adoption. ### implementation we will resolve this step with two distinct implementations. 1. resolve the missing binary release with a github action to build and publish it 2. automate deployment of the node binary to, and maintenance of, its daemon on a dedicated instance #### node binary releases convention dictates that releases are maintained within and closely coupled with the source control mechanism. since gnoland's official repository is on github and it appears there is already a *release* creation mechanism, it makes good sense to implement the *binary publish* mechanism as a new github action which is triggered by the creation of a *release*. ie: `.github/workflows/binary-publish.yml` ```yaml= --- on: release: types: - created permissions: contents: write packages: write jobs: publish-binaries: name: publish binaries strategy: fail-fast: false matrix: goversion: - "1.22" goarch: - "386" - amd64 - arm64 goos: - darwin - linux - windows program: - gnokey - gnoland - gnoweb exclude: - goarch: "386" goos: darwin - goarch: arm64 goos: windows runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version: ${{ matrix.goversion }}.x - uses: wangyoucao577/go-release-action@v1 with: github_token: ${{ secrets.GITHUB_TOKEN }} goos: ${{ matrix.goos }} goarch: ${{ matrix.goarch }} goversion: ${{ matrix.goversion }} project_path: ./gno.land/cmd/${{ matrix.program }} binary_name: ${{ matrix.program }} md5sum: true sha256sum: true ``` the github action above could be improved by - creating one `gno-$version-$os-$arch.tar.gz` release artifact for each target, containing all three binaries and the respective archive checksums, but since that isn't supported by the community action used, i opted to leave that for a future implementation when i can give it more of my attention - enabling a tag filter which only creates binary releases when the release tag matches a given pattern, ie: tags that contain only the first three semver components (`$major.$minor.$patch`) without an aditional label following a hyphen (`$major.$minor.$patch-$label`). this would require a little discussion with the team about the preferred tag naming conventions - integrate testing into the action workflow and prevent binary publish when any unit or integration tests fail after the binary is built - the dependency on a community github action (`wangyoucao577/go-release-action@v1`) is suboptimal. it introduces a vulnerability whereby our binary build process could be exploited if/when that gh action's source code is compromised. the exploit could potentially compromise some or all of our nodes, if or when they are updated to a future release, if the 3rd party gh action were to inject malicious instructions during its build process. since it is possible to create a new gh action whose source code is under our control, it would be safer to do just that, time permitting in the interest of demonstrating a solution on a repo where i have push permissions, the above action is enabled on a gno repo fork here: https://github.com/grenade/gno/blob/master/.github/workflows/binary-publish.yml. a successful binary publish action run, after a little trial and error, can be seen here: https://github.com/grenade/gno/actions/runs/9050588857 the resulting binary releases are automatically published by the action here: https://github.com/grenade/gno/releases/tag/v1.0.5 #### node deployment orchestration we can automate deployment using a number of different mechanisms (terraform, ansible, puppet, chef, docker, etc), however for the purpose of this excercise, we'd like to keep things simple and require no dependency or orchestration agent installations on the target instance. we could use simple bash commands and tools (like ssh, rsync and systemd) that are already installed on the vanilla linux installations most frequently utilised by node-runners (like ubuntu and fedora). under the hood, many of the orchestration mechanisms mentioned earlier are just going to use those same tools after their respective token replacement and scripting abstractions have run. almost every orchestration platform supports some sort of scripting engine that does things like token replacement in manifest files. this allows for reducing duplication when, for example, two instances both need the same software installed. this has the unfortunate consequence that manifests quickly become unreadable because they refer to tokens whose runtime values need to be looked up elsewhere. fortunately, whilst working at mozilla, i authored an orchestration tool called *rubberneck*, which ideologically borrows a lot from and then extends [cloud-init](https://cloud-init.io) to work well on non-cloud deployments. although by design, it does not support token replacement in manifests. this intentional "limitation" ensures that an instance manifest contains all of the deployment and configuration required by the described instance. rubberneck is intentionally verbose and where two or more instances have identical configuration, there are always going to be two or more duplicate identical manifest configurations. creating manifests can be a little tedious because of this enforced duplication, but where this idea begins to make sense is when things go wrong and someone who isn't familiar with the orchestration manifest templating language, needs to debug configuration which has been deployed to an instance. because rubberneck manifests are written in yaml and contain no replaceable tokens, anyone with limited or no orchestration experience can still understand the intended instance configuration by reading a single manifest file written in yaml (and possibly an associated configuration file under a subdirectory in the same folder as the manifest). another benefit of this approach is that it enables throttled rollouts of changes to large numbers of instances. it is possible to make changes to any particular percentage of an infra landscape and increase that percentage only after confidence around a particular rollout or upgrade is high. rubberneck relies heavily on source control to manage change on deployment targets. because of this, the timing of issues or failures makes it easy to associate problems with changes to instance manifests in source control. ##### the deployment platform are we deploying to a big cloud? a bare-metal provider? our own datacentre? our own pc? rubberneck doesn't care. it just needs for the target instance to grant it access by having a local sudo-enabled user whose authorised keys includes the correct rubberneck public key and for that instance to be running an ssh server on a known port. let's assume we already have (i do, since i run a small, bare-metal, datacenter with a couple of racks of spare capacity): - a web accessible, fedora or ubuntu instance - with an active and correctly resolving, dns record named `gnoyager1.v8r.io` - running an ssh server on an accessible port - with a maintainer user which has `sudo` rights that can be used to install software and configuration - an authorised keys file on the instance, under the home path of the maintainer user: `~/.ssh/authorized_keys` which contains a public key for which we hold a corresponding private key, so that we can access the instance with the maintainer user's sudo privileges let's assume we'd like to have the gnoland node running: - as a systemd service, so that it recovers and restarts after system reboots and/or crashes - running under a dedicated service user account named `gnoland` with enough rights to run the node and not much else, since that's good practice - using (os) conventional paths for: - node binary: `/usr/local/bin/gnoland` - unit file: `/etc/systemd/system/gnoland.service` - gnoland $HOME: `/var/lib/gnoland` ##### rubberneck configuration - instance manifest `manifest/gno/gnoyager1/manifest.yml` ```yaml= --- hostname: gnoyager1 domain: v8r.io action: sync ssh: port: 2212 os: name: fedora package: - git - cockpit-pcp - curl - dnf-automatic user: - username: grenade authorized: keys: - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPRO2rPB5URKyMSGeMwFd9Npzl/XywJWO9F2N/xylCVm - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPb24HEQ++aNFWaqVyMtIs6GotUB8R+q61XOoI2z6uMj - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID83JhRg/lgInWM/XwMfjaDzTMDPS5M7zuVeOm0O5Y5W - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGILqCEkUGPF3c+riHfLtkpsSP3nw/HjQUvTs66JsZ6a command: - sudo passwd -l root - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter ) - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland ) file: - source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service target: /etc/systemd/system/prometheus-node-exporter.service command: pre: - systemctl is-active --quiet prometheus-node-exporter.service && sudo systemctl stop prometheus-node-exporter.service post: - sudo systemctl daemon-reload - sudo systemctl enable prometheus-node-exporter.service - sudo systemctl start prometheus-node-exporter.service - source: https://raw.githubusercontent.com/gnolang/gno/7d44813eece8d8968408b4978ba86a536ea0d75d/gno.land/genesis/genesis_balances.txt target: /var/lib/gnoland/genesis/genesis_balances.txt command: pre: - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland - sudo -u gnoland mkdir -p /var/lib/gnoland/genesis post: - sudo chown -R gnoland:gnoland /var/lib/gnoland/genesis/genesis_balances.txt - sudo systemctl start gnoland.service - source: https://raw.githubusercontent.com/gnolang/gno/7d44813eece8d8968408b4978ba86a536ea0d75d/gno.land/genesis/genesis_txs.jsonl target: /var/lib/gnoland/genesis/genesis_txs.jsonl command: pre: - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland - sudo -u gnoland mkdir -p /var/lib/gnoland/genesis post: - sudo chown -R gnoland:gnoland /var/lib/gnoland/genesis/genesis_txs.jsonl - sudo systemctl start gnoland.service - source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/gnoland.service target: /etc/systemd/system/gnoland.service command: pre: - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland - sudo -u gnoland mkdir -p /var/lib/gnoland/data - test -d /var/lib/gnoland/gno || sudo -u gnoland git clone https://github.com/gnolang/gno.git /var/lib/gnoland/gno - test -d /var/lib/gnoland/gno && sudo -u gnoland git --git-dir /var/lib/gnoland/gno/.git --work-tree /var/lib/gnoland/gno pull - test -d /github || sudo mkdir /github - test -L /github/workspace || sudo ln -sf /var/lib/gnoland/gno /github/workspace post: - sudo systemctl daemon-reload - sudo systemctl enable gnoland.service - sudo systemctl start gnoland.service ``` - node unit file `manifest/gno/gnoyager1/etc/systemd/system/gnoland.service` ```ini= [Unit] Description=gnoland blockchain node Wants=network-online.target After=network-online.target ConditionPathExists=/var/lib/gnoland/data ConditionPathExists=/var/lib/gnoland/gno ConditionPathExists=/var/lib/gnoland/genesis/genesis_balances.txt ConditionPathExists=/var/lib/gnoland/genesis/genesis_txs.jsonl ConditionPathExists=/usr/local/bin/gnoland ConditionPathExists=/github/workspace [Service] Type=simple User=gnoland Group=gnoland WorkingDirectory=/var/lib/gnoland ExecStart=/usr/local/bin/gnoland \ start \ -data-dir /var/lib/gnoland/data \ -gnoroot-dir /var/lib/gnoland/gno \ -genesis /var/lib/gnoland/genesis.json \ -genesis-balances-file /var/lib/gnoland/genesis/genesis_balances.txt \ -genesis-txs-file /var/lib/gnoland/genesis/genesis_txs.jsonl Restart=always RestartSec=3 [Install] WantedBy=multi-user.target ``` ##### observations `gnoland start` fails to run if: - `genesis_balances.txt` or `genesis_txs.jsonl` is missing. this is reasonable but we ought to document specifically for node runners where to obtain these files for the specific blockchain they want to run (dev, testnet, mainnet) - `/github/workspace` does not exist. this should be corrected. even when running the binary with `-gnoroot-dir /some/path` the process will fail with inelegant logging which indicates that `/github/workspace` does not exist. it isn't all that reasonable to expect this specific path to exist on everyone's instance. it's easily worked around with symlinking or similar but creating a new specific path convention and expecting everyone to adhere to it feels sloppy. - the gnoland binary expects to have a copy of the source repo on the filesystem. this is really unusual and for that reason alone feels like we should try to remove this dependency. i don't know the history of why this dependency exists but it creates the impression that it was just done to get something working quickly without considering that non-developers (ie: node runners) are often the type of people who won't bother running your nodes if they have to learn to keep repos synced. it adds another entry barrier to community adoption that feels unneccessary. ## 2) start the transaction indexer :::success **completed** - deployment of (and updates to) tx-indexer binary and systemd service is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1 - tx-indexer runs as a systemd service on gnoyager1.v8r.io - tx-indexer [playground](https://gnoyager1.v8r.io/graphql) and [endpoint](https://gnoyager1.v8r.io/graphql/query) are served over tls ::: ### rubberneck configuration - update instance manifest to install `tx-indexer` binary and `tx-indexer.service` systemd unit file `git diff 57c9b78e98..57dedaf1e9 manifest/gno/gnoyager1/manifest.yml` ```diff diff --git a/manifest/gno/gnoyager1/manifest.yml b/manifest/gno/gnoyager1/manifest.yml index af860e447d..31a3ccc8a6 100644 --- a/manifest/gno/gnoyager1/manifest.yml +++ b/manifest/gno/gnoyager1/manifest.yml @@ -25,7 +25,7 @@ command: - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter ) - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland ) + - test -x /usr/local/bin/tx-indexer || ( curl -sLo /tmp/tx-indexer_0.3.0_linux_amd64 https://github.com/gnolang/tx-indexer/releases/download/v0.3.0/tx-indexer_0.3.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/tx-indexer_0.3.0_linux_amd64 -C /usr/local/bin tx-indexer ) file: - source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service @@ -75,3 +75,15 @@ file: - sudo systemctl daemon-reload - sudo systemctl enable gnoland.service - sudo systemctl start gnoland.service + - + source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/tx-indexer.service + target: /etc/systemd/system/tx-indexer.service + command: + pre: + - systemctl is-active --quiet tx-indexer.service && sudo systemctl stop tx-indexer.service + - id tx-indexer || sudo useradd --system --create-home --home-dir /var/lib/tx-indexer --user-group tx-indexer + - sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data + post: + - sudo systemctl daemon-reload + - sudo systemctl enable tx-indexer.service + - sudo systemctl start tx-indexer.service ``` - tx-indexer unit file `manifest/gno/gnoyager1/etc/systemd/system/tx-indexer.service` ```ini= [Unit] Description=gnoland transaction indexer Wants=network-online.target After=network-online.target ConditionPathExists=/var/lib/tx-indexer/data ConditionPathExists=/usr/local/bin/tx-indexer [Service] Type=simple User=tx-indexer Group=tx-indexer WorkingDirectory=/var/lib/tx-indexer ExecStart=/usr/local/bin/tx-indexer \ start \ -db-path /var/lib/tx-indexer/data \ -listen-address 0.0.0.0:8546 \ -remote http://127.0.0.1:26657 \ -log-level info \ -http-rate-limit 0 \ -max-chunk-size 100 \ -max-slots 100 Restart=always RestartSec=3 [Install] WantedBy=multi-user.target ``` #### observations tx-indexer already provides a binary release so we make use of that and just add a systemd service unit file to manage its daemon. everything works as expected without any surprises. since the tx-indexer deployment sits on a private network but i want to be able to reach it from the web, i had a little extra configuration to reverse proxy requests over tls. i achieved this by [modifying nginx config](https://github.com/grenade/rubberneck/compare/36924625a3..60e4830076) in rubberneck for the gateway instance which exposes the tx-indexer endpoints at: https://gnoyager1.v8r.io (ie: https://gnoyager1.v8r.io/graphql). #### crash handling tx-indexer occassionaly crashes with journal entries like so: ```text= May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024/05/17 06:03:13 [JOB 1] WAL file /var/lib/tx-indexer/data/001002.log with log number 001002 stopped reading at offset: 0; replayed 0 keys in 0 batches May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.552Z INFO http-server serve/server.go:46 HTTP server started {"address": "[::]:8546"} May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z INFO fetcher fetch/fetch.go:111 Fetching range {"from": 13409, "to": 13508} May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z INFO fetcher fetch/fetch.go:111 Fetching range {"from": 13509, "to": 13608} May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z INFO fetcher fetch/fetch.go:111 Fetching range {"from": 13609, "to": 13688} May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: panic: runtime error: index out of range [1] with length 1 May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: goroutine 48 [running]: May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.getTxResultFromBatch({0xc000545800?, 0x50, 0x80}, {0x12a3790?, 0xc000486090?}) May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:181 +0x553 May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.handleChunk.func1() May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:38 +0xf3 May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.handleChunk({0x12a3838, 0xc00019e000}, {0x12a3790?, 0xc000486090?}, 0xc00011ce58) May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:47 +0x8d May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: created by github.com/gnolang/tx-indexer/fetch.(*Fetcher).FetchChainData.func1 in goroutine 83 May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/fetch.go:123 +0x2dd May 17 06:03:13 gnoyager1.v8r.io systemd[1]: tx-indexer.service: Main process exited, code=exited, status=2/INVALIDARGUMENT May 17 06:03:13 gnoyager1.v8r.io systemd[1]: tx-indexer.service: Failed with result 'exit-code' ``` in order to maintain progress on step 5, i simply delete its data directory and restart. ```bash= sudo systemctl stop tx-indexer.service sudo -u tx-indexer rm -rf /var/lib/tx-indexer/data sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data sudo systemctl start tx-indexer.service ``` ## 3) start supernova stress testers :::success **completed** - deployment of (and updates to) supernova binary and 3 systemd services (realm call, realm deploy, package deploy) is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1 - each of the three supernova modes runs as a systemd service on gnoyager1.v8r.io ::: ### rubberneck configuration - update instance manifest ```bash= git diff 57dedaf..36924625a3 manifest/gno/gnoyager1/manifest.yml ``` ```diff= diff --git a/manifest/gno/gnoyager1/manifest.yml b/manifest/gno/gnoyager1/manifest.yml index 31a3ccc8a6..33be36fa9d 100644 --- a/manifest/gno/gnoyager1/manifest.yml +++ b/manifest/gno/gnoyager1/manifest.yml @@ -25,7 +25,8 @@ command: - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter ) - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland ) - test -x /usr/local/bin/tx-indexer || ( curl -sLo /tmp/tx-indexer_0.3.0_linux_amd64.tar.gz https://github.com/gnolang/tx-indexer/releases/download/v0.3.0/tx-indexer_0.3.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/tx-indexer_0.3.0_linux_amd64.tar.gz -C /usr/local/bin tx-indexer ) + - test -x /usr/local/bin/supernova || ( curl -sLo /tmp/supernova_1.2.0_linux_amd64.tar.gz https://github.com/gnolang/supernova/releases/download/v1.2.0/supernova_1.2.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/supernova_1.2.0_linux_amd64.tar.gz -C /usr/local/bin supernova ) file: - source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service @@ -87,3 +88,53 @@ file: - sudo systemctl daemon-reload - sudo systemctl enable tx-indexer.service - sudo systemctl start tx-indexer.service + - + source: https://raw.githubusercontent.com/gnolang/supernova/main/scripts/p/package.gno + target: /var/lib/supernova/scripts/p/package.gno + command: + pre: + - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova + - sudo -u supernova mkdir -p /var/lib/supernova/scripts/p + - + source: https://raw.githubusercontent.com/gnolang/supernova/main/scripts/r/realm.gno + target: /var/lib/supernova/scripts/r/realm.gno + command: + pre: + - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova + - sudo -u supernova mkdir -p /var/lib/supernova/scripts/r + - + source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-call.service + target: /etc/systemd/system/supernova-realm-call.service + command: + pre: + - systemctl is-active --quiet supernova-realm-call.service && sudo systemctl stop supernova-realm-call.service + - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova + - sudo -u supernova mkdir -p /var/lib/supernova/data + post: + - sudo systemctl daemon-reload + - sudo systemctl enable supernova-realm-call.service + - sudo systemctl start supernova-realm-call.service + - + source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-deployment.service + target: /etc/systemd/system/supernova-realm-deployment.service + command: + pre: + - systemctl is-active --quiet supernova-realm-deployment.service && sudo systemctl stop supernova-realm-deployment.service + - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova + - sudo -u supernova mkdir -p /var/lib/supernova/data + post: + - sudo systemctl daemon-reload + - sudo systemctl enable supernova-realm-deployment.service + - sudo systemctl start supernova-realm-deployment.service + - + source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-package-deployment.service + target: /etc/systemd/system/supernova-package-deployment.service + command: + pre: + - systemctl is-active --quiet supernova-package-deployment.service && sudo systemctl stop supernova-package-deployment.service + - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova + - sudo -u supernova mkdir -p /var/lib/supernova/data + post: + - sudo systemctl daemon-reload + - sudo systemctl enable supernova-package-deployment.service + - sudo systemctl start supernova-package-deployment.service ``` - supernova unit files - `manifest/gno/gnoyager1/etc/systemd/system/supernova-package-deployment.service` ```ini= [Unit] Description=supernova gnoland stress test package deployment Wants=network-online.target After=network-online.target ConditionPathExists=/var/lib/supernova/data ConditionPathExists=/usr/local/bin/supernova [Service] Type=simple User=supernova Group=supernova WorkingDirectory=/var/lib/supernova ExecStart=/usr/local/bin/supernova \ -batch 20 \ -chain-id dev \ -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \ -mode PACKAGE_DEPLOYMENT \ -output /var/lib/supernova/data/package-deployment.json \ -sub-accounts 10 \ -transactions 100 \ -url http://127.0.0.1:26657 Restart=always RestartSec=300 [Install] WantedBy=multi-user.target ``` - `manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-call.service` ```ini= [Unit] Description=supernova gnoland stress test realm call Wants=network-online.target After=network-online.target ConditionPathExists=/var/lib/supernova/data ConditionPathExists=/usr/local/bin/supernova [Service] Type=simple User=supernova Group=supernova WorkingDirectory=/var/lib/supernova ExecStart=/usr/local/bin/supernova \ -batch 20 \ -chain-id dev \ -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \ -mode REALM_CALL \ -output /var/lib/supernova/data/realm-call.json \ -sub-accounts 10 \ -transactions 100 \ -url http://127.0.0.1:26657 Restart=always RestartSec=300 [Install] WantedBy=multi-user.target ``` - `manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-deployment.service` ```ini= [Unit] Description=supernova gnoland stress test realm deployment Wants=network-online.target After=network-online.target ConditionPathExists=/var/lib/supernova/data ConditionPathExists=/usr/local/bin/supernova [Service] Type=simple User=supernova Group=supernova WorkingDirectory=/var/lib/supernova ExecStart=/usr/local/bin/supernova \ -batch 20 \ -chain-id dev \ -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \ -mode REALM_DEPLOYMENT \ -output /var/lib/supernova/data/realm-deployment.json \ -sub-accounts 10 \ -transactions 100 \ -url http://127.0.0.1:26657 Restart=always RestartSec=300 [Install] WantedBy=multi-user.target ``` #### observations supernova makes use of a couple of directories (`./scripts/p`, `./scripts/r`) which aren't mentioned in setup documentation so the first time a new user hears about them is when supernova errors because it can't find them. we can improve the user experience by describing these directories and what needs to be in them in the getting-started/setup docs or by implementing a setup/install script that creates the required folders and their contents when they don't exist (ie: when the user has installed the release binary, rather than building from source). i opted to create three separate services for triggering each of the three supernova modes. ## handling a borked (dev) chain state :::warning **restart fresh dev chain** ::: the dev chain seems to crash ~~consistently~~ sometimes around block height 2723. i haven't yet been able to determine why it does so and currently use the following script to reset the chain to the genesis block so that i can continue making progress on the step 5 dashboard. ```bash= sudo ls -ahl /var/lib/gnoland/data sudo ls -ahl /var/lib/tx-indexer/data sudo systemctl stop supernova-realm-deployment.service sudo systemctl stop supernova-realm-call.service sudo systemctl stop supernova-package-deployment.service sudo systemctl stop tx-indexer.service sudo systemctl stop gnoland.service sudo -u gnoland rm -rf /var/lib/gnoland/data sudo -u gnoland rm /var/lib/gnoland/genesis/genesis.json sudo -u tx-indexer rm -rf /var/lib/tx-indexer/data sudo -u gnoland mkdir -p /var/lib/gnoland/data sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data sudo systemctl start gnoland.service sudo systemctl start tx-indexer.service sleep 10 sudo systemctl start supernova-package-deployment.service sudo systemctl start supernova-realm-call.service sudo systemctl start supernova-realm-deployment.service sudo ls -ahl /var/lib/gnoland/data sudo ls -ahl /var/lib/tx-indexer/data ``` the journalctl output from a borked chain looks like so: ```log= May 16 09:12:40 gnoyager1.v8r.io systemd[1]: Started gnoland.service - gnoland blockchain node. May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: __ __ May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: ___ ____ ___ / /__ ____ ___/ / May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: / _ `/ _ \/ _ \_ / / _ `/ _ \/ _ / May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: \_, /_//_/\___(_)_/\_,_/_//_/\_,_/ May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: /___/ May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Starting multi {"module": "proxy", "impl": "multi"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Starting localClient {"module": "proxy", "module": "abci-client", "connection": "query", "impl": "localClient"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Starting localClient {"module": "proxy", "module": "abci-client", "connection": "mempool", "impl": "localClient"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Starting localClient {"module": "proxy", "module": "abci-client", "connection": "consensus", "impl": "localClient"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Starting EventStoreService {"module": "eventstore", "impl": "EventStoreService"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO ABCI Handshake App Info {"module": "consensus", "height": 2723, "hash": "BFEB55FAE4E4CD02CB9A5F0CF04ABC2A7741BC091D974624FFEE23D8C3F9131B", "abci-version": "", "app-version": ""} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO ABCI Replay Blocks {"module": "consensus", "appHeight": 2723, "storeHeight": 2723, "stateHeight": 2723} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Completed ABCI Handshake - Tendermint and App are synced {"module": "consensus", "appHeight": 2723, "appHash": "BFEB55FAE4E4CD02CB9A5F0CF04ABC2A7741BC091D974624FFEE23D8C3F9131B"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO Version info {"version": "v1.0.0-rc.0"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z INFO This node is a validator {"module": "consensus", "addr": "g1gx03kc927fwj7x365zuctnzjfvrt6nr3ch67rm", "pubKey": "gpub1pggj7ard9eg82cjtv4u52epjx56nzwgjyg9zq7wmjs8s4vdcawahsszf9mqxfua6scn8hsv9nnd4l69v9t5ur65hzse5c2"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO P2P Node ID {"module": "p2p", "ID": "g1y2m5c2nsy3ytza48mp5mr7fvefkd7yafskc4rk", "file": "/var/lib/gnoland/data/secrets/node_key.json"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Adding persistent peers {"module": "p2p", "addrs": []} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting Node {"impl": "Node"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting P2P Switch {"module": "p2p", "impl": "P2P Switch"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting BlockchainReactor {"module": "blockchain", "impl": "BlockchainReactor"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting ConsensusReactor {"module": "consensus", "impl": "ConsensusReactor"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO ConsensusReactor {"module": "consensus", "fastSync": false} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting ConsensusState {"module": "consensus", "impl": "ConsensusState"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z INFO Starting RPC HTTP server on 127.0.0.1:26657 {"module": "rpc-server"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z INFO Starting baseWAL {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "impl": "baseWAL"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z INFO Starting Group {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "impl": "Group"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z INFO Starting TimeoutTicker {"module": "consensus", "impl": "TimeoutTicker"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z INFO Searching for height {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "height": 2725, "min": 0, "max": 1} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z DEBUG Starting timeout routine {"module": "consensus"} May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: panic: should not happen May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: goroutine 1 [running]: May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/wal.(*baseWAL).SearchForHeight(0xc011fd82a0, 0xaa5, 0xc011f8d050) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/bft/wal/wal.go:316 +0xbf4 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusState).catchupReplay(0xc0128ea908, 0xaa4) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/bft/consensus/replay.go:109 +0xcc May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusState).OnStart(0xc0128ea908) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/bft/consensus/state.go:324 +0x2b2 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0128ea908) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/service/service.go:140 +0x22e May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusReactor).OnStart(0xc0150e6600) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/bft/consensus/reactor.go:77 +0x15a May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0150e6600) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/service/service.go:140 +0x22e May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/p2p.(*Switch).OnStart(0xc00a94b500) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/p2p/switch.go:204 +0x9c May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc00a94b500) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/service/service.go:140 +0x22e May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/node.(*Node).OnStart(0xc0078c96c0) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/bft/node/node.go:635 +0x445 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0078c96c0) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/service/service.go:140 +0x22e May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.execStart(0xc0003b8700, {0x11d91f0, 0xc0003a9c70}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/gno.land/cmd/gnoland/start.go:298 +0xa51 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.newStartCmd.func1({0x0?, 0xc00012c020?}, {0x54e6a0?, 0xc000356840?, 0xc00037b3b0?}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/gno.land/cmd/gnoland/start.go:71 +0x1f May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Run(0xc000317ee0?, {0x11ca550?, 0x1923720?}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/commands/command.go:247 +0x19d May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Run(0xc0003b58c0?, {0x11ca550?, 0x1923720?}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/commands/command.go:251 +0x149 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).ParseAndRun(0xc0003b58c0, {0x11ca550, 0x1923720}, {0xc00012c010?, 0xc0003bac60?, 0x2?}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/commands/command.go:132 +0x49 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Execute(0x11d91f0?, {0x11ca550?, 0x1923720?}, {0xc00012c010?, 0x186b778?, 0xc0000061c0?}) May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/tm2/pkg/commands/command.go:114 +0x29 May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.main() May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: /github/workspace/gno.land/cmd/gnoland/root.go:17 +0x5f May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Main process exited, code=exited, status=2/INVALIDARGUMENT May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Failed with result 'exit-code'. May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Consumed 9.921s CPU time. ``` ## 4) implement a transaction metrics processor :::danger **skipped** i haven't yet found something that step 5 requires that it cannot get directly from the [gql endpoint](https://gnoyager1.v8r.io/graphql/query) implemented in [step 2](#2-start-the-transaction-indexer). if i do find something needed by the step 5 implementation that it cannot obtain directly from tx-indexer, i'll revisit step 4. ::: initial look at the tx-indexer suggests that all output is graphql which makes it easy to query chain metrics without further backend processing since tx-indexer is effectively caching chain state at each block. if we were to implement another back-end processor we would likely duplicate everything in tx-indexer and introduce both a frontend delay in any dashboard metrics and a potential point of failure in that (likely unneccessary) extra processor implementation. if i'm wrong about this, it'll likely be because some data needed by the ui in step 5 is unavailable or inelegant to obtain directly from the tx-indexer gql endpoint. in which case, i'll make the effort. if i'm correct that everything necessary can be obtained from tx-indexer, i'll assume this step was part of the exercise in order to check if candidates were paying enough attention to identify shortcuts or optimisation paths. ## 5) implement or configure an extrinsics metrics dashboard :::success **completed** - online and visible at: https://rob.tn/gnoyager1 - source code at: https://github.com/grenade/gnoyager1 (deploy with `yarn deploy`) - dashboard could obviously be prettier and contain more interesting visualisations. i will add these when time permits. ::: :::info **in progress** use react-relay to implement a front-end dashboard to visualise chain and tx info/state. ::: specifically, we can use a relay/graphql subscription to keep track of block height and then relay/graphql queries to show tx stats between any two block numbers. - https://relay.dev/docs/ - https://the-guild.dev/graphql/ws/recipes - https://medium.com/@syedmahmad/956f33739f0 - https://stackoverflow.com/a/42010467/68115 ### approaches i've come at this a few different ways and have yet to find a workable solution. - my first attempt was to use the websocket subscription endpoint to observe the chain block height in near-realtime. i ran into issues with constructing messages in a format the subscriptions endpoint can decipher them. i tried using the observable type from both rxjs and relay-runtime, neither of which constructs the message in a way that is compatible with tx-indexer. - my second attempt was to use the json-rpc endpoints (`newBlockFilter`, `getFilterChanges`). here i ran into issues with decoding the amino encoded responses. [amino-js](https://github.com/cosmos/amino-js) was last updated 5 years ago and has dependencies on a library called `util` which modern webpack does not know how to obtain. sadly all util does is the base64 part of the decode (which can be achieved in modern js with a simple `atob(...)`). the protobuf3 side of the decode is what i really need, but couldn't coerce amino-js to just do that part. at this point i began to wonder why tx-indexer uses amino at all. cosmos had the following [to say](https://docs.cosmos.network/main/learn/advanced/encoding): > Amino having significant performance drawbacks, being reflection-based, and not having any meaningful cross-language/client support... it's being served over the json-rpc endpoint, where its use is surprising since the whole point of json-rpc is interoperability with clients that speak json-rpc. expecting clients to also speak amino would likely be better served by an amino dedicated endpoint so that the json endpoint can serve pure json. i also tried to [use protobufjs](https://protobufjs.github.io/protobuf.js/index.html#using-decorators) to decode the amino messages and this might still work if i can figure out the type definitions for the block headers coming back from the endpoint. - my third attempt uses only the graphql endpoint and successfully fetches all the data required to keep the dashboard in sync. - the query to discover the most recent block is: ```graphql= query { blocks( filter: { from_time: "$thirty_seconds_ago_iso_formatted" } ) { height version chain_id time proposer_address_raw } } ``` - the query to discover recent extrinsics (up to 100 blocks old) is: ```graphql= query { transactions( filter:{ from_block_height: ${Math.max(1, (blockchain.last.block.number - 99))} } ) { block_height hash messages { route, typeUrl value { __typename, ... on BankMsgSend { from_address to_address amount } ... on MsgAddPackage { creator package { name path } } ... on MsgCall { caller pkg_path args } ... on MsgRun { caller package { path } } } } } } ``` - for simplicity, i limit the scope of statistics to the last 100 blocks in order to keep the dashboard responsive. in order to increase the number of blocks being reported on to some amount the user could specify, we'd need to implement some caching of stats for different time frames. my current efforts on the dashboard are in the repository at: https://github.com/grenade/gnoyager1 there is a deployment (to github pages) of this dashboard at: https://rob.tn/gnoyager1 i am running the `dev` genesis on a single node. as such my handling of crashes is quite different to how they would be handled on mainnet or testnet. specifically, when either the tx-indexer or gnoland process crashes, i simply delete the data directories for each and restart the services which results in a new blockchain. for mainnet or testnet, we might do the same but add a step to download a snapshot of the data directory before restarting in order to realise quicker resync with peers.