---
# System prepended metadata

title: gnoyager 1

---

![](https://rob.tn/gnoyager1/drifting-through-the-cosmos.png "drifting through the cosmos" =720x)

# gnoyager 1

## 1) start a gnoland node

:::success
**completed**
- deployment of (and updates to) gnoland binary and systemd service is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1
- gnoland node runs as a systemd service on gnoyager1.v8r.io
:::

### notes
- there is currently no publicly visible/downloadable gnoland node binary. this means it is difficult to ascertain the causes of issues or problems experienced by the wider gnoland community when things go wrong. ie:
  - what version of the source code was used for the binary build when the issue occured?
  - what os or environment configuration was used in the deployment environment that may have contributed to the issue?

  we must fix this first, since we want to create repeatable environments for debugging issues in order to be able to use consistent language and instructions for solving problems. ie:
  > if you experience *x* when running *y* version `1.0.0`, upgrade to version `1.0.1` where issue *x* was resolved by commit *z*.

  in order to reduce friction for the node-running community, we must provide static binary downloads for at least the x86_64 linux os/arch target which is what the majority of node runners will be familiar with.

  while it is perfectly acceptable to expect the developer/source-code-contributor community to maintain a local golang development environment to facilitate a binary build, this is not the case for the node-runner community who are better served by access to pre-built, versioned, binary releases. node-runners maintain nodes for multiple projects and expecting them to maintain a development environment for the idiosyncracies of each project they run nodes for, creates an unnecessary barrier to entry and complicates adoption.

### implementation

we will resolve this step with two distinct implementations. 

1. resolve the missing binary release with a github action to build and publish it
2. automate deployment of the node binary to, and maintenance of, its daemon on a dedicated instance

#### node binary releases

convention dictates that releases are maintained within and closely coupled with the source control mechanism. since gnoland's official repository is on github and it appears there is already a *release* creation mechanism, it makes good sense to implement the *binary publish* mechanism as a new github action which is triggered by the creation of a *release*. ie:

`.github/workflows/binary-publish.yml`
```yaml=
---
on:
  release:
    types:
      - created

permissions:
  contents: write
  packages: write

jobs:
  publish-binaries:
    name: publish binaries
    strategy:
      fail-fast: false
      matrix:
        goversion:
          - "1.22"
        goarch:
          - "386"
          - amd64
          - arm64
        goos:
          - darwin
          - linux
          - windows
        program:
          - gnokey
          - gnoland
          - gnoweb
        exclude:
          -
            goarch: "386"
            goos: darwin
          -
            goarch: arm64
            goos: windows
    runs-on: ubuntu-latest
    steps:
      -
        uses: actions/checkout@v4
      -
        uses: actions/setup-go@v5
        with:
          go-version: ${{ matrix.goversion }}.x
      -
        uses: wangyoucao577/go-release-action@v1
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          goos: ${{ matrix.goos }}
          goarch: ${{ matrix.goarch }}
          goversion: ${{ matrix.goversion }}
          project_path: ./gno.land/cmd/${{ matrix.program }}
          binary_name: ${{ matrix.program }}
          md5sum: true
          sha256sum: true
```
the github action above could be improved by

- creating one `gno-$version-$os-$arch.tar.gz` release artifact for each target, containing all three binaries and the respective archive checksums, but since that isn't supported by the community action used, i opted to leave that for a future implementation when i can give it more of my attention
- enabling a tag filter which only creates binary releases when the release tag matches a given pattern, ie: tags that contain only the first three semver components (`$major.$minor.$patch`) without an aditional label following a hyphen (`$major.$minor.$patch-$label`). this would require a little discussion with the team about the preferred tag naming conventions
- integrate testing into the action workflow and prevent binary publish when any unit or integration tests fail after the binary is built
- the dependency on a community github action (`wangyoucao577/go-release-action@v1`) is suboptimal. it introduces a vulnerability whereby our binary build process could be exploited if/when that gh action's source code is compromised. the exploit could potentially compromise some or all of our nodes, if or when they are updated to a future release, if the 3rd party gh action were to inject malicious instructions during its build process. since it is possible to create a new gh action whose source code is under our control, it would be safer to do just that, time permitting

in the interest of demonstrating a solution on a repo where i have push permissions, the above action is enabled on a gno repo fork here: https://github.com/grenade/gno/blob/master/.github/workflows/binary-publish.yml.

a successful binary publish action run, after a little trial and error, can be seen here: https://github.com/grenade/gno/actions/runs/9050588857

the resulting binary releases are automatically published by the action here: https://github.com/grenade/gno/releases/tag/v1.0.5

#### node deployment orchestration

we can automate deployment using a number of different mechanisms (terraform, ansible, puppet, chef, docker, etc), however for the purpose of this excercise, we'd like to keep things simple and require no dependency or orchestration agent installations on the target instance.

we could use simple bash commands and tools (like ssh, rsync and systemd) that are already installed on the vanilla linux installations  most frequently utilised by node-runners (like ubuntu and fedora). under the hood, many of the orchestration mechanisms mentioned earlier are just going to use those same tools after their respective token replacement and scripting abstractions have run.

almost every orchestration platform supports some sort of scripting engine that does things like token replacement in manifest files. this allows for reducing duplication when, for example, two instances both need the same software installed. this has the unfortunate consequence that manifests quickly become unreadable because they refer to tokens whose runtime values need to be looked up elsewhere.

fortunately, whilst working at mozilla, i authored an orchestration tool called *rubberneck*, which ideologically borrows a lot from and then extends [cloud-init](https://cloud-init.io) to work well on non-cloud deployments. although by design, it does not support token replacement in manifests. this intentional "limitation" ensures that an instance manifest contains all of the deployment and configuration required by the described instance. rubberneck is intentionally verbose and where two or more instances have identical configuration, there are always going to be two or more duplicate identical manifest configurations. creating manifests can be a little tedious because of this enforced duplication, but where this idea begins to make sense is when things go wrong and someone who isn't familiar with the orchestration manifest templating language, needs to debug configuration which has been deployed to an instance. because rubberneck manifests are written in yaml and contain no replaceable tokens, anyone with limited or no orchestration experience can still understand the intended instance configuration by reading a single manifest file written in yaml (and possibly an associated configuration file under a subdirectory in the same folder as the manifest).

another benefit of this approach is that it enables throttled rollouts of changes to large numbers of instances. it is possible to make changes to any particular percentage of an infra landscape and increase that percentage only after confidence around a particular rollout or upgrade is high.

rubberneck relies heavily on source control to manage change on deployment targets. because of this, the timing of issues or failures makes it easy to associate problems with changes to instance manifests in source control.

##### the deployment platform

are we deploying to a big cloud? a bare-metal provider? our own datacentre? our own pc? rubberneck doesn't care. it just needs for the target instance to grant it access by having a local sudo-enabled user whose authorised keys includes the correct rubberneck public key and for that instance to be running an ssh server on a known port.

let's assume we already have (i do, since i run a small, bare-metal, datacenter with a couple of racks of spare capacity):

- a web accessible, fedora or ubuntu instance
- with an active and correctly resolving, dns record named `gnoyager1.v8r.io`
- running an ssh server on an accessible port
- with a maintainer user which has `sudo` rights that can be used to install software and configuration
- an authorised keys file on the instance, under the home path of the maintainer user: `~/.ssh/authorized_keys` which contains a public key for which we hold a corresponding private key, so that we can access the instance with the maintainer user's sudo privileges

let's assume we'd like to have the gnoland node running:
- as a systemd service, so that it recovers and restarts after system reboots and/or crashes
- running under a dedicated service user account named `gnoland` with enough rights to run the node and not much else, since that's good practice
- using (os) conventional paths for:
  - node binary: `/usr/local/bin/gnoland`
  - unit file: `/etc/systemd/system/gnoland.service`
  - gnoland $HOME: `/var/lib/gnoland`

##### rubberneck configuration

- instance manifest
  `manifest/gno/gnoyager1/manifest.yml`
  ```yaml=
  ---
  hostname: gnoyager1
  domain: v8r.io
  action: sync
  ssh:
    port: 2212
  os:
    name: fedora
  package:
    - git
    - cockpit-pcp
    - curl
    - dnf-automatic
  user:
    -
      username: grenade
      authorized:
        keys:
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPRO2rPB5URKyMSGeMwFd9Npzl/XywJWO9F2N/xylCVm
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPb24HEQ++aNFWaqVyMtIs6GotUB8R+q61XOoI2z6uMj
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID83JhRg/lgInWM/XwMfjaDzTMDPS5M7zuVeOm0O5Y5W
          - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGILqCEkUGPF3c+riHfLtkpsSP3nw/HjQUvTs66JsZ6a
  command:
    - sudo passwd -l root
    - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC
    - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter )
    - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland )
  file:
    -
      source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service
      target: /etc/systemd/system/prometheus-node-exporter.service
      command:
        pre:
          - systemctl is-active --quiet prometheus-node-exporter.service && sudo systemctl stop prometheus-node-exporter.service
        post:
          - sudo systemctl daemon-reload
          - sudo systemctl enable prometheus-node-exporter.service
          - sudo systemctl start prometheus-node-exporter.service
    -
      source: https://raw.githubusercontent.com/gnolang/gno/7d44813eece8d8968408b4978ba86a536ea0d75d/gno.land/genesis/genesis_balances.txt
      target: /var/lib/gnoland/genesis/genesis_balances.txt
      command:
        pre:
          - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service
          - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland
          - sudo -u gnoland mkdir -p /var/lib/gnoland/genesis
        post:
          - sudo chown -R gnoland:gnoland /var/lib/gnoland/genesis/genesis_balances.txt
          - sudo systemctl start gnoland.service
    -
      source: https://raw.githubusercontent.com/gnolang/gno/7d44813eece8d8968408b4978ba86a536ea0d75d/gno.land/genesis/genesis_txs.jsonl
      target: /var/lib/gnoland/genesis/genesis_txs.jsonl
      command:
        pre:
          - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service
          - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland
          - sudo -u gnoland mkdir -p /var/lib/gnoland/genesis
        post:
          - sudo chown -R gnoland:gnoland /var/lib/gnoland/genesis/genesis_txs.jsonl
          - sudo systemctl start gnoland.service
    -
      source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/gnoland.service
      target: /etc/systemd/system/gnoland.service
      command:
        pre:
          - systemctl is-active --quiet gnoland.service && sudo systemctl stop gnoland.service
          - id gnoland || sudo useradd --system --create-home --home-dir /var/lib/gnoland --user-group gnoland
          - sudo -u gnoland mkdir -p /var/lib/gnoland/data
          - test -d /var/lib/gnoland/gno || sudo -u gnoland git clone https://github.com/gnolang/gno.git /var/lib/gnoland/gno
          - test -d /var/lib/gnoland/gno && sudo -u gnoland git --git-dir /var/lib/gnoland/gno/.git --work-tree /var/lib/gnoland/gno pull
          - test -d /github || sudo mkdir /github
          - test -L /github/workspace || sudo ln -sf /var/lib/gnoland/gno /github/workspace
        post:
          - sudo systemctl daemon-reload
          - sudo systemctl enable gnoland.service
          - sudo systemctl start gnoland.service
  ```
- node unit file
  `manifest/gno/gnoyager1/etc/systemd/system/gnoland.service`
  ```ini=
  [Unit]
  Description=gnoland blockchain node
  Wants=network-online.target
  After=network-online.target
  ConditionPathExists=/var/lib/gnoland/data
  ConditionPathExists=/var/lib/gnoland/gno
  ConditionPathExists=/var/lib/gnoland/genesis/genesis_balances.txt
  ConditionPathExists=/var/lib/gnoland/genesis/genesis_txs.jsonl
  ConditionPathExists=/usr/local/bin/gnoland
  ConditionPathExists=/github/workspace

  [Service]
  Type=simple
  User=gnoland
  Group=gnoland
  WorkingDirectory=/var/lib/gnoland
  ExecStart=/usr/local/bin/gnoland \
      start \
      -data-dir /var/lib/gnoland/data \
      -gnoroot-dir /var/lib/gnoland/gno \
      -genesis /var/lib/gnoland/genesis.json \
      -genesis-balances-file /var/lib/gnoland/genesis/genesis_balances.txt \
      -genesis-txs-file /var/lib/gnoland/genesis/genesis_txs.jsonl
  Restart=always
  RestartSec=3

  [Install]
  WantedBy=multi-user.target

  ```

##### observations

`gnoland start` fails to run if:
- `genesis_balances.txt` or `genesis_txs.jsonl` is missing. this is reasonable but we ought to document specifically for node runners where to obtain these files for the specific blockchain they want to run (dev, testnet, mainnet)
- `/github/workspace` does not exist. this should be corrected. even when running the binary with `-gnoroot-dir /some/path` the process will fail with inelegant logging which indicates that `/github/workspace` does not exist. it isn't all that reasonable to expect this specific path to exist on everyone's instance. it's easily worked around with symlinking or similar but creating a new specific path convention and expecting everyone to adhere to it feels sloppy.
- the gnoland binary expects to have a copy of the source repo on the filesystem. this is really unusual and for that reason alone feels like we should try to remove this dependency. i don't know the history of why this dependency exists but it creates the impression that it was just done to get something working quickly without considering that non-developers (ie: node runners) are often the type of people who won't bother running your nodes if they have to learn to keep repos synced. it adds another entry barrier to community adoption that feels unneccessary.

## 2) start the transaction indexer

:::success
**completed**
- deployment of (and updates to) tx-indexer binary and systemd service is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1
- tx-indexer runs as a systemd service on gnoyager1.v8r.io
- tx-indexer [playground](https://gnoyager1.v8r.io/graphql) and [endpoint](https://gnoyager1.v8r.io/graphql/query) are served over tls
:::

### rubberneck configuration

- update instance manifest to install `tx-indexer` binary and `tx-indexer.service` systemd unit file
  `git diff 57c9b78e98..57dedaf1e9 manifest/gno/gnoyager1/manifest.yml`
```diff
diff --git a/manifest/gno/gnoyager1/manifest.yml b/manifest/gno/gnoyager1/manifest.yml
index af860e447d..31a3ccc8a6 100644
--- a/manifest/gno/gnoyager1/manifest.yml
+++ b/manifest/gno/gnoyager1/manifest.yml
@@ -25,7 +25,7 @@ command:
   - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC
   - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter )
   - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland )
+  - test -x /usr/local/bin/tx-indexer || ( curl -sLo /tmp/tx-indexer_0.3.0_linux_amd64 https://github.com/gnolang/tx-indexer/releases/download/v0.3.0/tx-indexer_0.3.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/tx-indexer_0.3.0_linux_amd64 -C /usr/local/bin tx-indexer )
 file:
   -
     source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service
@@ -75,3 +75,15 @@ file:
         - sudo systemctl daemon-reload
       - sudo systemctl enable gnoland.service
       - sudo systemctl start gnoland.service
+  -
+    source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/tx-indexer.service
+    target: /etc/systemd/system/tx-indexer.service
+    command:
+      pre:
+        - systemctl is-active --quiet tx-indexer.service && sudo systemctl stop tx-indexer.service
+        - id tx-indexer || sudo useradd --system --create-home --home-dir /var/lib/tx-indexer --user-group tx-indexer
+        - sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data
+      post:
+        - sudo systemctl daemon-reload
+        - sudo systemctl enable tx-indexer.service
+        - sudo systemctl start tx-indexer.service
```
- tx-indexer unit file
  `manifest/gno/gnoyager1/etc/systemd/system/tx-indexer.service`
  ```ini=
  [Unit]
  Description=gnoland transaction indexer
  Wants=network-online.target
  After=network-online.target
  ConditionPathExists=/var/lib/tx-indexer/data
  ConditionPathExists=/usr/local/bin/tx-indexer

  [Service]
  Type=simple
  User=tx-indexer
  Group=tx-indexer
  WorkingDirectory=/var/lib/tx-indexer
  ExecStart=/usr/local/bin/tx-indexer \
      start \
      -db-path /var/lib/tx-indexer/data \
      -listen-address 0.0.0.0:8546 \
      -remote http://127.0.0.1:26657 \
      -log-level info \
      -http-rate-limit 0 \
      -max-chunk-size 100 \
      -max-slots 100
  Restart=always
  RestartSec=3

  [Install]
  WantedBy=multi-user.target
  ```

#### observations

tx-indexer already provides a binary release so we make use of that and just add a systemd service unit file to manage its daemon.

everything works as expected without any surprises.

since the tx-indexer deployment sits on a private network but i want to be able to reach it from the web, i had a little extra configuration to reverse proxy requests over tls. i achieved this by [modifying nginx config](https://github.com/grenade/rubberneck/compare/36924625a3..60e4830076) in rubberneck for the gateway instance which exposes the tx-indexer endpoints at: https://gnoyager1.v8r.io (ie: https://gnoyager1.v8r.io/graphql).

#### crash handling

tx-indexer occassionaly crashes with journal entries like so:

```text=
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024/05/17 06:03:13 [JOB 1] WAL file /var/lib/tx-indexer/data/001002.log with log number 001002 stopped reading at offset: 0; replayed 0 keys in 0 batches
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.552Z        INFO        http-server        serve/server.go:46        HTTP server started        {"address": "[::]:8546"}
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z        INFO        fetcher        fetch/fetch.go:111        Fetching range        {"from": 13409, "to": 13508}
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z        INFO        fetcher        fetch/fetch.go:111        Fetching range        {"from": 13509, "to": 13608}
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: 2024-05-17T06:03:13.553Z        INFO        fetcher        fetch/fetch.go:111        Fetching range        {"from": 13609, "to": 13688}
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: panic: runtime error: index out of range [1] with length 1
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: goroutine 48 [running]:
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.getTxResultFromBatch({0xc000545800?, 0x50, 0x80}, {0x12a3790?, 0xc000486090?})
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]:         /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:181 +0x553
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.handleChunk.func1()
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]:         /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:38 +0xf3
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: github.com/gnolang/tx-indexer/fetch.handleChunk({0x12a3838, 0xc00019e000}, {0x12a3790?, 0xc000486090?}, 0xc00011ce58)
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]:         /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/worker.go:47 +0x8d
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]: created by github.com/gnolang/tx-indexer/fetch.(*Fetcher).FetchChainData.func1 in goroutine 83
May 17 06:03:13 gnoyager1.v8r.io tx-indexer[1815505]:         /home/runner/go/pkg/mod/github.com/gnolang/tx-indexer@v0.3.0/fetch/fetch.go:123 +0x2dd
May 17 06:03:13 gnoyager1.v8r.io systemd[1]: tx-indexer.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 17 06:03:13 gnoyager1.v8r.io systemd[1]: tx-indexer.service: Failed with result 'exit-code'
```

in order to maintain progress on step 5, i simply delete its data directory and restart.

```bash=
sudo systemctl stop tx-indexer.service
sudo -u tx-indexer rm -rf /var/lib/tx-indexer/data
sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data
sudo systemctl start tx-indexer.service
```

## 3) start supernova stress testers

:::success
**completed**
- deployment of (and updates to) supernova binary and 3 systemd services (realm call, realm deploy, package deploy) is automated by rubberneck config at: https://github.com/grenade/rubberneck/tree/main/manifest/gno/gnoyager1
- each of the three supernova modes runs as a systemd service on gnoyager1.v8r.io
:::

### rubberneck configuration

- update instance manifest
```bash=
git diff 57dedaf..36924625a3 manifest/gno/gnoyager1/manifest.yml
```
```diff=
diff --git a/manifest/gno/gnoyager1/manifest.yml b/manifest/gno/gnoyager1/manifest.yml
index 31a3ccc8a6..33be36fa9d 100644
--- a/manifest/gno/gnoyager1/manifest.yml
+++ b/manifest/gno/gnoyager1/manifest.yml
@@ -25,7 +25,8 @@ command:
   - timedatectl show | grep Timezone=UTC &> /dev/null || sudo timedatectl set-timezone UTC
   - test -x /usr/local/bin/node_exporter || ( curl -sLo /tmp/node_exporter-1.8.0.linux-amd64.tar.gz https://github.com/prometheus/node_exporter/releases/download/v1.8.0/node_exporter-1.8.0.linux-amd64.tar.gz && sudo tar xvfz /tmp/node_exporter-1.8.0.linux-amd64.tar.gz -C /usr/local/bin --strip-components=1 node_exporter-1.8.0.linux-amd64/node_exporter )
   - test -x /usr/local/bin/gnoland || ( curl -sLo /tmp/gnoland-v1.0.5-linux-amd64.tar.gz https://github.com/grenade/gno/releases/download/v1.0.5/gnoland-v1.0.5-linux-amd64.tar.gz && sudo tar xvfz /tmp/gnoland-v1.0.5-linux-amd64.tar.gz -C /usr/local/bin gnoland )
  - test -x /usr/local/bin/tx-indexer || ( curl -sLo /tmp/tx-indexer_0.3.0_linux_amd64.tar.gz https://github.com/gnolang/tx-indexer/releases/download/v0.3.0/tx-indexer_0.3.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/tx-indexer_0.3.0_linux_amd64.tar.gz -C /usr/local/bin tx-indexer )
+  - test -x /usr/local/bin/supernova || ( curl -sLo /tmp/supernova_1.2.0_linux_amd64.tar.gz https://github.com/gnolang/supernova/releases/download/v1.2.0/supernova_1.2.0_linux_amd64.tar.gz && sudo tar xvfz /tmp/supernova_1.2.0_linux_amd64.tar.gz -C /usr/local/bin supernova )
 file:
   -
     source: https://raw.githubusercontent.com/grenade/rubberneck/main/static/etc/systemd/system/prometheus-node-exporter.service
@@ -87,3 +88,53 @@ file:
         - sudo systemctl daemon-reload
         - sudo systemctl enable tx-indexer.service
         - sudo systemctl start tx-indexer.service
+  -
+    source: https://raw.githubusercontent.com/gnolang/supernova/main/scripts/p/package.gno
+    target: /var/lib/supernova/scripts/p/package.gno
+    command:
+      pre:
+        - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova
+        - sudo -u supernova mkdir -p /var/lib/supernova/scripts/p
+  -
+    source: https://raw.githubusercontent.com/gnolang/supernova/main/scripts/r/realm.gno
+    target: /var/lib/supernova/scripts/r/realm.gno
+    command:
+      pre:
+        - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova
+        - sudo -u supernova mkdir -p /var/lib/supernova/scripts/r
+  -
+    source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-call.service
+    target: /etc/systemd/system/supernova-realm-call.service
+    command:
+      pre:
+        - systemctl is-active --quiet supernova-realm-call.service && sudo systemctl stop supernova-realm-call.service
+        - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova
+        - sudo -u supernova mkdir -p /var/lib/supernova/data
+      post:
+        - sudo systemctl daemon-reload
+        - sudo systemctl enable supernova-realm-call.service
+        - sudo systemctl start supernova-realm-call.service
+  -
+    source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-deployment.service
+    target: /etc/systemd/system/supernova-realm-deployment.service
+    command:
+      pre:
+        - systemctl is-active --quiet supernova-realm-deployment.service && sudo systemctl stop supernova-realm-deployment.service
+        - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova
+        - sudo -u supernova mkdir -p /var/lib/supernova/data
+      post:
+        - sudo systemctl daemon-reload
+        - sudo systemctl enable supernova-realm-deployment.service
+        - sudo systemctl start supernova-realm-deployment.service
+  -
+    source: https://raw.githubusercontent.com/grenade/rubberneck/main/manifest/gno/gnoyager1/etc/systemd/system/supernova-package-deployment.service
+    target: /etc/systemd/system/supernova-package-deployment.service
+    command:
+      pre:
+        - systemctl is-active --quiet supernova-package-deployment.service && sudo systemctl stop supernova-package-deployment.service
+        - id supernova || sudo useradd --system --create-home --home-dir /var/lib/supernova --user-group supernova
+        - sudo -u supernova mkdir -p /var/lib/supernova/data
+      post:
+        - sudo systemctl daemon-reload
+        - sudo systemctl enable supernova-package-deployment.service
+        - sudo systemctl start supernova-package-deployment.service
```
- supernova unit files
  - `manifest/gno/gnoyager1/etc/systemd/system/supernova-package-deployment.service`
    ```ini=
    [Unit]
    Description=supernova gnoland stress test package deployment
    Wants=network-online.target
    After=network-online.target
    ConditionPathExists=/var/lib/supernova/data
    ConditionPathExists=/usr/local/bin/supernova

    [Service]
    Type=simple
    User=supernova
    Group=supernova
    WorkingDirectory=/var/lib/supernova
    ExecStart=/usr/local/bin/supernova \
        -batch 20 \
        -chain-id dev \
        -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \
        -mode PACKAGE_DEPLOYMENT \
        -output /var/lib/supernova/data/package-deployment.json \
        -sub-accounts 10 \
        -transactions 100 \
        -url http://127.0.0.1:26657
    Restart=always
    RestartSec=300

    [Install]
    WantedBy=multi-user.target
    ```
  - `manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-call.service`
    ```ini=
    [Unit]
    Description=supernova gnoland stress test realm call
    Wants=network-online.target
    After=network-online.target
    ConditionPathExists=/var/lib/supernova/data
    ConditionPathExists=/usr/local/bin/supernova

    [Service]
    Type=simple
    User=supernova
    Group=supernova
    WorkingDirectory=/var/lib/supernova
    ExecStart=/usr/local/bin/supernova \
        -batch 20 \
        -chain-id dev \
        -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \
        -mode REALM_CALL \
        -output /var/lib/supernova/data/realm-call.json \
        -sub-accounts 10 \
        -transactions 100 \
        -url http://127.0.0.1:26657
    Restart=always
    RestartSec=300

    [Install]
    WantedBy=multi-user.target

    ```
  - `manifest/gno/gnoyager1/etc/systemd/system/supernova-realm-deployment.service`
    ```ini=
    [Unit]
    Description=supernova gnoland stress test realm deployment
    Wants=network-online.target
    After=network-online.target
    ConditionPathExists=/var/lib/supernova/data
    ConditionPathExists=/usr/local/bin/supernova

    [Service]
    Type=simple
    User=supernova
    Group=supernova
    WorkingDirectory=/var/lib/supernova
    ExecStart=/usr/local/bin/supernova \
        -batch 20 \
        -chain-id dev \
        -mnemonic "source bonus chronic canvas draft south burst lottery vacant surface solve popular case indicate oppose farm nothing bullet exhibit title speed wink action roast" \
        -mode REALM_DEPLOYMENT \
        -output /var/lib/supernova/data/realm-deployment.json \
        -sub-accounts 10 \
        -transactions 100 \
        -url http://127.0.0.1:26657
    Restart=always
    RestartSec=300

    [Install]
    WantedBy=multi-user.target
    ```

#### observations

supernova makes use of a couple of directories (`./scripts/p`, `./scripts/r`) which aren't mentioned in setup documentation so the first time a new user hears about them is when supernova errors because it can't find them. we can improve the user experience by describing these directories and what needs to be in them in the getting-started/setup docs or by implementing a setup/install script that creates the required folders and their contents when they don't exist (ie: when the user has installed the release binary, rather than building from source).

i opted to create three separate services for triggering each of the three supernova modes.

## handling a borked (dev) chain state
:::warning
**restart fresh dev chain**
:::
the dev chain seems to crash ~~consistently~~ sometimes around block height 2723. i haven't yet been able to determine why it does so and currently use the following script to reset the chain to the genesis block so that i can continue making progress on the step 5 dashboard.
```bash=
sudo ls -ahl /var/lib/gnoland/data
sudo ls -ahl /var/lib/tx-indexer/data

sudo systemctl stop supernova-realm-deployment.service
sudo systemctl stop supernova-realm-call.service
sudo systemctl stop supernova-package-deployment.service
sudo systemctl stop tx-indexer.service
sudo systemctl stop gnoland.service

sudo -u gnoland rm -rf /var/lib/gnoland/data
sudo -u gnoland rm /var/lib/gnoland/genesis/genesis.json
sudo -u tx-indexer rm -rf /var/lib/tx-indexer/data

sudo -u gnoland mkdir -p /var/lib/gnoland/data
sudo -u tx-indexer mkdir -p /var/lib/tx-indexer/data

sudo systemctl start gnoland.service
sudo systemctl start tx-indexer.service

sleep 10

sudo systemctl start supernova-package-deployment.service
sudo systemctl start supernova-realm-call.service
sudo systemctl start supernova-realm-deployment.service

sudo ls -ahl /var/lib/gnoland/data
sudo ls -ahl /var/lib/tx-indexer/data
```

the journalctl output from a borked chain looks like so:

```log=
May 16 09:12:40 gnoyager1.v8r.io systemd[1]: Started gnoland.service - gnoland blockchain node.
May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]:                     __             __
May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]:   ___ ____  ___    / /__ ____  ___/ /
May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]:  / _ `/ _ \/ _ \_ / / _ `/ _ \/ _  /
May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]:  \_, /_//_/\___(_)_/\_,_/_//_/\_,_/
May 16 09:12:44 gnoyager1.v8r.io gnoland[1121544]: /___/
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Starting multi        {"module": "proxy", "impl": "multi"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Starting localClient        {"module": "proxy", "module": "abci-client", "connection": "query", "impl": "localClient"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Starting localClient        {"module": "proxy", "module": "abci-client", "connection": "mempool", "impl": "localClient"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Starting localClient        {"module": "proxy", "module": "abci-client", "connection": "consensus", "impl": "localClient"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Starting EventStoreService        {"module": "eventstore", "impl": "EventStoreService"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         ABCI Handshake App Info        {"module": "consensus", "height": 2723, "hash": "BFEB55FAE4E4CD02CB9A5F0CF04ABC2A7741BC091D974624FFEE23D8C3F9131B", "abci-version": "", "app-version": ""}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         ABCI Replay Blocks        {"module": "consensus", "appHeight": 2723, "storeHeight": 2723, "stateHeight": 2723}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Completed ABCI Handshake - Tendermint and App are synced        {"module": "consensus", "appHeight": 2723, "appHash": "BFEB55FAE4E4CD02CB9A5F0CF04ABC2A7741BC091D974624FFEE23D8C3F9131B"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         Version info        {"version": "v1.0.0-rc.0"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.220Z        INFO         This node is a validator        {"module": "consensus", "addr": "g1gx03kc927fwj7x365zuctnzjfvrt6nr3ch67rm", "pubKey": "gpub1pggj7ard9eg82cjtv4u52epjx56nzwgjyg9zq7wmjs8s4vdcawahsszf9mqxfua6scn8hsv9nnd4l69v9t5ur65hzse5c2"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         P2P Node ID        {"module": "p2p", "ID": "g1y2m5c2nsy3ytza48mp5mr7fvefkd7yafskc4rk", "file": "/var/lib/gnoland/data/secrets/node_key.json"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Adding persistent peers        {"module": "p2p", "addrs": []}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting Node        {"impl": "Node"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting P2P Switch        {"module": "p2p", "impl": "P2P Switch"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting BlockchainReactor        {"module": "blockchain", "impl": "BlockchainReactor"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting ConsensusReactor        {"module": "consensus", "impl": "ConsensusReactor"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         ConsensusReactor         {"module": "consensus", "fastSync": false}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting ConsensusState        {"module": "consensus", "impl": "ConsensusState"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.222Z        INFO         Starting RPC HTTP server on 127.0.0.1:26657        {"module": "rpc-server"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z        INFO         Starting baseWAL        {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "impl": "baseWAL"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z        INFO         Starting Group        {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "impl": "Group"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z        INFO         Starting TimeoutTicker        {"module": "consensus", "impl": "TimeoutTicker"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z        INFO         Searching for height        {"module": "consensus", "wal": "/var/lib/gnoland/data/wal/cs.wal/wal", "height": 2725, "min": 0, "max": 1}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: 2024-05-16T09:12:45.223Z        DEBUG        Starting timeout routine        {"module": "consensus"}
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: panic: should not happen
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: goroutine 1 [running]:
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/wal.(*baseWAL).SearchForHeight(0xc011fd82a0, 0xaa5, 0xc011f8d050)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/bft/wal/wal.go:316 +0xbf4
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusState).catchupReplay(0xc0128ea908, 0xaa4)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/bft/consensus/replay.go:109 +0xcc
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusState).OnStart(0xc0128ea908)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/bft/consensus/state.go:324 +0x2b2
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0128ea908)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/service/service.go:140 +0x22e
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/consensus.(*ConsensusReactor).OnStart(0xc0150e6600)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/bft/consensus/reactor.go:77 +0x15a
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0150e6600)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/service/service.go:140 +0x22e
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/p2p.(*Switch).OnStart(0xc00a94b500)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/p2p/switch.go:204 +0x9c
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc00a94b500)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/service/service.go:140 +0x22e
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/bft/node.(*Node).OnStart(0xc0078c96c0)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/bft/node/node.go:635 +0x445
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/service.(*BaseService).Start(0xc0078c96c0)
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/service/service.go:140 +0x22e
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.execStart(0xc0003b8700, {0x11d91f0, 0xc0003a9c70})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/gno.land/cmd/gnoland/start.go:298 +0xa51
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.newStartCmd.func1({0x0?, 0xc00012c020?}, {0x54e6a0?, 0xc000356840?, 0xc00037b3b0?})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/gno.land/cmd/gnoland/start.go:71 +0x1f
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Run(0xc000317ee0?, {0x11ca550?, 0x1923720?})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/commands/command.go:247 +0x19d
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Run(0xc0003b58c0?, {0x11ca550?, 0x1923720?})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/commands/command.go:251 +0x149
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).ParseAndRun(0xc0003b58c0, {0x11ca550, 0x1923720}, {0xc00012c010?, 0xc0003bac60?, 0x2?})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/commands/command.go:132 +0x49
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: github.com/gnolang/gno/tm2/pkg/commands.(*Command).Execute(0x11d91f0?, {0x11ca550?, 0x1923720?}, {0xc00012c010?, 0x186b778?, 0xc0000061c0?})
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/tm2/pkg/commands/command.go:114 +0x29
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]: main.main()
May 16 09:12:45 gnoyager1.v8r.io gnoland[1121544]:         /github/workspace/gno.land/cmd/gnoland/root.go:17 +0x5f
May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Failed with result 'exit-code'.
May 16 09:12:45 gnoyager1.v8r.io systemd[1]: gnoland.service: Consumed 9.921s CPU time.
```

## 4) implement a transaction metrics processor

:::danger
**skipped**
i haven't yet found something that step 5 requires that it cannot get directly from the [gql endpoint](https://gnoyager1.v8r.io/graphql/query) implemented in [step 2](#2-start-the-transaction-indexer). if i do find something needed by the step 5 implementation that it cannot obtain directly from tx-indexer, i'll revisit step 4.
:::

initial look at the tx-indexer suggests that all output is graphql which makes it easy to query chain metrics without further backend processing since tx-indexer is effectively caching chain state at each block.

if we were to implement another back-end processor we would likely duplicate everything in tx-indexer and introduce both a frontend delay in any dashboard metrics and a potential point of failure in that (likely unneccessary) extra processor implementation.

if i'm wrong about this, it'll likely be because some data needed by the ui in step 5 is unavailable or inelegant to obtain directly from the tx-indexer gql endpoint. in which case, i'll make the effort.

if i'm correct that everything necessary can be obtained from tx-indexer, i'll assume this step was part of the exercise in order to check if candidates were paying enough attention to identify shortcuts or optimisation paths.

## 5) implement or configure an extrinsics metrics dashboard

:::success
**completed**
- online and visible at: https://rob.tn/gnoyager1
- source code at: https://github.com/grenade/gnoyager1 (deploy with `yarn deploy`)
- dashboard could obviously be prettier and contain more interesting visualisations. i will add these when time permits.
:::

:::info
**in progress**
use react-relay to implement a front-end dashboard to visualise chain and tx info/state.
:::

specifically, we can use a relay/graphql subscription to keep track of block height and then relay/graphql queries to show tx stats between any two block numbers.

- https://relay.dev/docs/
- https://the-guild.dev/graphql/ws/recipes
- https://medium.com/@syedmahmad/956f33739f0
- https://stackoverflow.com/a/42010467/68115

### approaches

i've come at this a few different ways and have yet to find a workable solution.

- my first attempt was to use the websocket subscription endpoint to observe the chain block height in near-realtime. i ran into issues with constructing messages in a format the subscriptions endpoint can decipher them. i tried using the observable type from both rxjs and relay-runtime, neither of which constructs the message in a way that is compatible with tx-indexer.
- my second attempt was to use the json-rpc endpoints (`newBlockFilter`, `getFilterChanges`). here i ran into issues with decoding the amino encoded responses. [amino-js](https://github.com/cosmos/amino-js) was last updated 5 years ago and has dependencies on a library called `util` which modern webpack does not know how to obtain. sadly all util does is the base64 part of the decode (which can be achieved in modern js with a simple `atob(...)`). the protobuf3 side of the decode is what i really need, but couldn't coerce amino-js to just do that part. at this point i began to wonder why tx-indexer uses amino at all. cosmos had the following [to say](https://docs.cosmos.network/main/learn/advanced/encoding):
  > Amino having significant performance drawbacks, being reflection-based, and not having any meaningful cross-language/client support...
  
  it's being served over the json-rpc endpoint, where its use is surprising since the whole point of json-rpc is interoperability with clients that speak json-rpc. expecting clients to also speak amino would likely be better served by an amino dedicated endpoint so that the json endpoint can serve pure json.
  
  i also tried to [use protobufjs](https://protobufjs.github.io/protobuf.js/index.html#using-decorators) to decode the amino messages and this might still work if i can figure out the type definitions for the block headers coming back from the endpoint.
  
- my third attempt uses only the graphql endpoint and successfully fetches all the data required to keep the dashboard in sync.
  - the query to discover the most recent block is:
    ```graphql=
    query {
      blocks(
        filter: {
          from_time: "$thirty_seconds_ago_iso_formatted"
        }
      ) {
        height
        version
        chain_id
        time
        proposer_address_raw
      }
    }
    ```
  - the query to discover recent extrinsics (up to 100 blocks old) is:
    ```graphql=
    query {
      transactions(
        filter:{
          from_block_height: ${Math.max(1, (blockchain.last.block.number - 99))}
        }
      ) {
        block_height
        hash
        messages {
          route,
          typeUrl
          value {
            __typename,
            ... on BankMsgSend {
              from_address
              to_address
              amount
            }
            ... on MsgAddPackage {
              creator
              package {
                name
                path
              }
            }
            ... on MsgCall {
              caller
              pkg_path
              args
            }
            ... on MsgRun {
              caller
              package {
                path
              }
            }
          }
        }
      }
    }
    ```
  - for simplicity, i limit the scope of statistics to the last 100 blocks in order to keep the dashboard responsive. in order to increase the number of blocks being reported on to some amount the user could specify, we'd need to implement some caching of stats for different time frames.
  
my current efforts on the dashboard are in the repository at: https://github.com/grenade/gnoyager1
there is a deployment (to github pages) of this dashboard at: https://rob.tn/gnoyager1

i am running the `dev` genesis on a single node. as such my handling of crashes is quite different to how they would be handled on mainnet or testnet. specifically, when either the tx-indexer or gnoland process crashes, i simply delete the data directories for each and restart the services which results in a new blockchain. for mainnet or testnet, we might do the same but add a step to download a snapshot of the data directory before restarting in order to realise quicker resync with peers.

