---
# System prepended metadata

title: eddie.thgttg.com

---

# eddie.thgttg.com

an ethereum full node built with rubberneck.

`thgttg.com` is a domain referencing douglas adam's well known work. hostnames are chosen from the character list of that plot. in this case, ***e**ddie* shares his initial with ***e**thereum*.

## system spec

hp proliant dl380 g7, 2u rack mount
- cpu: 2 × intel xeon e5620 @ 2.4ghz
- ram: 16gb
- hdd (os): 8 × 146gb, 15k rpm sas, hw raid 1+0 for 512gb total
- ssd (data): a 4tb wd ssd that i had laying around is mounted as the `/data` partition for the blockchain data dirs
- os: fedora server 38 (vanilla en-us)

in hindsight, and with the benefit of the [node dashboard](https://marvin.thgttg.com/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&refresh=10s), it's obvious this hardware is maxed out running geth/prysm. next time i spin up an eth node, it'll be on beefier hardware.

build references:
- https://geth.ethereum.org/docs/getting-started/installing-geth
- https://geth.ethereum.org/docs/monitoring/metrics
- https://docs.prylabs.network/docs/install/install-with-script
- https://docs.prylabs.network/docs/prysm-usage/parameters
- https://docs.prylabs.network/docs/prysm-usage/checkpoint-sync
- https://docs.prylabs.network/docs/troubleshooting/issues-errors

## server and network init

- create a dns entry for eddie.thgttg.com as a cname record pointing to the dynamic dns entry for the rack location
  ```bash
  #!/usr/bin/env bash

  curl \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $(pass cloudflare/thgttg.com/dns-api-key)" \
    --data '{
      "type": "CNAME",
      "name": "eddie.thgttg.com",
      "content": "mp.thgttg.com"
    }' \
    https://api.cloudflare.com/client/v4/zones/80ac5f73f4e147bb0079d96e1497bf07/dns_records
  ```
- manually configure the site router to
  - assign persistent lan ip `192.168.0.194` to `eddie.thgttg.com`
  - port forward
    - inbound tcp on wan port `2204` to lan `192.168.0.194:22`
    - inbound udp on wan port `12000` to lan `192.168.0.194:12000`
    - inbound tcp on wan port `13000` to lan `192.168.0.194:13000`
    - inbound tcp+udp on wan port `30303` to lan `192.168.0.194:30303`
- configure site nginx to:
  - reverse proxy eddie.thgttg.com:80 to 192.168.0.194:80
  - reverse proxy eddie.thgttg.com:443 to 192.168.0.194:443

- enable ssh access on first boot
  ```bash
  #!/usr/bin/env bash
  
  mkdir -p ~/.ssh
  curl -Lo ~/.ssh/authorized_keys https://github.com/grenade.keys
  ```

- set hostname
  ```bash
  #!/usr/bin/env bash

  sudo hostnamectl set-hostname eddie.thgttg.com
  ```

- open firewall ports (handled in rubberneck but documented here because it took a bit of trial and error to get all the ports)
  ```bash
  #!/usr/bin/env bash

  # see: https://docs.prylabs.network/docs/prysm-usage/p2p-host-ip
  for port_proto in {{3500,6060,8545,9100,13000}/tcp,12000/udp,30303/{tcp,udp}}; do
    sudo firewall-cmd --zone=$(sudo firewall-cmd --get-default-zone) --add-port=${port_proto} --permanent
  done
  sudo firewall-cmd --reload
  sudo firewall-cmd --list-all
  ```

- reset chain state. i got errors in the prysm journal (`journalctl -fu prysm.service`), including:
  > Sep 01 08:07:48 eddie.thgttg.com prysm.sh[404877]: could not get ancestor state: slot 7223583 not in db due to checkpoint sync: cannot retrieve data for slot

  which suggests a corrupt chain database which [requires](https://docs.prylabs.network/docs/troubleshooting/issues-errors) clearing out the data directory and restarting the chain.
  ```bash
  #!/usr/bin/env bash

  sudo systemctl stop geth.service
  sudo systemctl stop prysm.service
  sudo rm -rf /data/ethereum/{beaconchaindata,geth,geth.ipc,keystore}
  sudo systemctl start geth.service
  sudo systemctl start prysm.service
  ```
  
- rubberneck automation handles the rest: https://github.com/grenade/rubberneck/compare/7c2f943..d5b4b09
  - geth/prysm user (eddie) and data directory (`/data/ethereum`) creation and permissions
  - geth binary install
  - geth service install, enable and start
  - prysm script/binary install
  - prysm service install, enable and start
  - services keep-alive/hard-restart
  - os security updates

# marvin.thgttg.com
  
a prometheus and grafana server.
  
***m**arvin* shares his initial with ***m**etrics*.

## server and network init

- create a dns entry for marvin.thgttg.com as an A record pointing to the static public ip for the hetzner instance
  ```bash
  #!/usr/bin/env bash
  
  curl \
    --request POST \
    --header 'Content-Type: application/json' \
    --header "Authorization: Bearer $(pass cloudflare/thgttg.com/dns-api-key)" \
    --data '{
      "type": "A",
      "name": "marvin.thgttg.com",
      "content": "95.216.75.94"
    }' \
    https://api.cloudflare.com/client/v4/zones/80ac5f73f4e147bb0079d96e1497bf07/dns_records

  # flush the local dns cache
  sudo resolvectl flush-caches
  ```

- (re)image instance with clean partitions and a vanilla os install
  - set hostname
  - configure software raid
  - partition drives (boot, lvm)
  - create logical volumes on lvm:
    - swap (16gb)
    - / (64gb)
    - /var/lib (512gb)
    - /var/log (32gb)
  - install os
  - reboot
  - remove rescue system host key

  ```bash=
  #!/usr/bin/env bash

  # see https://github.com/hetzneronline/installimage/blob/master/get_options.sh
  fqdn=marvin.thgttg.com
  ssh-keygen -R ${fqdn}
  ssh -o StrictHostKeyChecking=accept-new root@${fqdn} \
    /root/.oldroot/nfs/install/installimage \
      -a \
      -n ${fqdn} \
      -r yes \
      -l 1 \
      -d sda,sdb \
      -p /boot:ext3:1024M,lvm:vg0:all \
      -v vg0:swap:swap:swap:16G,vg0:root:/:ext4:64G,vg0:lib:/var/lib:ext4:512G,vg0:log:/var/log:ext4:32G \
      -i /root/images/CentOS-90-stream-amd64-base.tar.gz \
      -G yes \
      -K https://github.com/grenade.keys
  ssh root@${fqdn} reboot
  ssh-keygen -R ${fqdn}
  sleep 300
  ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=1 root@${fqdn} dnf update -y
  ```

- create superuser
  ```bash=
  #!/usr/bin/env bash

  user=grenade
  fqdn=marvin.thgttg.com
  ssh root@${fqdn} "
    getent passwd ${user} &> /dev/null || useradd --create-home --shell /bin/bash --user-group --groups wheel,systemd-journal ${user};
    test -d /home/${user}/.ssh || sudo -H -u ${user} mkdir /home/${user}/.ssh;
    chmod 700 /home/${user}/.ssh;
    sudo -H -u ${user} bash -c 'curl -Lo /home/${user}/.ssh/authorized_keys https://github.com/grenade.keys';
    chmod 644 /home/${user}/.ssh/authorized_keys;
    echo '${user} ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/${user};
    chmod 0440 /etc/sudoers.d/${user};
  "
  ```
  
- rubberneck automation handles the rest: https://github.com/grenade/rubberneck/compare/d5b4b09..4c404a4
  - prometheus binary install
  - prometheus configuration
  - prometheus service install, enable  and start
  - grafana install
  - grafana configuration
  - grafana service enable  and start
  - nginx install
  - nginx configuration
    - reverse proxy prometheus over tls
    - reverse proxy grafana over tls
  - certbot install
  - obtain tls cert, configured for auto-renewals
  - services keep-alive/hard-restart
  - os security updates

- grafana manual config (in the [grafana ui](https://marvin.thgttg.com/grafana))
  - enable github oauth in grafana config
  - create (local) prometheus datasource
  - import dashboards:
    - [geth overview](https://grafana.com/grafana/dashboards/14053-geth-overview)
    - [node exporter full](https://grafana.com/grafana/dashboards/1860-node-exporter-full)

# outputs from this excercise
- geth/prysm full node on private datacenter lan bare metal with nat and tls proxy for access to public endpoints: eddie.thgttg.com
- prometheus/grafana server on public (hetzner robot) bare metal: marvin.thgttg.com
- grafana dashboards (login with github. you must be a member of a github org shortlist):
  - [geth overview](https://marvin.thgttg.com/grafana/d/FPpjH6Hik/geth-overview?orgId=1&refresh=10s)
  - [node exporter full](https://marvin.thgttg.com/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&var-DS_PROMETHEUS=default&var-job=thgttg.com%20-%20node&var-node=eddie.thgttg.com:443&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B)
