# eddie.thgttg.com
an ethereum full node built with rubberneck.
`thgttg.com` is a domain referencing douglas adam's well known work. hostnames are chosen from the character list of that plot. in this case, ***e**ddie* shares his initial with ***e**thereum*.
## system spec
hp proliant dl380 g7, 2u rack mount
- cpu: 2 × intel xeon e5620 @ 2.4ghz
- ram: 16gb
- hdd (os): 8 × 146gb, 15k rpm sas, hw raid 1+0 for 512gb total
- ssd (data): a 4tb wd ssd that i had laying around is mounted as the `/data` partition for the blockchain data dirs
- os: fedora server 38 (vanilla en-us)
in hindsight, and with the benefit of the [node dashboard](https://marvin.thgttg.com/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&refresh=10s), it's obvious this hardware is maxed out running geth/prysm. next time i spin up an eth node, it'll be on beefier hardware.
build references:
- https://geth.ethereum.org/docs/getting-started/installing-geth
- https://geth.ethereum.org/docs/monitoring/metrics
- https://docs.prylabs.network/docs/install/install-with-script
- https://docs.prylabs.network/docs/prysm-usage/parameters
- https://docs.prylabs.network/docs/prysm-usage/checkpoint-sync
- https://docs.prylabs.network/docs/troubleshooting/issues-errors
## server and network init
- create a dns entry for eddie.thgttg.com as a cname record pointing to the dynamic dns entry for the rack location
```bash
#!/usr/bin/env bash
curl \
--request POST \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $(pass cloudflare/thgttg.com/dns-api-key)" \
--data '{
"type": "CNAME",
"name": "eddie.thgttg.com",
"content": "mp.thgttg.com"
}' \
https://api.cloudflare.com/client/v4/zones/80ac5f73f4e147bb0079d96e1497bf07/dns_records
```
- manually configure the site router to
- assign persistent lan ip `192.168.0.194` to `eddie.thgttg.com`
- port forward
- inbound tcp on wan port `2204` to lan `192.168.0.194:22`
- inbound udp on wan port `12000` to lan `192.168.0.194:12000`
- inbound tcp on wan port `13000` to lan `192.168.0.194:13000`
- inbound tcp+udp on wan port `30303` to lan `192.168.0.194:30303`
- configure site nginx to:
- reverse proxy eddie.thgttg.com:80 to 192.168.0.194:80
- reverse proxy eddie.thgttg.com:443 to 192.168.0.194:443
- enable ssh access on first boot
```bash
#!/usr/bin/env bash
mkdir -p ~/.ssh
curl -Lo ~/.ssh/authorized_keys https://github.com/grenade.keys
```
- set hostname
```bash
#!/usr/bin/env bash
sudo hostnamectl set-hostname eddie.thgttg.com
```
- open firewall ports (handled in rubberneck but documented here because it took a bit of trial and error to get all the ports)
```bash
#!/usr/bin/env bash
# see: https://docs.prylabs.network/docs/prysm-usage/p2p-host-ip
for port_proto in {{3500,6060,8545,9100,13000}/tcp,12000/udp,30303/{tcp,udp}}; do
sudo firewall-cmd --zone=$(sudo firewall-cmd --get-default-zone) --add-port=${port_proto} --permanent
done
sudo firewall-cmd --reload
sudo firewall-cmd --list-all
```
- reset chain state. i got errors in the prysm journal (`journalctl -fu prysm.service`), including:
> Sep 01 08:07:48 eddie.thgttg.com prysm.sh[404877]: could not get ancestor state: slot 7223583 not in db due to checkpoint sync: cannot retrieve data for slot
which suggests a corrupt chain database which [requires](https://docs.prylabs.network/docs/troubleshooting/issues-errors) clearing out the data directory and restarting the chain.
```bash
#!/usr/bin/env bash
sudo systemctl stop geth.service
sudo systemctl stop prysm.service
sudo rm -rf /data/ethereum/{beaconchaindata,geth,geth.ipc,keystore}
sudo systemctl start geth.service
sudo systemctl start prysm.service
```
- rubberneck automation handles the rest: https://github.com/grenade/rubberneck/compare/7c2f943..d5b4b09
- geth/prysm user (eddie) and data directory (`/data/ethereum`) creation and permissions
- geth binary install
- geth service install, enable and start
- prysm script/binary install
- prysm service install, enable and start
- services keep-alive/hard-restart
- os security updates
# marvin.thgttg.com
a prometheus and grafana server.
***m**arvin* shares his initial with ***m**etrics*.
## server and network init
- create a dns entry for marvin.thgttg.com as an A record pointing to the static public ip for the hetzner instance
```bash
#!/usr/bin/env bash
curl \
--request POST \
--header 'Content-Type: application/json' \
--header "Authorization: Bearer $(pass cloudflare/thgttg.com/dns-api-key)" \
--data '{
"type": "A",
"name": "marvin.thgttg.com",
"content": "95.216.75.94"
}' \
https://api.cloudflare.com/client/v4/zones/80ac5f73f4e147bb0079d96e1497bf07/dns_records
# flush the local dns cache
sudo resolvectl flush-caches
```
- (re)image instance with clean partitions and a vanilla os install
- set hostname
- configure software raid
- partition drives (boot, lvm)
- create logical volumes on lvm:
- swap (16gb)
- / (64gb)
- /var/lib (512gb)
- /var/log (32gb)
- install os
- reboot
- remove rescue system host key
```bash=
#!/usr/bin/env bash
# see https://github.com/hetzneronline/installimage/blob/master/get_options.sh
fqdn=marvin.thgttg.com
ssh-keygen -R ${fqdn}
ssh -o StrictHostKeyChecking=accept-new root@${fqdn} \
/root/.oldroot/nfs/install/installimage \
-a \
-n ${fqdn} \
-r yes \
-l 1 \
-d sda,sdb \
-p /boot:ext3:1024M,lvm:vg0:all \
-v vg0:swap:swap:swap:16G,vg0:root:/:ext4:64G,vg0:lib:/var/lib:ext4:512G,vg0:log:/var/log:ext4:32G \
-i /root/images/CentOS-90-stream-amd64-base.tar.gz \
-G yes \
-K https://github.com/grenade.keys
ssh root@${fqdn} reboot
ssh-keygen -R ${fqdn}
sleep 300
ssh -o StrictHostKeyChecking=accept-new -o ConnectTimeout=1 root@${fqdn} dnf update -y
```
- create superuser
```bash=
#!/usr/bin/env bash
user=grenade
fqdn=marvin.thgttg.com
ssh root@${fqdn} "
getent passwd ${user} &> /dev/null || useradd --create-home --shell /bin/bash --user-group --groups wheel,systemd-journal ${user};
test -d /home/${user}/.ssh || sudo -H -u ${user} mkdir /home/${user}/.ssh;
chmod 700 /home/${user}/.ssh;
sudo -H -u ${user} bash -c 'curl -Lo /home/${user}/.ssh/authorized_keys https://github.com/grenade.keys';
chmod 644 /home/${user}/.ssh/authorized_keys;
echo '${user} ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/${user};
chmod 0440 /etc/sudoers.d/${user};
"
```
- rubberneck automation handles the rest: https://github.com/grenade/rubberneck/compare/d5b4b09..4c404a4
- prometheus binary install
- prometheus configuration
- prometheus service install, enable and start
- grafana install
- grafana configuration
- grafana service enable and start
- nginx install
- nginx configuration
- reverse proxy prometheus over tls
- reverse proxy grafana over tls
- certbot install
- obtain tls cert, configured for auto-renewals
- services keep-alive/hard-restart
- os security updates
- grafana manual config (in the [grafana ui](https://marvin.thgttg.com/grafana))
- enable github oauth in grafana config
- create (local) prometheus datasource
- import dashboards:
- [geth overview](https://grafana.com/grafana/dashboards/14053-geth-overview)
- [node exporter full](https://grafana.com/grafana/dashboards/1860-node-exporter-full)
# outputs from this excercise
- geth/prysm full node on private datacenter lan bare metal with nat and tls proxy for access to public endpoints: eddie.thgttg.com
- prometheus/grafana server on public (hetzner robot) bare metal: marvin.thgttg.com
- grafana dashboards (login with github. you must be a member of a github org shortlist):
- [geth overview](https://marvin.thgttg.com/grafana/d/FPpjH6Hik/geth-overview?orgId=1&refresh=10s)
- [node exporter full](https://marvin.thgttg.com/grafana/d/rYdddlPWk/node-exporter-full?orgId=1&var-DS_PROMETHEUS=default&var-job=thgttg.com%20-%20node&var-node=eddie.thgttg.com:443&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B%7Cmmcblk%5B0-9%5D%2B)
Sign in with Wallet
Connect another wallet