# Hachyderm Migration to Hetzner

## Architecture Overview

### Hetzner Servers
* Freud (Intel W-2145, 128 GB RAM, 2x500G SATA Enterprise SSD)
* PostgreSQL
* Persistent Redis
* prometheus exporters
* node
* postgresql

* Fritz (Intel W-2145, 128 GB RAM, 2x500G SATA Consumer SSD)
* mastodon-web
* mastodon-streaming
* nginx
* Redis (no persistency, for cache)
* prometheus exporters
* nginx
* nginxlog
* node
* statsd

* Franz (Intel W-2145, 128 GB RAM, 2x500G SATA Consumer SSD)
* sidekiq-default (16x25) (queues: default, push, pull)
* sidekiq-ingress (8x75)
* sidekiq-mailers (1x5)
* sidekiq-scheduler (1x5)
* prometheus exporters
* node
* statsd

* Bad-Nixie (Intel W-2145, 128 GB RAM, 2x1,9T SATA Enterprise SSD)
* tbd

* Buttons (Intel W-2145, 128 GB RAM, 2x500G SATA SSD)
- [X] Prometheus scraper
- [X] prometheus exporters
- [X] node
- [X] Grafana

### CDN
We got four servers at Linode:
* cdn-frankfurt-1 - Frankfurt (EU)
* cdn-fremont-1 - obviously in Fremont (rest of world)
* slappy - Newark (US East, South America)
* esme - (east coast?)

They will cache and forward hachyderm.io traffic via tailscale to Fritz. And they will cache and forward traffic for media.hachyderm.io to Alice.

We will optimize the regional coverage of the CDN nodes after everything is stable. Esme will become default and fremont will cover US west.

## Outcomes:

- [ ] Alice shutdown plan in place after data is transferred to DO
- [ ] Power off dot (possible)
- [ ] Power off yakko (possible)
- [ ] Power off wakko (possible)
- [ ] The DNS records will updated entirely after this migration is complete

## Step by Step

- [X] Announce downtime
- [x] Yakko shutdown
- [x] Shutdown Puma - `systemctl stop mastodon-web`
- [x] Shutdown Streaming - `systemctl stop mastodon-streaming`
- [x] Wakko shutdown
- [x] Shutdown Puma - `systemctl stop mastodon-web`
- [x] Shutdown Streaming - `systemctl stop mastodon-streaming`
- [x] Alice: Shutdown Streaming - `systemctl stop mastodon-streaming`
- [x] Wait for Queues to be 0
- [x] Dot: Shutdown Sidekiqs
- [x] `for i in {1..16}; do systemctl stop mastodon-sidekiq-default@$i; done`
- [x] `for i in {1..8}; do systemctl stop mastodon-sidekiq-ingress@$i; done`
- [x] Alice: Shutdown Sidekiqs (scheduler + mailers)
- [x] `systemctl stop mastodon-sidekiq-mailers`
- [x] `systemctl stop mastodon-sidekiq-scheduler`
- [x] Freud: Ensure Replication Lag is near 0s - `sudo -u postgres psql -c "select now()-pg_last_xact_replay_timestamp() as replication_lag;"`
- [x] Alice: Stop Postgresql - `systemctl stop postgresql`
- [x] Freud: Remove `standby.signal` file from postgres data dir - `rm /var/lib/postgres/data/standby.signal`
- [x] Freud: comment out `primary_conninfo` and `hot_standby` in `/var/lib/postgres/data/postgresql.conf`
- [x] Freud: Restart Postgres - `systemctl restart postgresql`
- [x] Freud: Check Postgresql logs if everything is fine
- [x] Franz: Install pgbouncer and configure `/etc/pgbouncer/pgbouncer.ini`
- [x] Franz: Start Sidekiq Workers at Hetzner
- [x] `for i in {1..16}; do systemctl start mastodon-sidekiq-default@$i; done`
- [x] `for i in {1..8}; do systemctl start mastodon-sidekiq-ingress@$i; done`
- [x] `systemctl start mastodon-sidekiq-mailers`
- [x] `systemctl start mastodon-sidekiq-scheduler`
- [x] Fritz: Start Pumas - `systemctl start mastodon-web`
- [x] Fritz: Start Streaming - `systemctl start mastodon-streaming`
- [x] Change CDN hachyderm.io upstream to point to Fritz
- [x] cdn-fremont-1
- [x] `sed -i "s/alice/fritz/g" /etc/nginx/sites-available/hachyderm.io`
- [x] `systemctl reload nginx`
- [x] slappy
- [x] `sed -i "s/alice/fritz/g" /etc/nginx/sites-available/hachyderm.io`
- [x] `systemctl reload nginx`
- [x] esme
- [x] `sed -i "s/alice/fritz/g" /etc/nginx/sites-available/hachyderm.io`
- [x] `systemctl reload nginx`
- [ ] Enjoy

### rollback

- [ ] Yakko startup
- [ ] Start Puma - `systemctl start mastodon-web`
- [ ] Start Streaming - `systemctl start mastodon-streaming`
- [ ] Wakko startup
- [ ] Start Puma - `systemctl start mastodon-web`
- [ ] Start Streaming - `systemctl start mastodon-streaming`

### Current Work Status

- Problem with CDN
- Info errors in sidekiq

Dec 01 21:10:41 fritz bundle[37876]: [4c9eee37-d1c6-4e8d-806e-d606e11541b4] method=GET path=/api/v1/announcements format=html controller=Api::V1::AnnouncementsController action=index status=200 duration=22.38 view=14.02 db=4.61
Dec 01 21:10:41 fritz bundle[37897]: [5671df5c-7142-4c72-9851-c7d565ddcc21] method=GET path=/api/v1/announcements format=html controller=Api::V1::AnnouncementsController action=index status=200 duration=27.32 view=17.35 db=5.30
Dec 01 21:10:41 fritz bundle[38029]: [9ee92005-9885-4b5e-9213-73ad81e3a34f] method=GET path=/api/v1/notifications format=html controller=Api::V1::NotificationsController action=index status=200 duration=18.40 view=0.43 db=5.58
Dec 01 21:10:41 fritz bundle[38202]: [f9bfe1ca-fabc-4053-a23f-c326b94b92f5] method=GET path=/sw.js format=*/* controller=ApplicationController action=raise_not_found status=500 error='ActionView::Template::Error: Webpacker can't find common.css in /var/lib/mastodon/public/packs/manifest.json. Possible causes:
Dec 01 21:10:41 fritz bundle[38202]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 01 21:10:41 fritz bundle[38202]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 01 21:10:41 fritz bundle[38202]: 2. webpack has not yet re-run to reflect updates.
Dec 01 21:10:41 fritz bundle[38202]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 01 21:10:41 fritz bundle[38202]: 4. Your webpack configuration is not creating a manifest.
Dec 01 21:10:41 fritz bundle[38202]: Your manifest contains:
Dec 01 21:10:41 fritz bundle[38202]: {
Dec 01 21:10:41 fritz bundle[38202]: }
Dec 01 21:10:41 fritz bundle[38202]: ' duration=15.74 view=0.00 db=4.81
Dec 01 21:10:41 fritz bundle[38162]: [a940b4ec-04cc-48f3-9202-1b9704aa5ef1] method=GET path=/api/v1/tags/photography format=html controller=Api::V1::TagsController action=show status=200 duration=24.52 view=15.56 db=3.80
Dec 01 21:10:41 fritz bundle[38202]: [f9bfe1ca-fabc-4053-a23f-c326b94b92f5]
Dec 01 21:10:41 fritz bundle[38202]: [f9bfe1ca-fabc-4053-a23f-c326b94b92f5] ActionView::Template::Error (Webpacker can't find common.css in /var/lib/mastodon/public/packs/manifest.json. Possible causes:
Dec 01 21:10:41 fritz bundle[38202]: 1. You want to set webpacker.yml value of compile to true for your environment
Dec 01 21:10:41 fritz bundle[38202]:    unless you are using the `webpack -w` or the webpack-dev-server.
Dec 01 21:10:41 fritz bundle[38202]: 2. webpack has not yet re-run to reflect updates.
Dec 01 21:10:41 fritz bundle[38202]: 3. You have misconfigured Webpacker's config/webpacker.yml file.
Dec 01 21:10:41 fritz bundle[38202]: 4. Your webpack configuration is not creating a manifest.
Dec 01 21:10:41 fritz bundle[38202]: Your manifest contains:
Dec 01 21:10:41 fritz bundle[38202]: {
Dec 01 21:10:41 fritz bundle[38202]: }
Dec 01 21:10:41 fritz bundle[38202]: ):
Dec 01 21:10:41 fritz bundle[38202]: [f9bfe1ca-fabc-4053-a23f-c326b94b92f5] ```


## Root Cause of Failure on mastodon-web

RAILS_ENV=production bundle exec rails assets:precompile
RAILS_ENV=production bundle exec rails webpacker:compile
export NODE_OPTIONS=--openssl-legacy-provider