# Hachyderm PoPs

## Idea
We want to install a single server at a remote location (Point of Presence) to better serve hachyderm to a specific region (e.g. Europe).

## Do's

### Geo Load Balancing
As we do not announce our IP addresses using BGP, we cannot use anycast routing like Cloudflare does. Instead we have to use Geo DNS and provide different A records to different parts of the world. [AWS Route 53 supports GeoDNS](https://docs.aws.amazon.com/en_us/Route53/latest/DeveloperGuide/routing-policy-geo.html).

### Puma
Puma should be run on the PoP server. It reduces the latency the users experiences when loading their timelines or browsing through the fediverse. This requires a [Postgres read replica](https://cloud.google.com/community/tutorials/setting-up-postgres-hot-standby) to be installed locally at the PoP. Writes still must be sent to hachyderm HQ (see [Makara configuration](https://docs.joinmastodon.org/admin/scaling/#read-replicas)). Also file uploads need to be sent to the HQ's S3 endpoint (hachyderm HQ S3 must be configured in `/etc/mastodon.conf`). As long as Ceph is not set up, the PoP needs to mount HQ's NFS share to write files.
Further, the Rails application running on Puma needs access to Redis to push new jobs to the Sidekiq queues. This requires a TLS protected Redis, or a VPN connection to the HQ.
In front of the puma processes is an nginx, that does TLS termination.

### Media
The media storage of hachyderm is currently at 250 GiB (mid of November 2022). Looking forward, the media share may outgrow the capacity of a single server. Therefore we are going to set up a Ceph cluster in the Hachyderm HQ.
Considering this, the media share cannot be replicated fully to a remote PoP, if a PoP is just one single server. Instead, we can replicate media of the last X days and additionally implement a cache for media, that is older but requested by a client.
This ~~can be~~ should not be realized using an rsync of files newer than X from Hachyderm HQ to the PoP's storage. nginx would be configured using `try_files $uri @proxy` to look for requested media on the local disk and if not found forward the request to hachyderm HQ using reverse proxying.
Additionally the reverse proxy would implement `proxy_cache`.
*Question*: How to always send the newest files to the PoP? rsync run as a cronjob is not transferring media when it gets pushed into the timelines/feeds.
*Remark*: We could disable syncing before having Ceph + Geo Replication in place. Until then media is always retrieved via reverse proxying and cached by nginx.

// This config is not tested and will not work out of the box.
// It's just to get the idea.

server {
server_name media.hachyderm.io;

root /mnt/media-share; // rsynced media from the last X days

listen 443 ssl;
ssl_certificate ...;
ssl_certificate_key ...;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam ...;
add_header Strict-Transport-Security "max-age=31536000";

location / {
try_files $uri @proxy;
}

location @proxy {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Proxy "";
proxy_pass_header Server;

proxy_pass https://media.hq.hachyderm.io; // media served by HQ's Ceph

// carefully check and tune the parameters below:
proxy_cache CACHE;
proxy_cache_path "/mnt/proxy-cache";
proxy_cache_valid 200 48h;
proxy_cache_valid 410 24h;
proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
add_header X-Cached $upstream_cache_status;
}
}

### Postgres Read Replica
As ligned out in the Puma section, a Postgres read replica should be installed at the PoP.

### Redis
Puma needs to write to the HQ's Redis. For this the Redis server must be exposed to the internet, allowing only TLS connections - or - a VPN connection between HQ and PoP needs to be established, so the PoP can access HQ's Redis via private IP addresses.

## Don'ts
### Sidekiq
There will be no sidekiq workers running at PoPs. The HQ remains the central installation of hachyderm. All processing happens in the HQ.

### Streaming
We do not need to install a streaming server at the PoP. The streaming is mostly a firehose of Toots to send updates to the local and federated feed. Latency is no issue, as the user does not recognize if a toot arrives 500ms earlier or later.

## Step by Step
1. Get a Server
2. Partition it
a. Root Partition
b. Media Partition
c. Cache Partition
(d. Postgres Partition?)
3. Create a VPN to Hachyderm HQ
4. Set up Postgres Read Replica ([link](https://cloud.google.com/community/tutorials/setting-up-postgres-hot-standby))
5. ~~Setup sync of latest media~~ Mount NFS
6. Configure nginx
a. Sync Letsencrypt certs from HQ for media.hachyderm.io and hachyderm.io (rsync cron-job)
b. Set up media ~~serving~~ proxying and caching
c. Puma reverse proxy
7. Set up Mastodon / Puma
a. Using Postgres replica@PoP for reads and master@HQ for writes ([link](https://docs.joinmastodon.org/admin/scaling/#read-replicas))
b. Connect to Redis@HQ
c. Write files to HQ media storage
8. Test PoP by configuring `/etc/hosts` and point media.hachyderm.io and hachyderm.io to the PoP
9. Set up GeoDNS