# Hachyderm v2 Proposal

## Introduction
We need to move hachyderm.io out of Nóva's basement as the internet connectivity and UPS is shaky and we seem to have "cabling problems" during Twitch Streams.
It is hard to define the hardware requirements as we still see significant growth of hachyderm's user base.
Still, we need to start somewhere. The starting point should be a setup, that can be quickly set up but also quickly ripped down without long contractual bindings.

## Status Quo
Right now the hachyderm infrastructure can be separated into two parts: The Core and Content Delivery.

### The Core
The Core lives in Nóva's basement: `alice`, `yakko` and `wakko`.
`alice` runs Postgresql, Redis and an nginx serving media and distributing API requests to the Pumas.
`yakko` and `wakko` run the sidekiq workers and the web application servers (Puma).

*Sidenote: The Ruby applications use lots of database connections to the postgres. This results in a high memory usage by PostgreSQL on `alice`. We must try to use pgbouncer to bundle multiple database connections into one and by that reduce the memory load on `alice`.*

*Sidenote: `alice` also runs the Prometheus scraper and Grafana. Moving these to another node, maybe `buttons`, would provide us with reliable observation and reduce memory load on `alice`.*

### Content Delivery
The Content Distribution is working similar to Cloudflare: We run caching reverse proxies at multiple locations around the world. Those proxies are simple nginx servers running on Linode VMs. We have `slappy` in Newark and `esme` in Frankfurt. Linode provides a high quality backbone between their locations, which shows lower latency and especially lower jitter than transit providers like Level3 (aka Centurylink aka Lumen) across the Atlantic.

We increased service quality to Europe by picking up requests by `esme` and forwarding them via `slappy` to the US, compared to sending requests directly to `alice` in Seattle. However, we undid this as we were rate-limiting `esme` as requests on `alice` appeared to all be coming from the same (`esme`'s) IP address.

The nginx servers cache responses with static content on their local SSD; only the first request from a region results in downloading the media file from Seattle. All subsequent requests are served from cache. The cache has a configureable size and can use most of the local SSD. Nginx deletes files from its cache, as soon as the cache is hitting its size limit. Using an LRU cache policy nginx deletes those entities, that were Least Recently Used (LRU). This results in a very efficient use of the cache, holding only *hot data*.

## Scaling Characteristics
With the exception of the PostgreSQL database and Redis, all components are horizontally scalable.

The Content Delivery is easily scalable. We just need to add more nodes at different locations. Every node has a local cache and forwards requests to The Core, if it cannot serve them from the cache. The Content Delivery nodes at the moment are very small and have low CPU and RAM usage. Most important is a local SSD drive, that can be used for caching. There is lots of room to scale the Content Delivery horizontally as well as vertically.

The Core has multiple parts that can be scaled without too much trouble. The Media Vault for example could be served by storage optimized nodes (e.g. 14x 16TB HDDs + 2x 3.8TB SSD cache) and afterwards be scaled out using Ceph or Garage, for example.

The Pumas can be distributed to multiple application servers, which are then load balanced.

Same for the Sidekiq workers: They can be scaled out to multiple compute nodes, though it's critical that the Sidekiq workers are kept close to The Core to avoid latency between Redis and the workers.

The Redis database has two roles: Serving the sidekiq queues and providing an application cache. Mastodon allows to configure separate Redis instances for both roles. This allows to run one Redis server for the sidekiq queues and another one for application cache. We do not currently do this.

The PostgreSQL database is not horizontally scalable without lots of analysis and optimization work on the Mastodon application. As of now, the Postgresql can only be scaled vertically. Up to 2TB, maybe 4TB of RAM the costs of a database server grow linearly. After that you'd need 4 or 8 processor intel machines, which will become fucking expensive.

## Proposal

I don't think a *buy now and be happy for 3 years* strategy works for us right now. I think we should do two steps:

1. Rent dedicated servers, which are available *now*. Scaling can be done be adding more servers.
2. Rent a rack in a colo and run on own hardware.

I [@kris-nova](https://github.com/krisnova) believe we should build an entire new infrastructure in Hetzner, then we destroy the water tower, and rebuild the water tower as a hetzner backup.

### 1. Step - Rent Dedicated Servers
I suggest to rent dedicated servers from a provider, that is well connected to Linode. In a location near Newark, London or Frankfurt. [Hetzner](https://www.hetzner.com) is an option here (FSN or NBG datacenters), and we have recently acquired `buttons`.

#### Compute Nodes
We should have two or three servers, that run Postgres, Redis, Puma and Sidekiq. Maybe an additional one to home the Media Vault.
@dma's `Buttons` server has the following specs:

| Spec | Value |
| --------- | ---------------------------------- |
| **CPU** | Intel Xeon W-2145 (8 Cores, 16 Threads, 3.7 GHz) ([Benchmark](https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+W-2145+%40+3.70GHz&id=3156)) |
| **RAM** | 128 GB (4x 32GB) DDR4 ECC reg. |
| **Disk** | 2x 960 GB Datacenter SATA SSD |
| **NIC** | 1 Gbps - intel i210 |
| **Price** | 72 EUR/mo - [Hetzner Server Auction](https://www.hetzner.com/sb) |

Such a server could take over Alice's role and serve Postgres, Redis and nginx (media vault + Load Balancer for Puma). In a first step also run Puma itself.

A second server hosted close-by would run Sidekiq processes.

#### Hetzner Cloud
For traffic peaks or for experimenting with different Puma/Sidekiq configurations, we could leverage [Hetzner Cloud](https://www.hetzner.com/cloud) instances temporarily.

#### Storage / Media Vault
LRU caching on SSDs should allow us to use spinning disks for our media vault. The spinning disks will hold the complete data set, while all hot data would be served by LRU caches. Only requests to read cold data (and the very first read of a new asset) would hit the spinning disks.

Our first storage server could look like this:

| Spec | Value |
| --------- | ---------------------------------- |
| **CPU** | Intel Xeon E3-1275v5 (4 Cores, 8 Threads @ 3.6 GHz) ([Benchmark](https://www.cpubenchmark.net/cpu_lookup.php?cpu=Intel+Xeon+E3-1275+v5+%40+3.60GHz&id=2672)) |
| **RAM** | 64 GB (4x 16GB) DDR4 ECC |
| **Disk** | 4x 16TB SATA HDD |
| **NIC** | 1 Gbps - intel i210 |
| **Price** | 90 EUR/mo - [Hetzner Server Auction](https://www.hetzner.com/sb) |

or a bit larger the SX297:

| Spec | Value |
| --------- | ---------------------------------- |
| **CPU** | Intel Xeon W-2145 (8 Cores, 16 Threads, 3.7 GHz) ([Benchmark](https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+W-2145+%40+3.70GHz&id=3156)) |
| **RAM** | 256 GB DDR4 ECC |
| **SSD** | 2x 3,84TB NVMe |
| **HDD** | 14x 16TB SATA 7200rpm HDD |
| **NIC** | 1 Gbps - intel i210 |
| **Price** | [404 EUR/mo - SX297](https://www.hetzner.com/de/dedicated-rootserver/matrix-sx) |

All storage data is rsync'ed regularly to Nóva's basement.

### 2. Step - Rent a Rack and run on own Hardware
When hachyderm has reached a size, that can not be handled by standard dedicated hardware anymore, we should rent a rack in a colo and run on own hardware. Now is the time to deploy Ceph and run large Postgres servers with 1TB RAM or more. Hurricane Electric is not a bad choice when you're [on a budget](https://he.net/colocation.html).

#### Why not go directly into a colo?
Because it takes time and a lot of work and a lot of money - at Hetzner you get the dedicated servers within an hour. It takes a few days to get a rack at HE. The rack is in Fremont, so no quick visit from Seattle. We need to rack and cable the servers. And we have to have these servers in the first place! We need to keep replacement disks on stock.

An own rack is more professional, it provides flexibility, you don't get from rented servers. But at the same time it's also less flexible because you have to order servers, install them etc.

An own rack is only about hosting The Core. You still need a Content Delivery Network.

### Content Delivery Network
We would run multiple Linode VMs, forming our CDN. As the Linode backbone is of high quality, all traffic flowing between a PoP and The Core should use the Linode backbone. Therefore we would set up an *Entry Node* in Frankfurt. Hetzner peers with Linode via DECIX Frankfurt, so we got a fast, reliable, low latency connection here. Europe itself would be served directly from The Core at Hetzner (especially as Hetzner Dedicated Servers have unlimited traffic included, only limited by their 1 Gbps connection). Additionally we would run Linode VMs as PoPs in Newark and Fremont and optionally Singapore.

We should calculate with 30 USD/mo per PoP including 4 TB of traffic. Plus 10 USD for every additional TB of traffic. In sum about 150 - 200 USD/mo.

### Costs
The costs for the 1. Step with dedicated servers can be approximated like this:

| Product | Object | Monthly Cost |
| -------------- | -------------------------- | ------------ |
| **The Core** | | **306 EUR** |
| | DB Server | 72 EUR |
| | Sidekiq Server | 72 EUR |
| | Puma Server | 72 EUR |
| | Storage Server (4x 16 TB) | 90 EUR |
| **CDN** | | **190 EUR** |
| | Entry Node FRA | 30 EUR |
| | PoP Newark | 30 EUR |
| | PoP Fremont | 30 EUR |
| | Additional Traffic (10 TB) | 100 EUR |
| **Sum** | | **496 EUR** |

This includes unlimited Traffic for Europe, Africa & Asia. PoP Newark and PoP Fremont include 4 TB each. So with the additional 10 TB the calculation includes 18TB traffic only for the Americas.
Hetzner's *Server Auction* servers have no setup fee and can be canceled within one month.

---

## Workloads

### Media Share
Serves static content that was uploaded by users or fetched by sidekiq workers from other Mastodon instances.

### Puma
Puma (mastodon-web) is an application server, that runs Rails web applications.

### PostgreSQL
The Postgres database is accessed by a lot of clients: puma and sidekiq workers, multiplied by their processes and their threads.

PostgreSQL must be backed up. It makes sense to run a hot-standby instance that can take over if the primary instance goes down.

### Sidekiq Workers
Asynchronous jobs are scheduled via Redis queues to a set of workers.
#### Queues
* default
* ingress
* pull
* push
* mailers
* scheduler (must be processed by *exactly one* worker)

### Redis
Redis acts as a queue for the Sidekiq workers and as a cache for Puma.
A separate Redis to serve the cache can be set up if one Redis instance is too weak to serve both.

---

## machines

see [here](https://github.com/hachyderm/infrastructure/blob/main/docs/infrastructure.md)

## machine types

### the core

_currently alice_

one primary tank at any time, with potential for a second one for redundancy.

runs pumas (_mastodon-web_), _postgres_ and _redis_.

optionally runs central sidekiqs (scheduler, mailers).

requires:
- moderate storage (database + cache)
- high RAM
- moderate CPU

### compute nodes

_currently wakko, yakko_

compute nodes running sidekiq workers (_mastodon_sidekiq_) for default, pull, push.

requires:
- moderate CPU (we can run many of these)
- moderate RAM
- low storage

### pops

_currently esme, slappy_

run nginx as a proxy to the tank.

requires:
- plenty of bandwidth
- low CPU
- low RAM
- moderate storage (cache)

### Outstanding TODO Items

- [ ] Operating System Plan (NixOS, SSH Key management)
- [ ] Edge footprint layer 2 (TCP Ports Open, Exposed, Where, What is public?)
- [ ] DDoS protection (No Cloudflare unless we have to)
- [ ] Rsync strategy (Cron? Daemon? Etc)
- [ ] Postgres replica strategy (Over what bridge? tailscale?)
- [ ] How do we "toggle" between EU headquarters in hetzner and on-premise in Seattle? Do we?
- [ ] Read/Write model (Who gets the writes? Where do we read?)
- [ ] Official Terms/Language/Glossary. What is a pop? What is a "tank"? What is a "media vault"? Just give us words to use once-and-for-all

### Immediate Action Items

- [ ] We need a "corporate" account on a credit card
- [ ] Hetzner
- [ ] Linode
- [ ] Small investment into Seattle rack
- [ ] UPS
- [ ] Fix dot's memory
- [ ] Destroy alice, wakko, and yakko and re-build with new-plan

### Decisions

- [X] Guiding Principle: Boring and Safe
- All of our decisions are rooted in simplicity and safety for the service
- [X] Guiding Principle: Cost Effective, Legally Sound
- We need to keep hachyderm sustainable, so we keep it cheap and legally safe
- [X] Officialy are moving to Hetzner (with corporate account).
- FSN is the official primary data center/colocation of Hachyderm.io
- [X] Corporate Account Setup (Malte, Tani, DMA, Esk, Q, Nova)
- [X] Official codename of "Nóva's Rack" is "The Water Tower"
- Keep the water tower online
- [ ] 2X 20 amp circuits
- [ ] Shielded cat6 to fiber drop
- [ ] Secure/Lock/Hardwire Gateways
- [ ] UPS
- [ ] Fix Dot's memory usage
- [X] Official operating system is NixOS
- [X] Official edge/pop strategy is Linode (No Cloudflare)
- We are convinced we will need to be able to build/deply edge/cache proxies arbitrarily
- So we run in Linode and understand we can deploy to other providers in the future
- [X] Official decision not to run on Kubernetes, but to leverage systemd directly
- In the future we may "tiptoe" on to Aurae when the time is right and we feel good