# Infra changes

Currently all of the yttrx mastodon services are managed on a single server, with configuration files manually crafted. The point of the upgrade is two fold:

1. Move the postgresql database off host to ensure better backups and better operations/scaling
2. Update `.env.production` on `tusky.masto.yttrx.com` to point to the new DB location
3. (Optional) Create a new VM (wooly.masto.yttrx.com)
4. (Optional) Re-create all of the services on wooly from ansible runbooks to ensure that nothing is configured but not documented

## postgresql

We'll need to take the services offline before backing up the postgresql database to ensure that we don't lose any data, but there are some steps that we can still do ahead of time.

1. Create the new database on digitalocean (DO).
2. Create the `mastodon` postgres user on DO
3. Test restoring a previous backup to DO and connect from `psql`

Once all of these steps have been verified to work, it's time to turn off all of the mastodon services, and nginx, then create a backup of the database.

Tear down any existing databases in DO, then restore using `pg_restore -Fc`, and test that it's working with `psql` from `tusky`.

We can at this points update the `.env.production` file and start turning services back on by running `docker-compose up -d` within the `~mastodon/live` directory

https://docs.digitalocean.com/products/databases/postgresql/how-to/create/

### Creating mastodon user on postgres server

Use `psql` to log into the new database, and create the mastodon user with permissions to create databases:
`CREATE USER mastodon WITH PASSWORD 'CHANGEME' CREATEDB;`

### Prepping the system

All databases are being backed up to `tusky` in the `/backup/dailies` directory, but we will want to get a new snapshot. First, we must stop all `nginx` services to ensure that nothing can update the database after we make a snapshot of it. Run:

systemctl stop nginx

to stop the service. For good measure, stop the web docker instance and sidekiq instances as well:

docker stop live-web-1
docker stop live-sidekiq-1

### Backing up existing database

Run:

docker exec -i live-db-1 pg_dump -Fc mastodon_production -U mastodon > /tmp/DB_TO_MIGRATE.bin

At this point, we should stop the local database just to be safe

docker stop live-db-1

And update the `docker-compose.yml` file to comment out the db section so that we don't accidentally restart it later.

### Restoring backup to new server

Per https://docs.digitalocean.com/products/databases/postgresql/how-to/import-databases/#import-data-in-custom-format & https://simplebackups.com/blog/postgresql-pgdump-and-pgrestore-guide-examples/ we want to run:

pg_restore -U mastodon -Fc -C -d mastodon_production < /tmp/DB_TO_MIGRATE.bin

Note, we'll have to wrap `pg_restore` in a shell script that has the correct connection strings for it to authenticate against the new database properly.

## Update docker-compose.yml

Since we're using `pgbouncer`, we don't actually want to update the `.env.production` file, but rather the config for `pgbouncer`. This is in:

~mastodon/live/docker-compose.yml

Update line 40 and change `db` to the hostname of the new database, and change the `DB_PASSWORD` to be the username of the `mastodon` user that we previously set above.

## Restart pgbouncer and stopped services

Run:

docker-compose up -d

from within `~mastodon/live`. This should restart `pgbouncer` and restart the `live-web-1` and `live-sidekiq-1` services. Watch the logs to ensure that it can talk to the new database:

docker logs -f live-sidekiq-1