# PLUG Server Upgrade Plan

- this document: https://hackmd.io/@plug/server-upgrade-plan ( alias of https://hackmd.io/hWc4sXeRRkymUlWI21Adtw )

## Current State

- We currently have two cloud VMs up and running:

1. `power.plug.org.au`
* Running in AWS
* 1GiB RAM, 4GiB swap (1.4GiB in use)
* Debian 7/wheezy (released in 2013, support ended in 2018)
* Runs most services
* Configured over time.
2. `edison.plug.org.au`
* Running in Digital Ocean
* 2GiB RAM
* Debian 9/stretch (released in 2017, support ended in 2022)
* Runs DNS and Letsencrypt certificate requests
* Configured via Ansible playbook

## Goals

1. Decommission `edison`.
2. Replace with a server running Debian 12/bookworm (released in 2023, support ends in 2028)
* preferably using a version of our Ansible playbook.
3. Decommission `power`.
4. Have a plan to upgrade the new server to Debian 13 later this year.
* and beyond: we can't let our infrastructure get so out of date.

## Blockers

1. PLUG DNS has an `NS ns1.plug.org.au` record pointing at `edison`. We will need to coordinate with Tim White to update the glue records and point his secondary DNS servers elsewhere.
2. `power` currently uses TLS certificates issued by `edison`'s letsencrypt setup.
* We will need a new system capable of requesting certificates before the 90 day expiry of the last certs from `edison`.
3. Testing plan: web/postfix/mailman/ugmm/PHP/fail2ban
4. ugmm may run, or may have new errors and warnings

## Plan
0. Backup all the things
* Archive other AWS data to plug.michael5collins.com and delete from power/S3/EBS snapshots/EFS

1. Stand up a new VM in binarylane.com.au :
* Image: latest official Debian 12 AMI
* Instance type: 2GB $7.50 (+GST, AUD=$8.25)
* Hostname: usb for its brevity
2. Run the Ansible playbook on new server until error.
3. Try to manually configure certbot, similar to the following?
* The Ansible playbook won't bootstrap a system from scratch. It does not configure certbot to request certificates, so config files for nginx, etc are not created. On a fresh machine this causes those services not to start, and the playbook fails.

       # certbot --dns-rfc2136 --preferred-challenges dns -d 'plug.org.au' -d '*.plug.org.au' -m [e-mail] certonly
       # certbot --nginx -d mail.plug.org.au -m [e-mail] certonly
       # cp /usr/lib/python3/dist-packages/certbot/ssl-dhparams.pem /etc/letsencrypt
       # cp /usr/lib/python3/dist-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf /etc/letsencrypt
       # ln -s plug.org.au /etc/letsencrypt/live/mail.plug.org.au

`mail.plug.org.au` on edison is set up with nginx mode authenticator, so can't set up until we migrate that hostname. Maybe this is what created the `options-ssl-nginx.conf` and `ssl-dhparams.pem` files?

4. Run playbook again, hopefully to completion.
* This should bring up the services listed below, under "Start migrating services"
5. Nick: Testing and iteration
* The services that have web endpoints have some basic and easy tests with our `webcheck.py` or similar, comparing to `power`
* Mail and mailing lists might need more manual? but repeatable-and-repeated testing, previous testing was started on a subdomain po1.plug.org.au
* `swaks(1)` is handy
* `postsuper -h ALL`
* Don't send floods of testing/duplicate/membership expiry email to our mailman subscribers/UGMM members
* Clear the held-for-moderation messages in our mailing lists before cutover
* Look for log errors and warnings, especially from mail/postfix/mailman/ugmm - iterate, record or script/playbook the "manual" steps until we've fixed enough of them
6. Update DNS zone files in playbook to point `ns1.plug.org.au` at new server's IP.
7. Contact Tim White, and request that he update glue records and his secondary DNS servers to the new `ns1`.
8. Verify that new server can successfully request certificates.
9. Turn off `edison`. If nothing breaks, consider closing Digital Ocean account.
10. Start migrating services from `power`, updating CNAMEs as we go and using `power`'s data backups:
* website
* mailing lists
* postfix
* mailman
* pipermail archives
* LDAP plus UGMM
* Nick: backups on the new machine
* Nick: fail2ban on login services: ssh, ugmm, mailman, dovecot?
* sshd` MaxAuthTries/MaxStartups/PerSourceMaxStartups/PerSourceNetBlockSize
* Other low priority services like mumble
11. Turn off `power`.

## Future

1. Once mailing lists have been migrated, attempt migration to Mailman 3.
2. When Debian 13 is released or close to release, try snapshotting new machine's storage and perform the upgrade on the live system. Rollback if it fails.

----
## Mail problems

Postfix complained about missing aliases because mailman hadn't yet created `/var/lib/mailman/data/aliases`.

clamav-daemon.service hadn't started due to missing virus definitions, and hadn't retried. Seems to have been a bootstrap issue, since they have since been updated.

----
## 12 October Migration Notes

Migrating mailman data to iec:
1. On power: `sudo service postfix stop`
2. On iec: `sudo systemctl stop postfix`
3. On power as root: `cd /var/lib/mailman; tar czf ~admin/mailman.tar.gz archives data lists messages qfiles`
4. `scp(1)` to `iec.plug.org.au`
5. `tar xzf ...`
6. `mail -r` test message, all mail is deferred
7. `sendmail -q` to run the queue and send test messages! Arrived!
1. https://lists2.plug.org.au/mailman/private/admin/2025-October/012259.html
2. https://lists2.plug.org.au/mailman/private/admin/2025-October/012260.html
3. https://lists2.plug.org.au/mailman/private/admin/2025-October/012261.html
- SPF/DKIM/DMARC pass!

Migrating ugmm to iec:

1. LDAP/UGMM data: On power, `/etc/cron.daily/20-ldapdump`
1. `power:/etc/cron.daily/borgbackup`
2. `power:/etc/cron.daily/borgbackup-pchq`
2. Copied /var/backup/slapd/data.ldif` to iec
4. import `data.ldif`
5. `systemd restart slapd`
6. test logins at: https://ugmm.plug.org.au/
1. success!
2. Resent payment#795: success! https://lists2.plug.org.au/mailman/private/admin/2025-October/012262.html

Next:
1. configure plug.org.au as mailname on iec
2. stop deferring outgoing email
3. Update ugmm nginx config:
* allow signups
* Enable javascript for tablesort
4. Update DNS:
* plug.org.au MX points at mail.plug.org.au (CNAME for iec)
* lists.plug.org.au CNAME to iec
*

$ host -tNS plug.org.au
plug.org.au name server ns1.plug.org.au.
plug.org.au name server ns1.whiteitsolutions.com.au.
plug.org.au name server ns4.whiteitsolutions.com.au.
$ host -tSOA plug.org.au
plug.org.au has SOA record ns1.plug.org.au. hostmaster.plug.org.au. 2025092501 4800 900 2419200 300

$ date;dig +short NS plug.org.au|parallel -v dig +short SOA plug.org.au
Sun 12 Oct 2025 13:20:44 AWST
dig +short SOA plug.org.au ns1.plug.org.au.
ns1.plug.org.au. hostmaster.plug.org.au. 2025101200 4800 900 2419200 300
dig +short SOA plug.org.au ns1.whiteitsolutions.com.au.
ns1.plug.org.au. hostmaster.plug.org.au. 2025101200 4800 900 2419200 300
dig +short SOA plug.org.au ns4.whiteitsolutions.com.au.
ns1.plug.org.au. hostmaster.plug.org.au. 2025101200 4800 900 2419200 300

5. Sent test email: success!
1. https://lists.plug.org.au/mailman/private/committee/2025-October/012088.html
2. https://lists.plug.org.au/pipermail/plug/2025-October/085092.html
3. https://lists.plug.org.au/pipermail/plug/2025-October/085093.html

List-Archive: <http://lists.plug.org.au/pipermail/plug/>

iec:/var/lib/mailman/bin# ./withlist -h
Mailman/Defaults.py:PUBLIC_ARCHIVE_URL has http://${listname}... ?
-> update with overrides: iec:/etc/mailman/mm_cfg.py in https://github.com/plugorgau/plug-services/blob/main/roles/mailman2/templates/mm_cfg.py.j2

6. Probably still under attack by the fast-mail-bomber, but
1. Lots of incoming noise in:
- `/var/log/nginx/access.log`
- `less +/subscribe /var/log/nginx/access.log`
- `116.96.77.113 - - [12/Oct/2025:15:06:42 +0800] "GET /mailman/subscribe/off-topic?email=[e-mail]&fullname=&pw=1ja19f2m9f&pw-conf=1ja19f2m9f&digest=0&language=en&email-button=Subscribe HTTP/1.1" 200 1476 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:85.0) Gecko/20100101 Firefox/85.0"
`
- `/var/log/nginx/error.log`

# journalctl -xu postfix

iec:/etc/postfix# mailq |head -20
-Queue ID-  --Size-- ----Arrival Time---- -Sender/Recipient-------
BFF4C43A8A     1854 Sun Oct 12 15:25:26  [e-mail]
                                                          (deferred transport)
                                         [e-mail]
[...]

https://wiki.zimbra.com/wiki/Managing-The-Postfix-Queues

iec:/etc/postfix# postsuper -d BFF4C43A8A
postsuper: BFF4C43A8A: removed
postsuper: Deleted: 1 message

# postcat -q 138A743A8C
[more fast-mail-bomber subscribe spam]

# postqueue -p | awk 'BEGIN { RS = "" } { if ($10 == "[e-mail]" ) print }'
# postqueue -p | awk 'BEGIN { RS = "" } { if ($10 == "[e-mail]" ) print $10 }'
# postqueue -p | awk 'BEGIN { RS = "" } { if (/Knester/) print $1}'

1. the new mailman2 should not let that work with a blind HTTP submission, due to CSRF protection `crsf_check`
- nope, need something closer to mailman 2.1.35 , not 2.1.29 ?
- by cookie, not nonce in the subscribe form?
- other bots are trying too
2. build a new mailman2 package?
- new container
- `apt install lxd lxd-tools` ?
- see https://github.com/plugorgau/ugmm/blob/main/HACKING.md
- https://launchpad.net/~mjeanson/+archive/ubuntu/ppa
- mailman 1:2.1.31-0~ubuntu18.0.4.1 Michael Jeanson (2020-05-05)
- mailman 1:2.1.39-1~ubuntu20.04.1 Michael Jeanson (2022-02-16)
- `apt build-deps` - needs old https://tracker.debian.org/pkg/dh-python ?
3. add recaptcha on committee@plug account to subscribe form
- limit of 10000 checks/month on a free account?
- but that might be when our code actually checks, and the subscribe spam might not trigger that
5. moderate new subscription requests/subscribers?
6. `nginx` workaround?
- block the subscribe URL?
- require a correct `Referer` (sic) header?
- block spammy `GET` method vs `POST`?

7. Backups
1. Note: Trixie `apt install borgmatic` requires Borg1, but its Borgmatic is new enough to support Borg2 too
- https://tracker.debian.org/pkg/borgbackup / borgbackup2 / borgmatic