Try   HackMD

PLUG Hack Day 2020-06-13 + 2020-06-14 + 2023-03-19 + 2023-04-09 + 2023-07-09 + 2023-08-08 + 2023-09-10/12 2023-11-21 2024-02-11 2024-02-13 2024-04-14 2024-05-14 2024-06-09 2024-09-08 2024-10-13 2024-11-10 2024-12-08 2025-01-07 2025-02-18 2025-03-09 2025-04-13 2025-06-08 2025-07-13

(update title when jamesh/harry signed in)

  • this document: https://hackmd.io/@plug/hack-day-notes

    • ( alias of https://hackmd.io/_o_65OZbQMin0ANI2lz6-g )
    • wget https://hackmd.io/_o_65OZbQMin0ANI2lz6-g/download -O $(date +%Y-%m-%d)-infra.md
    • (FNAME=$(date +%Y-%m-%d)-infra.md; DOC=hack-day-notes; CODIMD_SERVER=https://hackmd.io/@plug codimd export --md $DOC "./$FNAME")
  • was https://meetings.ucc.asn.au/b/nic-4cd-3jg

  • 2020 Nick+BenjaminA

  • 2023-03-19 , 2023-04-09 Nick+Niall

  • 2023-07-09 Nick + Michael + Niall + James + BenjaminIDS

  • 2023-08-08 Niall + Nick + Aiden + Dylan + Craig + James + Harry

  • 2023-08-13 Niall + Nick + Jason

  • 2023-09-10/12 Nick + Michael + BenjaminIDS + JasperG + JamesH

  • 2023-11-21 Nick + James

  • 2024-02-11 Niall + DanB + Nick + BenjaminIDS

  • 2024-02-13 BenjaminIDS + Dan + Nick + Sarah + James

  • 2024-04-14 Niall + Nick + BenjaminIDS

  • 2024-05-14 Nick, BenIDS + Sarah + MarkW + Owain + LawrenceL

  • 2024-06-09 Nick + MarkW

  • 2024-09-10 Nick + MichaelC + JamesH + DanB

  • 2024-10-13 Harry + Nick + Jacek + JamesH + Dan + Mark

  • 2024-11-10 Nick + Dan

  • 2024-11-10 , 2025-01-07 Nick + Wyatt

  • 2025-02-18 Nick + Wyatt + JamesH

  • 2025-03-09 Nick + JamesH + MarkW

  • 2025-04-13 Nick + MarkW + JamesH + TimL

  • 2025-06-08 Nick + HarryMc + JamesH

  • 2025-07-13 Nick + Roy + JamesH + JamesStewart

TODO:

  • Second Tuesday 2024-02-13: Committee ops/handover

    • Spacecubed venue contact/liason
      • done: ACTION: Dan: Contact Alastair
      • Call out on main plug@plug list?
    • Sending email from role addresses:
    • gcalendar
    • youtube: (JamesB phone contact)
    • meetup
    • mailing lists+archives
    • ugmm groups / LDAP: Done
    • membership processing
    • bank signatories
    • website updates
      • including redeploy ugmm
    • passwordstore.org / pass(1): ACTION: Ben, Nick
    • backups
    • done: event promotion! win a Pi5!
    • update Facebook admins:
      • done: JamesH add Sarah
    • update linkedin admins:
    • X/twitter admins:
      • Settings->Security&AccountAccess->Delegation
      • current: JamesH, PatrickC, Niall
  • Can't do a remote talk without BBB?

  • Upcoming committee Tuesday 2023-04-18

  • Check certs

  • Pay

    • digitalocean.com (Wings' team PLUG), autopayment first attempted 2023-04-01
    • AWS, autopayment first attempted 2023-04-03
All previous months were paid with card ending 1216.
This card appears to have expired as the payment for March 2023, ($55.24) Has not been paid. 
​​​​* Done! with BendigoBank card expires 2023-06
  • rsync.net, payment due now/2023-04 , but expired card noticed at 2022-05-15 , autopayment first attempted 2023-04-09
    • Done! with BendigoBank card expires 2023-06
  • Raspberry Jam claim (Niall filled out form to receive camera and magazine. will bring to events when they arrive)
Date: Wed, 5 Apr 2023 11:15:25 -0400
From: Matt Richardson <matt@raspberrypi.com>
Subject: [plug-ctte] March Raspberry Jam gift

Date: Sun, 09 Apr 2023 10:13:31 +0000
From: Niall <niall.navin@protonmail.com>
Subject: Re: [plug-ctte] March Raspberry Jam gift
​​​​* Done! 
  • Replace lastpass with https://www.passwordstore.org/ , test for admin/committee
  • Cleanup cloud users
  • (DONE) Deal with UGMM errors
    • This isn't strictly finished as there are still warnings on the signup completion page, but we are ready to move forward.
  • (DONE) nginx
    • (DONE) commit change of maps
    • (DONE) Add lists redirect for / -> /mailman/listinfo
    • (DONE) Add lists redirect for HTTP -> HTTPS
    • (DONE) ugmm redirects in main site config
  • (DONE) Point mumble at new wildcard cert
    • tested: but unless we start with murmurd -wipeSSL it still uses a self-signed certificate?!
    • tested working after dpkg-reconfigure mumble-server ?
    • Plan is to wipe Mumble on the day
    • Issue: backups were done, but the wrong ones were pruned: WIP ones were retained but working test states were not
  • (NEEDS TEST) Migrate mail
    We only need to migrate a couple of mailboxes
    We'll keep the remaining ones available in the backups of Power.
  • (DONE) Migrate /etc/aliases
  • (DONE) Daily backups - /etc/cron.daily/
  • (DONE) SSL cert for lists,lists2
  • (DONE) Fix broken PLUG logo on https://www.plug.org.au/ugmm/signup and check other pages
  • spamassassin not enabled after a reboot: fixed

TODO: continued

Notes from Wings 2020-11-01:

example: opendkim-genkey -r -s myselector -b 2048 -d example.com

re-generate key: opendkim-genkey -r -s mail -b 2048 -d po1.plug.org.au

Disabled DKIM

Gen aliases for mailman: glass@edison:/usr/lib/mailman/bin$ sudo ./genaliases

Set passwd for glass: riots stunt triple thongs

enable spamassassin to run on boot: glass@edison:~$ sudo systemctl enable spamassassin

TODO: Later

  • Later
    • fail2ban
      • Regular spam? SASL logins from 45.142.120.53 , 45.142.120.74 , .121 .192 since 2020-10-18 rebuild
        • 141.98.10.136 ?
    • DKIM, testing on po1.plug.org.au
      ​​​​/etc/postfix/master.cf:
      ​​​​# OpenDKIM stuff
      ​​​​milter_default_action = accept
      ​​​​milter_protocol = 6
      ​​​​# from inside the chroot, the socket will be in /var/run/opendkim
      ​​​​smtpd_milters = unix:/var/run/opendkim/opendkim.sock
      ​​​​non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock
      
      ​​​​root@edison:/etc/dkimkeys# ls -l /etc/dkimkeys/mail.*
      ​​​​-rw------- 1 root root 1679 Nov  1 12:20 /etc/dkimkeys/mail.private
      ​​​​-rw------- 1 root root  514 Nov  1 12:20 /etc/dkimkeys/mail.txt
      
      ​​​​Nov  1 12:24:47 edison opendkim[6119]: can't load key from /etc/dkimkeys/mail.private: Permission denied
      ​​​​Nov  1 12:24:47 edison opendkim[6119]: BDF6563AF2: error loading key 'mail._domainkey.po1.plug.org.au'
      ​​​​Nov  1 12:24:47 edison postfix/cleanup[6411]: BDF6563AF2: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 4.7.1 Service unavailable - try again later; from=<glass@po1.plug.org.au> to=<zorlin@gmail.com>
      
    • Regenerate the mailman archives
      • regeneration skipped: will change message numbers if fix-mbox.pl is run on plug.mbox, for example
      • (ISSUES DURING TEST) Migrate mailman archives
        We ran into some invalid emails (which contained some special strings - leading "From "). @niceness looked into them and was able to correct them, but the corrections left holes in the archives which screwed up numbering.
        We plan to address this eventually by padding the invalid emails with dummy ones (perhaps with an explanation of what happened).
        For now we are going to take the latest copy from Power and resume from there.

/var/lib/mailman/lists
/var/lib/mailman/archives

jekyll workaround

There is an old version of Jekyll in Debian 9 which is not really compatible with our jekyll builds. (e.g. post generation in committee minutes)

So apt uninstall jekyll, then install ruby2.3 and ruby2.3-dev, then sudo gem install jekyll.

You specifically need version 3.8.5

sudo gem install jekyll -v 3.8.5

This should fix builds. It won't be necessary once we move to Buster (Debian 10).

knot restore

  • restoring /etc/knot from backups
    • repo contains only /etc/knot/plug.org.au.example.zone , not the live data:
      • _acme-challenge.plug.org.au.zone
      • _acme-challenge.po1.plug.org.au.zone
      • plug.org.au.zone

Maintenance email

Hi PLUG,

From 12pm until 5pm tomorrow (2020-06-14 12pm-5pm AWST) we will be performing maintenance on PLUG infrastructure. This will mean service interruptions to all services including web, email and membership management.

We'll be minimizing the impact as much as possible but some downtime will be necessary.

Cutover checklist

  • System maintenance downtime warning
  • Change all references in Ansible from "lists2" to "lists" and do a run (first dry, then normal).
  • Move www.plug.org.au A record to point to Edison
  • Shut down mail services + ugmm + mailman
sudo service dovecot stop
sudo service exim4 stop
sudo service postfix stop
sudo service apache2 stop
sudo service mailman stop
  • Run the LDAP backup script
    /etc/cron.daily/20-ldapdump
  • Run /root/bin/borgauto.sh on power
  • Run rsync delete to pluck /var/lib/mailman from power
  • Shut down power
  • Run the brestore script to pluck data
    • Inject test email, see outgoing message in queue+pipermail archive
  • Check results
  • Move plug.org.au A record to point to Edison

Mailman Migration

New way - use @NB script
root@bayonet:~/bin# ./brestore.sh --mount-all

Restore lists from backup (with cp), OR

cd /tmp/latest-power/var/lib/mailman/lists/
cp -R * /var/lib/mailman/lists/

Restore lists from backup (with rsync)

rsync -av /tmp/latest-power/var/lib/mailman/lists/ /var/lib/mailman/lists/

FIXME: has left a number of pending.pck.tmp.* files

Fix surface-level permissions and ownership.

chown root:list /var/lib/mailman/lists/*
chmod 2775 /var/lib/mailman/lists/*

Restore archives from backup

cd /tmp/latest-power/var/lib/mailman/archives/private
sudo cp -R *.mbox /var/lib/mailman/archives/private/
chown root:list /var/lib/mailman/archives/private/*
chmod 2775 /var/lib/mailman/archives/private/*

Recreate archives from mailboxes (SKIP FOR NOW)
We run plug last as it is the biggest and hardest to build (SKIP FOR NOW)
(We are NOT recreating the archives at this stage)

cd /var/lib/mailman/archives/private
sudo /var/lib/mailman/bin/arch admin
sudo /var/lib/mailman/bin/arch av
sudo /var/lib/mailman/bin/arch committee
sudo /var/lib/mailman/bin/arch hackers
sudo /var/lib/mailman/bin/arch jobs
sudo /var/lib/mailman/bin/arch mailman
sudo /var/lib/mailman/bin/arch off-topic
sudo /var/lib/mailman/bin/arch ugmm
sudo /var/lib/mailman/bin/arch userconf
sudo /var/lib/mailman/bin/arch plug

Fix URLs and make lists properly appear on frontpage where appropriate

cd /var/lib/mailman/lists
sudo withlist -l -r fix_url admin
sudo withlist -l -r fix_url av
sudo withlist -l -r fix_url committee
sudo withlist -l -r fix_url hackers
sudo withlist -l -r fix_url jobs
sudo withlist -l -r fix_url mailman
sudo withlist -l -r fix_url off-topic
sudo withlist -l -r fix_url plug
sudo withlist -l -r fix_url ugmm
sudo withlist -l -r fix_url userconf

(DONE) nginx old ugmm to new

https://stackoverflow.com/questions/22224441/nginx-redirect-all-requests-from-subdirectory-to-another-subdirectory-root/22261287

In the end we decided to point old UGMM to new UGMM with a "dumb" redirect to help prevent certain kinds of attacks. IE: any old UGMM URL will redirect to https://ugmm.plug.org.au/, with no arguments or paths brought across.

mailman archive transplant

We want to transplant the current set of mailman archives, instead of regenerating them (for various reasons - bad emails and avoid re-numbering). Once imported, we want to run fix_urls to ensure that they are consistent and that the links work.

https://docs.borgbase.com/restore/borg/

/var/lib/mailman/archives

borg extract list dry-run $BORG_REPO::'power.plug.org.au-2020-07-17 06:47:14.255843' /var/lib/mailman/archives

root@power:~/.ssh# time rsync --delete -e 'ssh -i /root/.ssh/borgkey' -az --stats /var/lib/mailman/archives/. root@edison.plug.org.au:/tmp/power_var_lib_mailman_archives/.

Number of files: 168440
Number of files transferred: 0
Total file size: 2475392433 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3253302
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3277980
Total bytes received: 24673

sent 3277980 bytes  received 24673 bytes  188723.03 bytes/sec
total size is 2475392433  speedup is 749.52

real	0m17.654s
user	0m0.380s
sys	0m1.520s

test cases

main website

sed -n '/test cases/,$ p' checklist.md.txt |grep http > urls.tocheck.txt

http://plug.org.au/ -> https://plug.org.au/
http://www.plug.org.au/ -> https://www.plug.org.au/
https://plug.org.au/
https://www.plug.org.au/
https://plug.org.au/resources/
https://plug.org.au/contact/
https://plug.org.au/events/
https://plug.org.au/events/archive/
https://plug.org.au/events/2004/
https://plug.org.au/events/committee/2020/04-21/
https://www.plug.org.au/contact/
https://www.plug.org.au/events/
https://www.plug.org.au/events/archive/
https://www.plug.org.au/events/2004/

ugmm

https://www.plug.org.au/ugmm/memberself ->
https://www.plug.org.au/ugmm/ ->
http://www.plug.org.au/ugmm/memberself ->
http://www.plug.org.au/ugmm/ ->
http://ugmm.plug.org.au/ -> https://ugmm.plug.org.au/
https://ugmm.plug.org.au/

mailman

http://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/admin/committee
http://lists.plug.org.au/mailman/admin/committee/members
http://lists.plug.org.au/ -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/ -> https://lists.plug.org.au/mailman/listinfo
?? http://lists.plug.org.au/mailman/listinfo/ -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/private/committee/
http://lists.plug.org.au/mailman/private/committee/2020-January.txt.gz
http://lists.plug.org.au/mailman/private/committee/2019-May/thread.html
http://lists.plug.org.au/mailman/private/committee/2010-August/000031.html
(spam) http://lists.plug.org.au/mailman/private/committee/2010-August/000041.html
http://lists.plug.org.au/mailman/listinfo/admin
http://lists.plug.org.au/mailman/listinfo/av
http://lists.plug.org.au/mailman/listinfo/committee
http://lists.plug.org.au/mailman/listinfo/hackers
http://lists.plug.org.au/mailman/listinfo/jobs
http://lists.plug.org.au/mailman/listinfo/off-topic
http://lists.plug.org.au/mailman/listinfo/mailman
http://lists.plug.org.au/mailman/listinfo/plug
http://lists.plug.org.au/mailman/listinfo/ugmm
http://lists.plug.org.au/mailman/listinfo/userconf
http://lists.plug.org.au/pipermail/jobs/2015-March/000015.html
http://lists.plug.org.au/pipermail/plug/
http://lists.plug.org.au/pipermail/plug/2020-April/thread.html
http://lists.plug.org.au/pipermail/plug/2020-April/084366.html
http://lists.plug.org.au/pipermail/plug/2024-May/084948.html

# ls /var/lib/mailman/archives/*
/var/lib/mailman/archives/private:
admin       av.mbox         hackers       jobs.mbox     off-topic       plug.mbox  userconf
admin.mbox  committee       hackers.mbox  mailman       off-topic.mbox  ugmm       userconf.mbox
av          committee.mbox  jobs          mailman.mbox  plug            ugmm.mbox

/var/lib/mailman/archives/public:
av  jobs  mailman  off-topic  plug  ugmm

mail

We can use defer_transports to safely test email (and manually approve).


Notes POSH+PLUG hack day 2023-07-09

  • Done at PLUG 2nd-Tuesday infra: 2023-08-08
  • Intro to Ansible/getting started: ad-hoc -> collections (modules)/playboks
$ ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa -i xxx admin@power.plug.org.au
The authenticity of host 'power.plug.org.au (54.252.97.56)' can't be established.
RSA key fingerprint is SHA256:3yCzV9ETTz1Wge9etcYTbibxozM3Hxmi3sNu+6Xedxs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

  • 2025-03-13 update: onboard new admins
    • create power.plug.org.au account
laptop$ ssh-keygen -t rsa plug2025
enter passphrase
copy .pub to doc below
cat >> ~/.ssh/config
Host power power.plug.org.au
	# RSA-SHA1 deprecated, use rsa-sha2 or newer # Legacy: HostKeyAlgorithms +ssh-rsa
	# see sshd_config(5) ; ssh -Q key
#@power.plug.org.au
#Host *.plug.org.au
#	User foo
       IdentityFile ~/.ssh/identity-plug
       HostKeyAlgorithms +ssh-rsa
       PubkeyAcceptedAlgorithms +ssh-rsa

power# adduser --uid 10566 zlqrvx
Enter new UNIX password: 
Retype new UNIX password: 
...
power# su - zlqrvx
$ ssh power.plug.org.au
yes
^C
cd .ssh
echo 'ssh-rsa AAAAB3NzaC1yc2EA...' > authorized_keys
$ exit
# adduser zlqrvx sudo

2023-08-08 plans for moving forward:

  • Review site.yml on github.
    • fork it for debian 12 and test it
    • comment out what is not working and keep trying.
    • go back and try and get what was commented out working.

2023-08-13 PLUG Hack

  • Ansible + AWS
    • Created user called ansible
    • Gave it a custom IAM policy that allowed:
      • EC2 Describe, Start, Stop, Terminate, Run (Creates and starts instance)
      • EC2 Describe, create and delete tags
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EC2InstanceManagement",
            "Effect": "Allow",
            "Action": [
                "ec2:Describe*",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:RunInstances",
                "ec2:*Tags"
            ],
            "Resource": "*"
        }
    ]
}
manually launched t2.nano: ssh -v admin@13.239.40.47
sudo apt update; sudo apt install -y eatmydata
time sudo eatmydata apt install -y ansible python3-boto ansible-core ansible-lint apt-file python3-botocore python3-boto3

task 1 returns quickly (~2s); instance startup ~150s

ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path.
ec2: -> amazon.aws.ec2_instance:

Notes 2023-03-19

  • run backups: DONE
  • ugmm: test password reset: DONE
  • ugmm: update committee group in LDAP, visible in ugmm: DONE
  • fail2ban
    • TODO: ugmm logins
    • fast-mail-bomber
      • TODO: commit this to plug-services
      • Updated to ban hits on mailman.
      • 3 hits in 1hr
      • ban for 1000000sec
root@power:/etc/fail2ban# git whatchanged 0efe0d2..f58a611
commit f58a6118c66081b317ee3ff289eaf6d30e1469b0
Author: root <root@plug.org.au>
Date:   Sun Mar 19 15:19:04 2023 +0800

    Thwart the fast mail bomber Nick, Niall

:100755 100755 cc78361... 61c73c0... M  .etckeeper
:000000 100644 0000000... e672809... A  fail2ban/filter.d/apache-fmb-plug.conf
:100644 100644 69f4902... 5db7991... M  fail2ban/jail.local

# fail2ban-client status apachefmb
# fail2ban-client get apachefmb actionunban 180.150.90.127

- ignore list is clear at startup; start and add two addresses:
# service fail2ban start;fail2ban-client set apachefmb addignoreip 180.150.90.127 ;fail2ban-client set apachefmb addignoreip 54.252.97.56

Notes 2023-11-21 Nick + James

Date: Tue, 21 Nov 2023 11:50:56 +0800
From: Nick Bannon <nick@ucc.gu.uwa.edu.au>
To: committee@plug.org.au
Subject: In-person ops meeting 2023-11-21 ... ops vs infra vs projects

[...]
Let's be practical and split up the tasks, making sure that:

Notes 2024-02-11 Nick, Ben

password-store

Moved to GitHub's repository README.md
https://github.com/asynthe/plug-pass

Google and Meetup Calendar updates

Notes 2024-04-14 Niall, Nick, Ben

  • backups: old AWS hosts, power, edison
  • website deploy
    • plus ugmm build
    • checklist:
      • was mostly automated as full machine build https://github.com/plugorgau/plug-services
        • if you're confident to rebuild the whole machine in case of issues, it's easy! but if you're working it out, wanted to see what has changed, one can go deep
      1. check website is working, before and after
        • test URLs above: fangs/minilandl webcheck.py
        • SSL expiring? ACME now running on edison: ansible: tls-copy-edison2power
      2. check backups have been running OK, or do one now
      3. build with hugo https://github.com/plugorgau/plugorgau.github.io/blob/master/README.md
      4. diff -ur /tmp/latest /home/plug/public_html
      5. rsync -av /tmp/latest/. /home/plug/public_html/.
        • ownership/perms
        • --delete
        • check/diff for deleted/changed files against: cp -al /home/plug/public_html web.$(date +%Y%m%d)
      6. https://github.com/plugorgau/ugmm v0.5 / manually-installed parts of v0.5.2 is part of the website
        • build, check live URLs with webcheck.py, diffoscope against plug-ugmm_0.5_all.deb
        • build in stretch chroot?
          • time mmdebstrap --mode=proot --variant=apt --include=build-essential stretch debian9-stretch.tar https://archive.debian.org/debian
          • mkdir -p cache; chmod 1777 cache; time mmdebstrap --mode=proot --variant=apt --include=build-essential --skip=essential/unlink --setup-hook='mkdir -p ./cache "$1"/var/cache/apt/archives/' --setup-hook='sync-in ./cache /var/cache/apt/archives/' --customize-hook='sync-out /var/cache/apt/archives ./cache' stretch debian9-stretch3.tar https://archive.debian.org/debian
  • prune power backup objects: add lifecycle
    • AWS - S3 bucket 'plug-us' created 2012 had no objects held in it. - Deleted by Niall
  • vinyl/t-shirt printing

Ubuntu snaps

​​​​Distributor ID:	Ubuntu
​​​​Description:	Ubuntu 20.04.6 LTS
​​​​Release:	20.04
​​​​Codename:	focal
  • Open firefox, browse a few pages, surprise! an update is available.
    • Can no longer open new tabs:
    Restart Required: Restart to Keep Using Firefox
    An update to Firefox started in the background. You’ll need to restart to finish the update.
    Your windows and tabs will be quickly restored, but private ones will not.
    [Restart Firefox]
  • Normal apt-based upgrades also happening (including snapd):
    Software Updater: Updated software is available for this computer. Do you want to install it now?
    The computer also needs to restart to finish installing previous updates.

Notes 2024-05-14 Nick, BenIDS, Sarah, MarkW, Owain, LawrenceL

Notes 2024-06-09 Nick, MarkW

Notes 2024-09-10 Nick + MichaelC + JamesH + DanB

  • Membership processing

  • edison rebuild in AWS?

  • events

  • pcadmin has account on power.plug.org.au @AWS

    • power is very old, login with RSA keys only (or old DSA), needs ssh -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes +ssh-rsa
Host power.plug.org.au
    HostName power.plug.org.au
    User xxxx
    Port 22
    HostKeyAlgorithms ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa
    IdentityFile ~/.ssh/plug_2024_2
    IdentitiesOnly yes
  • Create accounts for jamesh, pcadmin, dan on plug.perthchat.org / plug.michael5ollins.com @PC-HQ
  • Create accounts for jamesh, pcadmin, dan on edison.plug.org.au ( @DigitalOcean )
  • Create account for jamesh, pcadmin, dan on https://plug.signin.aws.amazon.com/console
plug.perthchat.org:/home/pcadmin/process_memberships $ ls -l
-rwxr-xr-x 1 pcadmin pcadmin 1843 Sep 12  2023 extract-payments.py
-rw-r--r-- 1 pcadmin pcadmin 1480 May 29 20:16 fetch.yml

Agenda/minutes for next committee meeting:

edison rebuild in AWS

  • https://plug.signin.aws.amazon.com/console
  • test instance t3a.micro : http://13.210.141.88/
    • question: how do we bootstrap letsencrypt/certbot? how does it know what certs we want? read them out of nginx config?
  • Nick, Michael: shutdown edison droplet, resize to USD$12/month droplet (cannot go smaller with the current 50GiB storage)

2024-10-13

  • http://lists.plug.org.au/mailman/listinfo/

  • PLUG Website build on Power:

    • Set up a working copy of hugo on Power that can be run via /home/admin/hugo/hugo.sh
    • Can build the website if I comment out enableGitInfo = true in config.toml: Hugo's invocation of Git uses arguments not supported by the ancient version.
    ​​git clone https://github.com/plugorgau/plugorgau.github.io.git
    ​​cd plugorgau.github.io
    ​​sed -i 's/^enableGitInfo/#\0/' config.toml
    ​​../hugo/hugo.sh
    ​​rsync -a public/ /home/plug/public_html/
    
  • (plus 2025-02-18 updates, all done as admin@power user)

2024-11-10

  • website deploy?
    • ssh power, removed old users from admin,adm group
  • mailman/mailing list web UI
    • jayasekerakushan@gmail subscribing
    • Brave, Chrome on Android, not Chrome Desktop?: http ignored went to https
    • Mail sent from power->gmail, ended up in spam, reported not spam
      • Nov 10 14:40:27 power postfix/smtp[4870]: 3727D465C4: to=<jayasekerakushan@gmail.com>, relay=gmail-smtp-in.l.google.com[64.233.170.27]:25, delay=3, delays=0.01/0.01/1.6/1.4, dsn=2.0.0, status=sent (250 2.0.0 OK 1731221134 98e67ed59e1d1-2e9a5f5bc8fsi8807865a91.57 - gsmtp)
      • Saved from spam, self-confirm link, also http!
      • then the request shows up in http://lists.plug.org.au/mailman/admindb/plug for moderator approval:
        • Subscription Requests Address/name Defer Approve Reject Discard
  • mailman issue: upgrade to header munge to avoid conflicts with tight DMARC policies

2024-11-10 Nick + Wyatt

2025-01-07 Nick + Wyatt

  • Infra (technical, specific small tasks)
    • Issue with domains not appearing in emails: strict DMARC domain policies that prevent delivery
      • for DOMAINNAME in beckwith.net.au danscomp.net decisions-and-designs.com.au fnarfbargle.com iinet.net.au kenworthy.id.au mccormick.cx oranges.id.au stgeorge.com.au plug.org.au;do host -tTXT ${DOMAINNAME};host -tTXT _dmarc.${DOMAINNAME};done
      • mailman upgrade 2.x or 3.x to munge headers to come "from" us
    • To try:
      • turn on https for all links, run the link checker before and after
      • deploy a website update: edit, git commit/push/pull, hugo build
      • build of website with Github actions?
  • Projects (larger)
    • AV hardware (grants?)/software refresh (self-host BBB? self-starting OBS system?)
    • Share infrastructure with LA or other LUGs
  • Ops (Committee/organisational) for "What's on for PLUG in 2025-01": AGM prep, Committee meeting, PLUG-in-the-Pub
    • Wyatt will phone/speak around Wednesday 2025-01-08 to the committee members that haven't responded yet, confirm AGM hosts
    • State/Federal elections coming soon!

2025-02-18 Nick + Wyatt + JamesH

  • Website upload: JamesH
  • Wyatt login ssh to power, create SSH key, deploy website to live

2025-03-09 Nick + JamesH + MarkW

  • Processed video for Jasper's talk from September
  • Update to https://plugorgau.github.io/projects/video/network/
  • subscribe TimL
  • Update TLS cert with ansible: tls-copy-edison2power
  • Fetch URLs: webcheck.py - test before/after:
    • website deploy?
    • full HTTPS config?

2025-04-13 Nick + MarkW + JamesH + TimL

  • Fetch URLs: webcheck.py - test before/after:
    • website deploy?
    • full HTTPS config?

301 Moved Permanently,https://plug.org.au/membership -> https://plug.org.au/membership/,308

https://httpd.apache.org/docs/2.4/en/mod/mod_alias.html#redirect

  • James: fast-mail-bomber: can we blackhole the http version of these mailman form pages?
../mailman/pending-subscription-request-policy-spam/github.com/juzeon/fast-mail-bomber/raw/master/data$ grep plug.org nodes.20230213.json 
    "http://lists.plug.org.au/mailman/subscribe/av",
    "http://lists.plug.org.au/mailman/subscribe/jobs",
    "http://lists.plug.org.au/mailman/subscribe/off-topic",
    "http://lists.plug.org.au/mailman/subscribe/plug",
    "http://lists.plug.org.au/mailman/subscribe/ugmm",
    "http://lists.plug.org.au/mailman/subscribe/userconf",

/etc/apache2/sites-available/lists.plug.org.au
/etc/apache2/sites-available/plug.linux.org.au

/etc/apache2/sites-available/plug.org.au.conf
/etc/apache2/sites-available/plug.org.au-ssl.conf

power# grep 'lists.plug.org.au:443.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l
43
power:# grep 'lists.plug.org.au:80.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l
98

port80 -> 403, due to new "Deny from all" on "Location /mailman/subscribe/""
port443 -> 401, due to AuthType Basic on /mailman
# vi /etc/mailman/mm_cfg.py
...
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
power# service mailman restart

some http link references in config text:
# /var/lib/mailman/bin/config_list -o - plug|less

<li><a href="http://lists.plug.org.au/pipermail/plug/">Go to list archives</a><br>&nbsp;<br>
  • 2025-06-08 Nick + HarryMc + JamesH + JamesStewart + MarkWalker + DanB
    • borgbackup on power, run, fix with BORG_REMOTE_PATH=borg14 to rsync.net, test create and extract