PLUG Hack Day Notes

# PLUG Hack Day 2020-06-13 + 2020-06-14 + 2023-03-19 + 2023-04-09 + 2023-07-09 + 2023-08-08 + 2023-09-10/12 2023-11-21 2024-02-11 2024-02-13 2024-04-14 2024-05-14 2024-06-09 2024-09-08 2024-10-13 2024-11-10 2024-12-08 2025-01-07 2025-02-18 2025-03-09 2025-04-13 2025-06-08 2025-07-13 2025-08-10 2025-09-14 `(update title when jamesh/harry signed in)` - this document: https://hackmd.io/@plug/hack-day-notes - ( alias of https://hackmd.io/_o_65OZbQMin0ANI2lz6-g ) - `wget https://hackmd.io/_o_65OZbQMin0ANI2lz6-g/download -O $(date +%Y-%m-%d)-infra.md` - `(FNAME=$(date +%Y-%m-%d)-infra.md; DOC=hack-day-notes; CODIMD_SERVER=https://hackmd.io/@plug codimd export --md $DOC "./$FNAME")` - was https://meetings.ucc.asn.au/b/nic-4cd-3jg - 2020 Nick+BenjaminA - 2023-03-19 , 2023-04-09 Nick+Niall - 2023-07-09 Nick + Michael + Niall + James + BenjaminIDS - 2023-08-08 Niall + Nick + Aiden + Dylan + Craig + James + Harry - 2023-08-13 Niall + Nick + Jason - 2023-09-10/12 Nick + Michael + BenjaminIDS + JasperG + JamesH - 2023-11-21 Nick + James - 2024-02-11 Niall + DanB + Nick + BenjaminIDS - 2024-02-13 BenjaminIDS + Dan + Nick + Sarah + James - 2024-04-14 Niall + Nick + BenjaminIDS - 2024-05-14 Nick, BenIDS + Sarah + MarkW + Owain + LawrenceL - 2024-06-09 Nick + MarkW - 2024-09-10 Nick + MichaelC + JamesH + DanB - 2024-10-13 Harry + Nick + Jacek + JamesH + Dan + Mark - 2024-11-10 Nick + Dan - 2024-11-10 , 2025-01-07 Nick + Wyatt - 2025-02-18 Nick + Wyatt + JamesH - 2025-03-09 Nick + JamesH + MarkW - 2025-04-13 Nick + MarkW + JamesH + TimL - 2025-06-08 Nick + HarryMc + JamesH - 2025-07-13 Nick + Roy + JamesH + JamesStewart - 2025-08-10 Nick + Harry + JamesStewart + LucasP + PeterSz + DanB + Mathison + Alexander - 2025-09-14 Nick + JamesStewart + JamesH ### TODO: * Second Tuesday 2024-02-13: Committee ops/handover * Spacecubed venue contact/liason * done: ACTION: Dan: Contact Alastair * Call out on main plug@plug list? * Lawrence Lau <drlawrencelau@gmail.com> is willing@2024-05-14 , Spacecubed hot-desker * Sending email from role addresses: * https://support.google.com/mail/answer/22370 * gcalendar * youtube: (JamesB phone contact) * meetup * mailing lists+archives * ugmm groups / LDAP: Done * membership processing * bank signatories * website updates * including redeploy ugmm * passwordstore.org / pass(1): ACTION: Ben, Nick * https://github.com/asynthe/plug-pass * backups * rsync.net debit card * PCHQ * done: event promotion! win a Pi5! * update Facebook admins: * done: JamesH add Sarah * update linkedin admins: * https://www.linkedin.com/groups/3765623 * current: Euan, Luke, PeterL, JasonN, Alastair * add: BenIDS, DanB * X/twitter admins: * Settings->Security&AccountAccess->Delegation * current: JamesH, PatrickC, Niall * Can't do a remote talk without BBB? * Fix old BBB v2.3 meetings.ucc.asn.au ? * vs set up our own ephemeral one? * vs jitsi * vs jami * vs demo.bigbluebutton.org (no provided recording, 60 minute limit) * Upcoming committee Tuesday 2023-04-18 * Check certs * Pay * digitalocean.com (Wings' team PLUG), autopayment first attempted 2023-04-01 * AWS, autopayment first attempted 2023-04-03 ``` All previous months were paid with card ending 1216. This card appears to have expired as the payment for March 2023, ($55.24) Has not been paid. ``` * Done! with BendigoBank card expires 2023-06 * rsync.net, payment due now/2023-04 , but expired card noticed at 2022-05-15 , autopayment first attempted 2023-04-09 * Done! with BendigoBank card expires 2023-06 * Raspberry Jam claim (Niall filled out form to receive camera and magazine. will bring to events when they arrive) ``` Date: Wed, 5 Apr 2023 11:15:25 -0400 From: Matt Richardson <matt@raspberrypi.com> Subject: [plug-ctte] March Raspberry Jam gift Date: Sun, 09 Apr 2023 10:13:31 +0000 From: Niall <niall.navin@protonmail.com> Subject: Re: [plug-ctte] March Raspberry Jam gift ``` * Done! * Replace lastpass with https://www.passwordstore.org/ , test for admin/committee * Cleanup cloud users * (DONE) Deal with UGMM errors * This isn't strictly finished as there are still warnings on the signup completion page, but we are ready to move forward. * (DONE) nginx * (DONE) commit change of maps * (DONE) Add lists redirect for / -> /mailman/listinfo * (DONE) Add lists redirect for HTTP -> HTTPS * (DONE) ugmm redirects in main site config * (DONE) Point mumble at new wildcard cert * tested: but unless we start with `murmurd -wipeSSL` it still uses a self-signed certificate?! * tested working after `dpkg-reconfigure mumble-server` ? * Plan is to wipe Mumble on the day * Issue: backups were done, but the wrong ones were pruned: WIP ones were retained but working test states were not * (NEEDS TEST) Migrate mail We only need to migrate a couple of mailboxes We'll keep the remaining ones available in the backups of Power. * (DONE) Migrate /etc/aliases * (DONE) Daily backups - /etc/cron.daily/ * (DONE) SSL cert for lists,lists2 * (DONE) Fix broken PLUG logo on https://www.plug.org.au/ugmm/signup and check other pages * spamassassin not enabled after a reboot: fixed ## TODO: continued ### Notes from Wings 2020-11-01: example: opendkim-genkey -r -s myselector -b 2048 -d example.com re-generate key: opendkim-genkey -r -s mail -b 2048 -d po1.plug.org.au Disabled DKIM Gen aliases for mailman: glass@edison:/usr/lib/mailman/bin$ sudo ./genaliases Set passwd for glass: riots stunt triple thongs enable spamassassin to run on boot: glass@edison:~$ sudo systemctl enable spamassassin ## TODO: Later * Later * fail2ban * Regular spam? SASL logins from 45.142.120.53 , 45.142.120.74 , .121 .192 ... since 2020-10-18 rebuild * 141.98.10.136 ? * DKIM, testing on `po1.plug.org.au` ``` /etc/postfix/master.cf: # OpenDKIM stuff milter_default_action = accept milter_protocol = 6 # from inside the chroot, the socket will be in /var/run/opendkim smtpd_milters = unix:/var/run/opendkim/opendkim.sock non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock root@edison:/etc/dkimkeys# ls -l /etc/dkimkeys/mail.* -rw------- 1 root root 1679 Nov 1 12:20 /etc/dkimkeys/mail.private -rw------- 1 root root 514 Nov 1 12:20 /etc/dkimkeys/mail.txt Nov 1 12:24:47 edison opendkim[6119]: can't load key from /etc/dkimkeys/mail.private: Permission denied Nov 1 12:24:47 edison opendkim[6119]: BDF6563AF2: error loading key 'mail._domainkey.po1.plug.org.au' Nov 1 12:24:47 edison postfix/cleanup[6411]: BDF6563AF2: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 4.7.1 Service unavailable - try again later; from=<glass@po1.plug.org.au> to=<zorlin@gmail.com> ``` * Regenerate the mailman archives * regeneration skipped: will change message numbers if fix-mbox.pl is run on plug.mbox, for example * (ISSUES DURING TEST) Migrate mailman archives We ran into some invalid emails (which contained some special strings - leading "From "). @niceness looked into them and was able to correct them, but the corrections left holes in the archives which screwed up numbering. We plan to address this eventually by padding the invalid emails with dummy ones (perhaps with an explanation of what happened). For now we are going to take the latest copy from Power and resume from there. /var/lib/mailman/lists /var/lib/mailman/archives ## jekyll workaround There is an old version of Jekyll in Debian 9 which is not really compatible with our jekyll builds. (e.g. post generation in committee minutes) So... `apt uninstall jekyll`, then install ruby2.3 and ruby2.3-dev, then sudo gem install jekyll. You specifically need version 3.8.5... `sudo gem install jekyll -v 3.8.5` This should fix builds. It won't be necessary once we move to Buster (Debian 10). ## knot restore - restoring `/etc/knot` from backups - repo contains only `/etc/knot/plug.org.au.example.zone` , not the live data: - _acme-challenge.plug.org.au.zone - _acme-challenge.po1.plug.org.au.zone - plug.org.au.zone ## Maintenance email Hi PLUG, From 12pm until 5pm tomorrow (2020-06-14 12pm-5pm AWST) we will be performing maintenance on PLUG infrastructure. This will mean service interruptions to all services including web, email and membership management. We'll be minimizing the impact as much as possible but some downtime will be necessary. ## Cutover checklist * System maintenance downtime warning * Change all references in Ansible from "lists2" to "lists" and do a run (first dry, then normal). * Move www.plug.org.au A record to point to Edison * Shut down mail services + ugmm + mailman ``` sudo service dovecot stop sudo service exim4 stop sudo service postfix stop sudo service apache2 stop sudo service mailman stop ``` * Run the LDAP backup script `/etc/cron.daily/20-ldapdump` * Run `/root/bin/borgauto.sh` on power * Run rsync --delete to pluck /var/lib/mailman from power * Shut down power * Run the brestore script to pluck data * Inject test email, see outgoing message in queue+pipermail archive * Check results * LDAP migration * FIXME: LDAP dump/restore, replace with a rebuild from fresh config, e.g. * move BDB->MDB, https://www.adimian.com/blog/2014/10/how-to-enable-memberof-using-openldap/ * edison:/etc/ldap/secure/extra-modules-overlays-schemas * mailbox migration * Check/regenerate the mailman archives * regeneration skipped: will change message numbers if fix-mbox.pl is run on plug.mbox, for example * Move plug.org.au A record to point to Edison ## Mailman Migration New way - use @NB script `root@bayonet:~/bin# ./brestore.sh --mount-all` Restore lists from backup (with cp), OR... ``` cd /tmp/latest-power/var/lib/mailman/lists/ cp -R * /var/lib/mailman/lists/ ``` Restore lists from backup (with rsync) ``` rsync -av /tmp/latest-power/var/lib/mailman/lists/ /var/lib/mailman/lists/ ``` FIXME: has left a number of `pending.pck.tmp.*` files Fix surface-level permissions and ownership. ``` chown root:list /var/lib/mailman/lists/* chmod 2775 /var/lib/mailman/lists/* ``` Restore archives from backup ``` cd /tmp/latest-power/var/lib/mailman/archives/private sudo cp -R *.mbox /var/lib/mailman/archives/private/ chown root:list /var/lib/mailman/archives/private/* chmod 2775 /var/lib/mailman/archives/private/* ``` Recreate archives from mailboxes (SKIP FOR NOW) We run plug last as it is the biggest and hardest to build (SKIP FOR NOW) (We are NOT recreating the archives at this stage) ``` cd /var/lib/mailman/archives/private sudo /var/lib/mailman/bin/arch admin sudo /var/lib/mailman/bin/arch av sudo /var/lib/mailman/bin/arch committee sudo /var/lib/mailman/bin/arch hackers sudo /var/lib/mailman/bin/arch jobs sudo /var/lib/mailman/bin/arch mailman sudo /var/lib/mailman/bin/arch off-topic sudo /var/lib/mailman/bin/arch ugmm sudo /var/lib/mailman/bin/arch userconf sudo /var/lib/mailman/bin/arch plug ``` Fix URLs and make lists properly appear on frontpage where appropriate ``` cd /var/lib/mailman/lists sudo withlist -l -r fix_url admin sudo withlist -l -r fix_url av sudo withlist -l -r fix_url committee sudo withlist -l -r fix_url hackers sudo withlist -l -r fix_url jobs sudo withlist -l -r fix_url mailman sudo withlist -l -r fix_url off-topic sudo withlist -l -r fix_url plug sudo withlist -l -r fix_url ugmm sudo withlist -l -r fix_url userconf ``` ## (DONE) nginx old ugmm to new https://stackoverflow.com/questions/22224441/nginx-redirect-all-requests-from-subdirectory-to-another-subdirectory-root/22261287 In the end we decided to point old UGMM to new UGMM with a "dumb" redirect to help prevent certain kinds of attacks. IE: any old UGMM URL will redirect to https://ugmm.plug.org.au/, with no arguments or paths brought across. ## mailman archive transplant We want to transplant the current set of mailman archives, instead of regenerating them (for various reasons - bad emails and avoid re-numbering). Once imported, we want to run fix_urls to ensure that they are consistent and that the links work. https://docs.borgbase.com/restore/borg/ /var/lib/mailman/archives borg extract --list --dry-run $BORG_REPO::'power.plug.org.au-2020-07-17 06:47:14.255843' /var/lib/mailman/archives ``` root@power:~/.ssh# time rsync --delete -e 'ssh -i /root/.ssh/borgkey' -az --stats /var/lib/mailman/archives/. root@edison.plug.org.au:/tmp/power_var_lib_mailman_archives/. Number of files: 168440 Number of files transferred: 0 Total file size: 2475392433 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 3253302 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 3277980 Total bytes received: 24673 sent 3277980 bytes received 24673 bytes 188723.03 bytes/sec total size is 2475392433 speedup is 749.52 real 0m17.654s user 0m0.380s sys 0m1.520s ``` ## test cases ### main website sed -n '/test cases/,$ p' checklist.md.txt |grep http > urls.tocheck.txt http://plug.org.au/ -> https://plug.org.au/ http://www.plug.org.au/ -> https://www.plug.org.au/ https://plug.org.au/ https://www.plug.org.au/ https://plug.org.au/resources/ https://plug.org.au/contact/ https://plug.org.au/events/ https://plug.org.au/events/archive/ https://plug.org.au/events/2004/ https://plug.org.au/events/committee/2020/04-21/ https://www.plug.org.au/contact/ https://www.plug.org.au/events/ https://www.plug.org.au/events/archive/ https://www.plug.org.au/events/2004/ ### ugmm https://www.plug.org.au/ugmm/memberself -> https://www.plug.org.au/ugmm/ -> http://www.plug.org.au/ugmm/memberself -> http://www.plug.org.au/ugmm/ -> http://ugmm.plug.org.au/ -> https://ugmm.plug.org.au/ https://ugmm.plug.org.au/ ### mailman http://lists.plug.org.au/mailman/listinfo http://lists.plug.org.au/mailman/admin/committee http://lists.plug.org.au/mailman/admin/committee/members http://lists.plug.org.au/ -> https://lists.plug.org.au/mailman/listinfo http://lists.plug.org.au/mailman -> https://lists.plug.org.au/mailman/listinfo http://lists.plug.org.au/mailman/ -> https://lists.plug.org.au/mailman/listinfo ?? http://lists.plug.org.au/mailman/listinfo/ -> https://lists.plug.org.au/mailman/listinfo http://lists.plug.org.au/mailman/private/committee/ http://lists.plug.org.au/mailman/private/committee/2020-January.txt.gz http://lists.plug.org.au/mailman/private/committee/2019-May/thread.html http://lists.plug.org.au/mailman/private/committee/2010-August/000031.html (spam) http://lists.plug.org.au/mailman/private/committee/2010-August/000041.html http://lists.plug.org.au/mailman/listinfo/admin http://lists.plug.org.au/mailman/listinfo/av http://lists.plug.org.au/mailman/listinfo/committee http://lists.plug.org.au/mailman/listinfo/hackers http://lists.plug.org.au/mailman/listinfo/jobs http://lists.plug.org.au/mailman/listinfo/off-topic http://lists.plug.org.au/mailman/listinfo/mailman http://lists.plug.org.au/mailman/listinfo/plug http://lists.plug.org.au/mailman/listinfo/ugmm http://lists.plug.org.au/mailman/listinfo/userconf http://lists.plug.org.au/pipermail/jobs/2015-March/000015.html http://lists.plug.org.au/pipermail/plug/ http://lists.plug.org.au/pipermail/plug/2020-April/thread.html http://lists.plug.org.au/pipermail/plug/2020-April/084366.html http://lists.plug.org.au/pipermail/plug/2024-May/084948.html ``` # ls /var/lib/mailman/archives/* /var/lib/mailman/archives/private: admin av.mbox hackers jobs.mbox off-topic plug.mbox userconf admin.mbox committee hackers.mbox mailman off-topic.mbox ugmm userconf.mbox av committee.mbox jobs mailman.mbox plug ugmm.mbox /var/lib/mailman/archives/public: av jobs mailman off-topic plug ugmm ``` ### mail We can use defer_transports to safely test email (and manually approve). * Send email to zorlin@gmail.com * Send email to benjamin@riff.cc * Receive an email from zorlin@gmail.com * Working * Receive an email from benjamin@riff.cc * Working * Receive a spam email (GTUBE-TEST) from zorlin@gmail.com * Working * UGMM payment reminder goes out (force expire wings) * basic swaks test * `swaks --from glass+test2@po1.plug.org.au --to glass+test2@po1.plug.org.au --server edison.plug.org.au` * `mutt -f ~/Maildir` * Working * get some graphs * https://logit.io/blog/post/top-grafana-dashboards-and-visualisations/#14-ssl-expiry-tracker * https://logit.io/blog/post/top-grafana-dashboards-and-visualisations/#15-aws-billing-estimator * mail throughput, latency, errors * USE / RED / golden4 : mail/web/etc. ---- ### Notes POSH+PLUG hack day 2023-07-09 - Nick + Michael + Niall + James + BenjaminIDS - Niall tally up cloud expenses -> sponsor proposal - AWS payments: remove BA card, update NB card expiry+details - mailman 2to3 - https://gitlab.com/mailman/mailman/-/merge_requests/531 - https://docs.mailman3.org/en/latest/migration.html - Backups: logged into plug.perthchat.org @PCHQ - Updated Meetup calendar with Michael Collins talk on Matrix for September 2023-09-12. Google calendar is populated already. - Benjamin De Silva has been approached to do a talk on EMACS for October 2023-10-10 * Done at PLUG 2nd-Tuesday infra: 2023-08-08 - Intro to Ansible/getting started: ad-hoc -> collections (modules)/playboks ``` $ ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa -i xxx admin@power.plug.org.au The authenticity of host 'power.plug.org.au (54.252.97.56)' can't be established. RSA key fingerprint is SHA256:3yCzV9ETTz1Wge9etcYTbibxozM3Hxmi3sNu+6Xedxs. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes ``` - 2025-03-13 update: onboard new admins - create `power.plug.org.au` account ``` laptop$ ssh-keygen -t rsa plug2025 enter passphrase copy .pub to doc below cat >> ~/.ssh/config Host power power.plug.org.au # RSA-SHA1 deprecated, use rsa-sha2 or newer # Legacy: HostKeyAlgorithms +ssh-rsa # see sshd_config(5) ; ssh -Q key #@power.plug.org.au #Host *.plug.org.au # User foo IdentityFile ~/.ssh/identity-plug HostKeyAlgorithms +ssh-rsa PubkeyAcceptedAlgorithms +ssh-rsa power# adduser --uid 10566 zlqrvx Enter new UNIX password: Retype new UNIX password: ... power# su - zlqrvx $ ssh power.plug.org.au yes ^C cd .ssh echo 'ssh-rsa AAAAB3NzaC1yc2EA...' > authorized_keys $ exit # adduser zlqrvx sudo ``` ## 2023-08-08 plans for moving forward: * Review site.yml on github. - fork it for debian 12 and test it - comment out what is not working and keep trying. - go back and try and get what was commented out working. ## 2023-08-13 PLUG Hack - Ansible + AWS - Created user called ansible - Gave it a custom [IAM policy](https://us-east-1.console.aws.amazon.com/iamv2/home?region=ap-southeast-2#/policies) that allowed: - EC2 Describe, Start, Stop, Terminate, Run (Creates and starts instance) - EC2 Describe, create and delete tags ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "EC2InstanceManagement", "Effect": "Allow", "Action": [ "ec2:Describe*", "ec2:StartInstances", "ec2:StopInstances", "ec2:TerminateInstances", "ec2:RunInstances", "ec2:*Tags" ], "Resource": "*" } ] } ``` ``` manually launched t2.nano: ssh -v admin@13.239.40.47 sudo apt update; sudo apt install -y eatmydata time sudo eatmydata apt install -y ansible python3-boto ansible-core ansible-lint apt-file python3-botocore python3-boto3 task 1 returns quickly (~2s); instance startup ~150s ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path. ec2: -> amazon.aws.ec2_instance: ``` ### Notes 2023-03-19 - run backups: DONE - ugmm: test password reset: DONE - ugmm: update committee group in LDAP, visible in ugmm: DONE - fail2ban - TODO: ugmm logins - fast-mail-bomber - TODO: commit this to plug-services - Updated to ban hits on mailman. - 3 hits in 1hr - ban for 1000000sec ``` root@power:/etc/fail2ban# git whatchanged 0efe0d2..f58a611 commit f58a6118c66081b317ee3ff289eaf6d30e1469b0 Author: root <root@plug.org.au> Date: Sun Mar 19 15:19:04 2023 +0800 Thwart the fast mail bomber Nick, Niall :100755 100755 cc78361... 61c73c0... M .etckeeper :000000 100644 0000000... e672809... A fail2ban/filter.d/apache-fmb-plug.conf :100644 100644 69f4902... 5db7991... M fail2ban/jail.local # fail2ban-client status apachefmb # fail2ban-client get apachefmb actionunban 180.150.90.127 - ignore list is clear at startup; start and add two addresses: # service fail2ban start;fail2ban-client set apachefmb addignoreip 180.150.90.127 ;fail2ban-client set apachefmb addignoreip 54.252.97.56 ``` ### Notes 2023-11-21 Nick + James ``` Date: Tue, 21 Nov 2023 11:50:56 +0800 From: Nick Bannon <nick@ucc.gu.uwa.edu.au> To: committee@plug.org.au Subject: In-person ops meeting 2023-11-21 ... ops vs infra vs projects [...] Let's be practical and split up the tasks, making sure that: ``` - DONE: recent memberships/renewals are all processed - a new mailing list, maybe "members@plug.org.au"? is tested and able to be preloaded with a list of current financial members - also: checked nutmeg's scripts in UGMM - minutes are published and deployed to the website - EOY BBQ+hackday has a plan, I think that's mostly in hand - PLUG-in-the-Pub January is pencilled in - https://hackmd.io/@plug/pubs - Second Tuesday February 2024 and March 2024 have some kind of rough plan: talks? Projects:AV? Pi Jam? - anything else? - test raspberrypi.com ID login - query meetup user with default answers - process recent membership renewals --2023-11-21 - `slapcat | extract-payments.py > payments.$(date +%Y%m%d)a.csv` - https://www.plug.org.au/ugmm/ctte-members?expiredmembers=1 - http://lists.plug.org.au/mailman/listinfo ### Notes 2024-02-11 Nick, Ben #### password-store Moved to GitHub's repository README.md https://github.com/asynthe/plug-pass ##### Google and Meetup Calendar updates - February 2nd Tuesday event updated to reflect operations evening. https://www.meetup.com/perth-linux-users-group-plug/events/298751581/ ### Notes 2024-04-14 Niall, Nick, Ben - backups: old AWS hosts, power, edison - website deploy - plus ugmm build - checklist: - was mostly automated as full machine build https://github.com/plugorgau/plug-services - if you're confident to rebuild the whole machine in case of issues, it's easy! but if you're working it out, wanted to see what has changed, one can go deep 1. check website is working, before and after - test URLs above: fangs/minilandl `webcheck.py` - SSL expiring? ACME now running on `edison`: `ansible`: `tls-copy-edison2power` 2. check backups have been running OK, or do one now 3. build with `hugo` https://github.com/plugorgau/plugorgau.github.io/blob/master/README.md - was built with CI: https://github.com/plugorgau/plugorgau.github.io/blob/master/build-to-gh-pages.sh - copy or `rsync` to `power:/tmp` 4. `diff -ur /tmp/latest /home/plug/public_html ` 5. `rsync -av /tmp/latest/. /home/plug/public_html/.` - ownership/perms - `--delete` - check/diff for deleted/changed files against: `cp -al /home/plug/public_html web.$(date +%Y%m%d)` 6. https://github.com/plugorgau/ugmm v0.5 / manually-installed parts of v0.5.2 is part of the website - build, check live URLs with `webcheck.py`, `diffoscope` against `plug-ugmm_0.5_all.deb` - build in stretch chroot? - `time mmdebstrap --mode=proot --variant=apt --include=build-essential stretch debian9-stretch.tar https://archive.debian.org/debian` - `mkdir -p cache; chmod 1777 cache; time mmdebstrap --mode=proot --variant=apt --include=build-essential --skip=essential/unlink --setup-hook='mkdir -p ./cache "$1"/var/cache/apt/archives/' --setup-hook='sync-in ./cache /var/cache/apt/archives/' --customize-hook='sync-out /var/cache/apt/archives ./cache' stretch debian9-stretch3.tar https://archive.debian.org/debian` - prune `power` backup objects: add lifecycle - AWS - S3 bucket 'plug-us' created 2012 had no objects held in it. - Deleted by Niall - vinyl/t-shirt printing #### Ubuntu snaps Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal - Open firefox, browse a few pages, surprise! an update is available. - Can no longer open new tabs: ``` Restart Required: Restart to Keep Using Firefox An update to Firefox started in the background. You’ll need to restart to finish the update. Your windows and tabs will be quickly restored, but private ones will not. [Restart Firefox] ```` - Normal apt-based upgrades also happening (including snapd): ``` Software Updater: Updated software is available for this computer. Do you want to install it now? The computer also needs to restart to finish installing previous updates. ``` ### Notes 2024-05-14 Nick, BenIDS, Sarah, MarkW, Owain, LawrenceL - https://www.ato.gov.au/businesses-and-organisations/not-for-profit-organisations/your-organisation/in-detail/income-tax/mutuality-and-taxable-income-for-not-for-profits/lodgment-rules-and-tax-rates#ato-Taxrates - Taxable income <$416? and Constitution Objects S3.2 ? - https://www.ato.gov.au/businesses-and-organisations/not-for-profit-organisations/statements-and-returns/in-detail/reporting-requirements-to-self-assess-income-tax-exemption/who-doesnt-need-to-lodge ### Notes 2024-06-09 Nick, MarkW - PLUG programming options? - http://lists.plug.org.au/pipermail/plug/2024-May/084948.html has some including: - IRC matrix bridge (in haskell!) - pipermail archives: - try adding https://public-inbox.org/ ? - running Mailman2's pipermail (single python file) standalone on a new system, maybe Mailman3? - https://mail.python.org/archives/list/mailman-users@python.org/2002/10/?count=200 - https://mail.python.org/archives/list/mailman-users@python.org/thread/PSXRAYZG6JC2KCW5CJ2DF74BEIRHFKZP/ - make http://lists.plug.org.au/pipermail/plug/recent work, like https://lists.debian.org/debian-vote/recent - "report as spam" button, like https://lists.debian.org/debian-vote/2024/05/msg00000.html - get inspired by https://wiki.debian.org/Teams/ListMaster/ListArchiveSpam and https://lists.debian.org/msgid-search - expand eventcheckr / https://github.com/plugorgau/calendar-check - AV project ### Notes 2024-09-10 Nick + MichaelC + JamesH + DanB - Membership processing - `edison` rebuild in AWS? - events - pcadmin has account on `power.plug.org.au` @AWS - power is very old, login with RSA keys only (or old DSA), needs `ssh -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes +ssh-rsa` ``` Host power.plug.org.au HostName power.plug.org.au User xxxx Port 22 HostKeyAlgorithms ssh-rsa PubkeyAcceptedKeyTypes +ssh-rsa IdentityFile ~/.ssh/plug_2024_2 IdentitiesOnly yes ``` - Create accounts for jamesh, pcadmin, dan on `plug.perthchat.org` / `plug.michael5ollins.com` @PC-HQ - Create accounts for jamesh, pcadmin, dan on `edison.plug.org.au` ( @DigitalOcean ) - Create account for jamesh, pcadmin, dan on https://plug.signin.aws.amazon.com/console ``` plug.perthchat.org:/home/pcadmin/process_memberships $ ls -l -rwxr-xr-x 1 pcadmin pcadmin 1843 Sep 12 2023 extract-payments.py -rw-r--r-- 1 pcadmin pcadmin 1480 May 29 20:16 fetch.yml ``` ### Agenda/minutes for next committee meeting: - https://hackmd.io/@plug/committee-minutes-2024-09 - Dan, Nick: events #### `edison` rebuild in AWS - https://plug.signin.aws.amazon.com/console - test instance t3a.micro : http://13.210.141.88/ - question: how do we bootstrap letsencrypt/certbot? how does it know what certs we want? read them out of nginx config? - Nick, Michael: shutdown `edison` droplet, resize to USD$12/month droplet (cannot go smaller with the current 50GiB storage) ## 2024-10-13 - http://lists.plug.org.au/mailman/listinfo/ - PLUG Website build on Power: - Set up a working copy of hugo on Power that can be run via `/home/admin/hugo/hugo.sh` - Can build the website if I comment out `enableGitInfo = true` in `config.toml`: Hugo's invocation of Git uses arguments not supported by the ancient version. ``` git clone https://github.com/plugorgau/plugorgau.github.io.git cd plugorgau.github.io sed -i 's/^enableGitInfo/#\0/' config.toml ../hugo/hugo.sh rsync -a public/ /home/plug/public_html/ ``` - (plus 2025-02-18 updates, all done as `admin@power` user) ## 2024-11-10 - website deploy? - `ssh power`, removed old users from admin,adm group - mailman/mailing list web UI - jayasekerakushan@gmail subscribing - Brave, Chrome on Android, not Chrome Desktop?: http ignored went to https - Mail sent from power->gmail, ended up in spam, reported not spam - `Nov 10 14:40:27 power postfix/smtp[4870]: 3727D465C4: to=<jayasekerakushan@gmail.com>, relay=gmail-smtp-in.l.google.com[64.233.170.27]:25, delay=3, delays=0.01/0.01/1.6/1.4, dsn=2.0.0, status=sent (250 2.0.0 OK 1731221134 98e67ed59e1d1-2e9a5f5bc8fsi8807865a91.57 - gsmtp)` - Saved from spam, self-confirm link, also http! - **then** the request shows up in http://lists.plug.org.au/mailman/admindb/plug for moderator approval: - Subscription Requests Address/name Defer Approve Reject Discard - mailman issue: upgrade to header munge to avoid conflicts with tight DMARC policies ## 2024-11-10 Nick + Wyatt - Wyatt checking UGMM - works! - Wyatt checking committee mailing list - http://lists.plug.org.au/mailman/listinfo/committee - http://lists.plug.org.au/mailman/admin/committee - DMARC: Wyatt (at gmail) has not seen Harry's most recent reply from decisions-and-designs.com.au Re: [plug-ctte] Old Laptops - mailman not munging From: - Michael Barker@Geraldton ex-school laptop recycling HP ProBooks with 2x SODIMM slots, 4th gen CPU - 2024-01-09 previous AGM: https://www.meetup.com/perth-linux-users-group-plug/events/297310367 - cancel 2024-12-10 second Tuesday https://www.meetup.com/perth-linux-users-group-plug/events/302639149/ ## 2025-01-07 Nick + Wyatt - Infra (technical, specific small tasks) - Issue with domains not appearing in emails: strict DMARC domain policies that prevent delivery - `for DOMAINNAME in beckwith.net.au danscomp.net decisions-and-designs.com.au fnarfbargle.com iinet.net.au kenworthy.id.au mccormick.cx oranges.id.au stgeorge.com.au plug.org.au;do host -tTXT ${DOMAINNAME};host -tTXT _dmarc.${DOMAINNAME};done` - mailman upgrade 2.x or 3.x to munge headers to come "from" us - To try: - turn on https for all links, run the link checker before and after - deploy a website update: edit, git commit/push/pull, hugo build - build of website with Github actions? - Projects (larger) - AV hardware (grants?)/software refresh (self-host BBB? self-starting OBS system?) - Share infrastructure with LA or other LUGs - Ops (Committee/organisational) for "What's on for PLUG in 2025-01": AGM prep, Committee meeting, PLUG-in-the-Pub - Wyatt will phone/speak around Wednesday 2025-01-08 to the committee members that haven't responded yet, confirm AGM hosts - State/Federal elections coming soon! ## 2025-02-18 Nick + Wyatt + JamesH - Website upload: JamesH - Wyatt login ssh to `power`, create SSH key, deploy website to live ## 2025-03-09 Nick + JamesH + MarkW * Processed video for Jasper's talk from September * Update to https://plugorgau.github.io/projects/video/network/ * subscribe TimL * Update TLS cert with `ansible`: `tls-copy-edison2power` * Fetch URLs: `webcheck.py` - test before/after: * website deploy? * full HTTPS config? ## 2025-04-13 Nick + MarkW + JamesH + TimL * Fetch URLs: `webcheck.py` - test before/after: * website deploy? * full HTTPS config? 301 Moved Permanently,https://plug.org.au/membership -> https://plug.org.au/membership/,308 https://httpd.apache.org/docs/2.4/en/mod/mod_alias.html#redirect - James: fast-mail-bomber: can we blackhole the http version of these mailman form pages? ``` ../mailman/pending-subscription-request-policy-spam/github.com/juzeon/fast-mail-bomber/raw/master/data$ grep plug.org nodes.20230213.json "http://lists.plug.org.au/mailman/subscribe/av", "http://lists.plug.org.au/mailman/subscribe/jobs", "http://lists.plug.org.au/mailman/subscribe/off-topic", "http://lists.plug.org.au/mailman/subscribe/plug", "http://lists.plug.org.au/mailman/subscribe/ugmm", "http://lists.plug.org.au/mailman/subscribe/userconf", ``` /etc/apache2/sites-available/lists.plug.org.au /etc/apache2/sites-available/plug.linux.org.au /etc/apache2/sites-available/plug.org.au.conf /etc/apache2/sites-available/plug.org.au-ssl.conf - working: http://lists.plug.org.au/pipermail/plug/ - 404 not found: http://lists.plug.org.au/pipermail/plug/ - add TLS for https://lists.plug... plus redirects - check redirects for subscribe apachefmb bot activity: ``` power# grep 'lists.plug.org.au:443.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l 43 power:# grep 'lists.plug.org.au:80.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l 98 port80 -> 403, due to new "Deny from all" on "Location /mailman/subscribe/"" port443 -> 401, due to AuthType Basic on /mailman ``` - `fail2ban` apachefmb is scanning - https://www.passwordstore.org/ - list accounts. services, passwords - borgbackup -> rsync.net , pchq: email admin@plug needs new RSA? key, not ssh-dss - `/root/bin/borgauto.sh` - `/root/bin/borgauto-pchq.sh` - problem: https://lists.plug.org.au/mailman/... is working on HTTPS, but mailman pages contain many pre-generated references to HTTP links, e.g. http://lists.plug.org.au/mailman/admin/plug/members?letter=m from https://lists.plug.org.au/mailman/admin/plug/members - not fixed by s/http/https/ in: ``` # vi /etc/mailman/mm_cfg.py ... DEFAULT_URL_PATTERN = 'https://%s/mailman/' power# service mailman restart some http link references in config text: # /var/lib/mailman/bin/config_list -o - plug|less <li><a href="http://lists.plug.org.au/pipermail/plug/">Go to list archives</a><br> <br> ``` - 2025-06-08 Nick + HarryMc + JamesH + JamesStewart + MarkWalker + DanB - `borgbackup` on `power`, run, fix with `BORG_REMOTE_PATH=borg14` to rsync.net, test create and extract - 2025-09-14 Nick + JamesStewart + JamesH - https://search.google.com/search-console - now domain-validated in DNS with `google-site-verification` TXT record on `plug.org.au` - new round SVG logo PLUG-on-Tux for the youtube channel

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.