# PLUG Hack Day 2020-06-13 + 2020-06-14 + 2023-03-19 + 2023-04-09 + 2023-07-09 + 2023-08-08 + 2023-09-10/12 2023-11-21 2024-02-11 2024-02-13 2024-04-14 2024-05-14 2024-06-09 2024-09-08 2024-10-13 2024-11-10 2024-12-08 2025-01-07 2025-02-18 2025-03-09 2025-04-13 2025-06-08 2025-07-13 2025-08-10 2025-09-14 2025-10-12 2025-11-09 2025-12-14

- this document: https://hackmd.io/@plug/hack-day-notes
- ( alias of https://hackmd.io/_o_65OZbQMin0ANI2lz6-g )
- `wget https://hackmd.io/_o_65OZbQMin0ANI2lz6-g/download -O $(date +%Y-%m-%d)-infra.md`
- `(FNAME=$(date +%Y-%m-%d)-infra.md; DOC=hack-day-notes; CODIMD_SERVER=https://hackmd.io/@plug codimd export --md $DOC "./$FNAME")`

- was https://meetings.ucc.asn.au/b/plug

- 2020 Nick+BenjaminA
- 2023-03-19 , 2023-04-09 Nick+Niall
- 2023-07-09 Nick + Michael + Niall + James + BenjaminIDS
- 2023-08-08 Niall + Nick + Aiden + Dylan + Craig + James + Harry
- 2023-08-13 Niall + Nick + Jason
- 2023-09-10/12 Nick + Michael + BenjaminIDS + JasperG + JamesH
- 2023-11-21 Nick + James
- 2024-02-11 Niall + DanB + Nick + BenjaminIDS
- 2024-02-13 BenjaminIDS + Dan + Nick + Sarah + James
- 2024-04-14 Niall + Nick + BenjaminIDS
- 2024-05-14 Nick, BenIDS + Sarah + MarkW + Owain + LawrenceL
- 2024-06-09 Nick + MarkW
- 2024-09-10 Nick + MichaelC + JamesH + DanB
- 2024-10-13 Harry + Nick + Jacek + JamesH + Dan + Mark
- 2024-11-10 Nick + Dan
- 2024-11-10 , 2025-01-07 Nick + Wyatt
- 2025-02-18 Nick + Wyatt + JamesH
- 2025-03-09 Nick + JamesH + MarkW
- 2025-04-13 Nick + MarkW + JamesH + TimL
- 2025-06-08 Nick + HarryMc + JamesH
- 2025-07-13 Nick + Roy + JamesH + JamesStewart
- 2025-08-10 Nick + Harry + JamesStewart + LucasP + PeterSz + DanB + Mathison + Alexander
- 2025-09-14 Nick + JamesStewart + JamesH
- 2025-10-12 Nick + JohnM-D + JamesH + JamesStewart + MarkWalker
- 2025-11-09 Nick + Karl + Dan
- 2025-12-14 Nick + Jamesh + JasperG + JamesStewart + DanBuzzard + BarkoteN

### TODO:

* Second Tuesday 2024-02-13: Committee ops/handover
* Spacecubed venue contact/liason
* done: ACTION: Dan: Contact Alastair
* Call out on main plug@plug list?
* Lawrence Lau <[e-mail]> is willing@2024-05-14 , Spacecubed hot-desker
* Sending email from role addresses:
* https://support.google.com/mail/answer/22370
* gcalendar
* youtube: (JamesB phone contact)
* meetup
* mailing lists+archives
* ugmm groups / LDAP: Done
* membership processing
* bank signatories
* website updates
* including redeploy ugmm
* passwordstore.org / pass(1): ACTION: Ben, Nick
* https://github.com/asynthe/plug-pass
* backups
* rsync.net debit card
* PCHQ
* done: event promotion! win a Pi5!
* update Facebook admins:
* done: JamesH add Sarah
* update linkedin admins:
* https://www.linkedin.com/groups/3765623
* current: Euan, Luke, PeterL, JasonN, Alastair
* add: BenIDS, DanB
* X/twitter admins:
* Settings->Security&AccountAccess->Delegation
* current: JamesH, PatrickC, Niall

* Can't do a remote talk without BBB?
* Fix old BBB v2.3 meetings.ucc.asn.au ?
* vs set up our own ephemeral one?
* vs jitsi
* vs jami
* vs demo.bigbluebutton.org (no provided recording, 60 minute limit)
* Upcoming committee Tuesday 2023-04-18
* Check certs
* Pay
* digitalocean.com (Wings' team PLUG), autopayment first attempted 2023-04-01
* AWS, autopayment first attempted 2023-04-03

All previous months were paid with card ending 1216.
This card appears to have expired as the payment for March 2023, ($55.24) Has not been paid. 

* Done! with BendigoBank card expires 2023-06
* rsync.net, payment due now/2023-04 , but expired card noticed at 2022-05-15 , autopayment first attempted 2023-04-09
* Done! with BendigoBank card expires 2023-06
* Raspberry Jam claim (Niall filled out form to receive camera and magazine. will bring to events when they arrive)

Date: Wed, 5 Apr 2023 11:15:25 -0400
From: Matt Richardson <[e-mail]>
Subject: [plug-ctte] March Raspberry Jam gift

Date: Sun, 09 Apr 2023 10:13:31 +0000
From: Niall <[e-mail]>
Subject: Re: [plug-ctte] March Raspberry Jam gift

* Done!
* Replace lastpass with https://www.passwordstore.org/ , test for admin/committee
* Cleanup cloud users
* (DONE) Deal with UGMM errors
* This isn't strictly finished as there are still warnings on the signup completion page, but we are ready to move forward.
* (DONE) nginx
* (DONE) commit change of maps
* (DONE) Add lists redirect for / -> /mailman/listinfo
* (DONE) Add lists redirect for HTTP -> HTTPS
* (DONE) ugmm redirects in main site config
* (DONE) Point mumble at new wildcard cert
* tested: but unless we start with `murmurd -wipeSSL` it still uses a self-signed certificate?!
* tested working after `dpkg-reconfigure mumble-server` ?
* Plan is to wipe Mumble on the day
* Issue: backups were done, but the wrong ones were pruned: WIP ones were retained but working test states were not
* (NEEDS TEST) Migrate mail
We only need to migrate a couple of mailboxes
We'll keep the remaining ones available in the backups of Power.
* (DONE) Migrate /etc/aliases
* (DONE) Daily backups - /etc/cron.daily/
* (DONE) SSL cert for lists,lists2
* (DONE) Fix broken PLUG logo on https://www.plug.org.au/ugmm/signup and check other pages
* spamassassin not enabled after a reboot: fixed

## TODO: continued

### Notes from Wings 2020-11-01:

example: opendkim-genkey -r -s myselector -b 2048 -d example.com

re-generate key: opendkim-genkey -r -s mail -b 2048 -d po1.plug.org.au

Disabled DKIM

Gen aliases for mailman: glass@edison:/usr/lib/mailman/bin$ sudo ./genaliases

Set passwd for glass: riots stunt triple thongs

enable spamassassin to run on boot: glass@edison:~$ sudo systemctl enable spamassassin

## TODO: Later
* Later
* fail2ban
* Regular spam? SASL logins from 45.142.120.53 , 45.142.120.74 , .121 .192 ... since 2020-10-18 rebuild
* 141.98.10.136 ?
* DKIM, testing on `po1.plug.org.au`

    /etc/postfix/master.cf:
    # OpenDKIM stuff
    milter_default_action = accept
    milter_protocol = 6
    # from inside the chroot, the socket will be in /var/run/opendkim
    smtpd_milters = unix:/var/run/opendkim/opendkim.sock
    non_smtpd_milters = unix:/var/run/opendkim/opendkim.sock

    root@edison:/etc/dkimkeys# ls -l /etc/dkimkeys/mail.*
    -rw------- 1 root root 1679 Nov  1 12:20 /etc/dkimkeys/mail.private
    -rw------- 1 root root  514 Nov  1 12:20 /etc/dkimkeys/mail.txt

    Nov  1 12:24:47 edison opendkim[6119]: can't load key from /etc/dkimkeys/mail.private: Permission denied
    Nov  1 12:24:47 edison opendkim[6119]: BDF6563AF2: error loading key 'mail._domainkey.po1.plug.org.au'
    Nov  1 12:24:47 edison postfix/cleanup[6411]: BDF6563AF2: milter-reject: END-OF-MESSAGE from localhost[127.0.0.1]: 4.7.1 Service unavailable - try again later; from=<[e-mail]> to=<[e-mail]>

* Regenerate the mailman archives
* regeneration skipped: will change message numbers if fix-mbox.pl is run on plug.mbox, for example
* (ISSUES DURING TEST) Migrate mailman archives
We ran into some invalid emails (which contained some special strings - leading "From "). @niceness looked into them and was able to correct them, but the corrections left holes in the archives which screwed up numbering.
We plan to address this eventually by padding the invalid emails with dummy ones (perhaps with an explanation of what happened).
For now we are going to take the latest copy from Power and resume from there.

/var/lib/mailman/lists
/var/lib/mailman/archives

## jekyll workaround
There is an old version of Jekyll in Debian 9 which is not really compatible with our jekyll builds. (e.g. post generation in committee minutes)

So... `apt uninstall jekyll`, then install ruby2.3 and ruby2.3-dev, then sudo gem install jekyll.

You specifically need version 3.8.5...

`sudo gem install jekyll -v 3.8.5`

This should fix builds. It won't be necessary once we move to Buster (Debian 10).

## knot restore
- restoring `/etc/knot` from backups
- repo contains only `/etc/knot/plug.org.au.example.zone` , not the live data:
- _acme-challenge.plug.org.au.zone
- _acme-challenge.po1.plug.org.au.zone
- plug.org.au.zone

## Maintenance email
Hi PLUG,

From 12pm until 5pm tomorrow (2020-06-14 12pm-5pm AWST) we will be performing maintenance on PLUG infrastructure. This will mean service interruptions to all services including web, email and membership management.

We'll be minimizing the impact as much as possible but some downtime will be necessary.

## Cutover checklist
* System maintenance downtime warning
* Change all references in Ansible from "lists2" to "lists" and do a run (first dry, then normal).
* Move www.plug.org.au A record to point to Edison
* Shut down mail services + ugmm + mailman

sudo service dovecot stop
sudo service exim4 stop
sudo service postfix stop
sudo service apache2 stop
sudo service mailman stop

* Run the LDAP backup script
`/etc/cron.daily/20-ldapdump`
* Run `/root/bin/borgauto.sh` on power
* Run rsync --delete to pluck /var/lib/mailman from power
* Shut down power
* Run the brestore script to pluck data
* Inject test email, see outgoing message in queue+pipermail archive
* Check results
* LDAP migration
* FIXME: LDAP dump/restore, replace with a rebuild from fresh config, e.g.
* move BDB->MDB, https://www.adimian.com/blog/2014/10/how-to-enable-memberof-using-openldap/
* edison:/etc/ldap/secure/extra-modules-overlays-schemas
* mailbox migration
* Check/regenerate the mailman archives
* regeneration skipped: will change message numbers if fix-mbox.pl is run on plug.mbox, for example
* Move plug.org.au A record to point to Edison

## Mailman Migration
New way - use @NB script
`root@bayonet:~/bin# ./brestore.sh --mount-all`

Restore lists from backup (with cp), OR...

cd /tmp/latest-power/var/lib/mailman/lists/
cp -R * /var/lib/mailman/lists/

Restore lists from backup (with rsync)

rsync -av /tmp/latest-power/var/lib/mailman/lists/ /var/lib/mailman/lists/

FIXME: has left a number of `pending.pck.tmp.*` files

Fix surface-level permissions and ownership.

chown root:list /var/lib/mailman/lists/*
chmod 2775 /var/lib/mailman/lists/*

Restore archives from backup

cd /tmp/latest-power/var/lib/mailman/archives/private
sudo cp -R *.mbox /var/lib/mailman/archives/private/
chown root:list /var/lib/mailman/archives/private/*
chmod 2775 /var/lib/mailman/archives/private/*

Recreate archives from mailboxes (SKIP FOR NOW)
We run plug last as it is the biggest and hardest to build (SKIP FOR NOW)
(We are NOT recreating the archives at this stage)

cd /var/lib/mailman/archives/private
sudo /var/lib/mailman/bin/arch admin
sudo /var/lib/mailman/bin/arch av
sudo /var/lib/mailman/bin/arch committee
sudo /var/lib/mailman/bin/arch hackers
sudo /var/lib/mailman/bin/arch jobs
sudo /var/lib/mailman/bin/arch mailman
sudo /var/lib/mailman/bin/arch off-topic
sudo /var/lib/mailman/bin/arch ugmm
sudo /var/lib/mailman/bin/arch userconf
sudo /var/lib/mailman/bin/arch plug

Fix URLs and make lists properly appear on frontpage where appropriate

cd /var/lib/mailman/lists
sudo withlist -l -r fix_url admin
sudo withlist -l -r fix_url av
sudo withlist -l -r fix_url committee
sudo withlist -l -r fix_url hackers
sudo withlist -l -r fix_url jobs
sudo withlist -l -r fix_url mailman
sudo withlist -l -r fix_url off-topic
sudo withlist -l -r fix_url plug
sudo withlist -l -r fix_url ugmm
sudo withlist -l -r fix_url userconf

## (DONE) nginx old ugmm to new
https://stackoverflow.com/questions/22224441/nginx-redirect-all-requests-from-subdirectory-to-another-subdirectory-root/22261287

In the end we decided to point old UGMM to new UGMM with a "dumb" redirect to help prevent certain kinds of attacks. IE: any old UGMM URL will redirect to https://ugmm.plug.org.au/, with no arguments or paths brought across.

## mailman archive transplant
We want to transplant the current set of mailman archives, instead of regenerating them (for various reasons - bad emails and avoid re-numbering). Once imported, we want to run fix_urls to ensure that they are consistent and that the links work.

https://docs.borgbase.com/restore/borg/

/var/lib/mailman/archives

borg extract --list --dry-run $BORG_REPO::'power.plug.org.au-2020-07-17 06:47:14.255843' /var/lib/mailman/archives

root@power:~/.ssh# time rsync --delete -e 'ssh -i /root/.ssh/borgkey' -az --stats /var/lib/mailman/archives/. [e-mail]:/tmp/power_var_lib_mailman_archives/.

Number of files: 168440
Number of files transferred: 0
Total file size: 2475392433 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 3253302
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 3277980
Total bytes received: 24673

sent 3277980 bytes  received 24673 bytes  188723.03 bytes/sec
total size is 2475392433  speedup is 749.52

real	0m17.654s
user	0m0.380s
sys	0m1.520s

## test cases
### main website
sed -n '/test cases/,$ p' checklist.md.txt |grep http > urls.tocheck.txt

http://plug.org.au/ -> https://plug.org.au/
http://www.plug.org.au/ -> https://www.plug.org.au/
https://plug.org.au/
https://www.plug.org.au/
https://plug.org.au/resources/
https://plug.org.au/contact/
https://plug.org.au/events/
https://plug.org.au/events/archive/
https://plug.org.au/events/2004/
https://plug.org.au/events/committee/2020/04-21/
https://www.plug.org.au/contact/
https://www.plug.org.au/events/
https://www.plug.org.au/events/archive/
https://www.plug.org.au/events/2004/
### ugmm
https://www.plug.org.au/ugmm/memberself ->
https://www.plug.org.au/ugmm/ ->
http://www.plug.org.au/ugmm/memberself ->
http://www.plug.org.au/ugmm/ ->
http://ugmm.plug.org.au/ -> https://ugmm.plug.org.au/
https://ugmm.plug.org.au/
### mailman
http://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/admin/committee
http://lists.plug.org.au/mailman/admin/committee/members
http://lists.plug.org.au/ -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/ -> https://lists.plug.org.au/mailman/listinfo
?? http://lists.plug.org.au/mailman/listinfo/ -> https://lists.plug.org.au/mailman/listinfo
http://lists.plug.org.au/mailman/private/committee/
http://lists.plug.org.au/mailman/private/committee/2020-January.txt.gz
http://lists.plug.org.au/mailman/private/committee/2019-May/thread.html
http://lists.plug.org.au/mailman/private/committee/2010-August/000031.html
(spam) http://lists.plug.org.au/mailman/private/committee/2010-August/000041.html
http://lists.plug.org.au/mailman/listinfo/admin
http://lists.plug.org.au/mailman/listinfo/av
http://lists.plug.org.au/mailman/listinfo/committee
http://lists.plug.org.au/mailman/listinfo/hackers
http://lists.plug.org.au/mailman/listinfo/jobs
http://lists.plug.org.au/mailman/listinfo/off-topic
http://lists.plug.org.au/mailman/listinfo/mailman
http://lists.plug.org.au/mailman/listinfo/plug
http://lists.plug.org.au/mailman/listinfo/ugmm
http://lists.plug.org.au/mailman/listinfo/userconf
http://lists.plug.org.au/pipermail/jobs/2015-March/000015.html
http://lists.plug.org.au/pipermail/plug/
http://lists.plug.org.au/pipermail/plug/2020-April/thread.html
http://lists.plug.org.au/pipermail/plug/2020-April/084366.html
http://lists.plug.org.au/pipermail/plug/2024-May/084948.html

# ls /var/lib/mailman/archives/*
/var/lib/mailman/archives/private:
admin       av.mbox         hackers       jobs.mbox     off-topic       plug.mbox  userconf
admin.mbox  committee       hackers.mbox  mailman       off-topic.mbox  ugmm       userconf.mbox
av          committee.mbox  jobs          mailman.mbox  plug            ugmm.mbox

/var/lib/mailman/archives/public:
av  jobs  mailman  off-topic  plug  ugmm

### mail
We can use defer_transports to safely test email (and manually approve).
* Send email to [e-mail]
* Send email to [e-mail]
* Receive an email from [e-mail]
* Working
* Receive an email from [e-mail]
* Working
* Receive a spam email (GTUBE-TEST) from [e-mail]
* Working
* UGMM payment reminder goes out (force expire wings)
* basic swaks test
* `swaks --from [e-mail] --to [e-mail] --server edison.plug.org.au`
* `mutt -f ~/Maildir`
* Working

* get some graphs
* https://logit.io/blog/post/top-grafana-dashboards-and-visualisations/#14-ssl-expiry-tracker
* https://logit.io/blog/post/top-grafana-dashboards-and-visualisations/#15-aws-billing-estimator
* mail throughput, latency, errors
* USE / RED / golden4 : mail/web/etc.

----

### Notes POSH+PLUG hack day 2023-07-09
- Nick + Michael + Niall + James + BenjaminIDS

- Niall tally up cloud expenses -> sponsor proposal
- AWS payments: remove BA card, update NB card expiry+details
- mailman 2to3
- https://gitlab.com/mailman/mailman/-/merge_requests/531
- https://docs.mailman3.org/en/latest/migration.html
- Backups: logged into plug.perthchat.org @PCHQ

- Updated Meetup calendar with Michael Collins talk on Matrix for September 2023-09-12. Google calendar is populated already.
- Benjamin De Silva has been approached to do a talk on EMACS for October 2023-10-10

* Done at PLUG 2nd-Tuesday infra: 2023-08-08

- Intro to Ansible/getting started: ad-hoc -> collections (modules)/playboks

$ ssh -o PubkeyAcceptedKeyTypes=+ssh-rsa -o HostKeyAlgorithms=+ssh-rsa -i xxx [e-mail]
The authenticity of host 'power.plug.org.au (54.252.97.56)' can't be established.
RSA key fingerprint is SHA256:3yCzV9ETTz1Wge9etcYTbibxozM3Hxmi3sNu+6Xedxs.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

- 2025-03-13 update: onboard new admins
- create `power.plug.org.au` account

laptop$ ssh-keygen -t rsa plug2025
enter passphrase
copy .pub to doc below
cat >> ~/.ssh/config
Host power power.plug.org.au
	# RSA-SHA1 deprecated, use rsa-sha2 or newer # Legacy: HostKeyAlgorithms +ssh-rsa
	# see sshd_config(5) ; ssh -Q key
#@power.plug.org.au
#Host *.plug.org.au
#	User foo
       IdentityFile ~/.ssh/identity-plug
       HostKeyAlgorithms +ssh-rsa
       PubkeyAcceptedAlgorithms +ssh-rsa

power# adduser --uid 10566 zlqrvx
Enter new UNIX password: 
Retype new UNIX password: 
...
power# su - zlqrvx
$ ssh power.plug.org.au
yes
^C
cd .ssh
echo 'ssh-rsa AAAAB3NzaC1yc2EA...' > authorized_keys
$ exit
# adduser zlqrvx sudo

## 2023-08-08 plans for moving forward:
* Review site.yml on github.
- fork it for debian 12 and test it
- comment out what is not working and keep trying.
- go back and try and get what was commented out working.

## 2023-08-13 PLUG Hack

- Ansible + AWS
- Created user called ansible
- Gave it a custom [IAM policy](https://us-east-1.console.aws.amazon.com/iamv2/home?region=ap-southeast-2#/policies) that allowed:
- EC2 Describe, Start, Stop, Terminate, Run (Creates and starts instance)
- EC2 Describe, create and delete tags

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EC2InstanceManagement",
            "Effect": "Allow",
            "Action": [
                "ec2:Describe*",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:RunInstances",
                "ec2:*Tags"
            ],
            "Resource": "*"
        }
    ]
}

manually launched t2.nano: ssh -v admin@13.239.40.47
sudo apt update; sudo apt install -y eatmydata
time sudo eatmydata apt install -y ansible python3-boto ansible-core ansible-lint apt-file python3-botocore python3-boto3

task 1 returns quickly (~2s); instance startup ~150s

ERROR! couldn't resolve module/action 'ec2'. This often indicates a misspelling, missing collection, or incorrect module path.
ec2: -> amazon.aws.ec2_instance:

### Notes 2023-03-19

- run backups: DONE
- ugmm: test password reset: DONE
- ugmm: update committee group in LDAP, visible in ugmm: DONE
- fail2ban
- TODO: ugmm logins
- fast-mail-bomber
- TODO: commit this to plug-services
- Updated to ban hits on mailman.
- 3 hits in 1hr
- ban for 1000000sec

root@power:/etc/fail2ban# git whatchanged 0efe0d2..f58a611
commit f58a6118c66081b317ee3ff289eaf6d30e1469b0
Author: root <[e-mail]>
Date:   Sun Mar 19 15:19:04 2023 +0800

    Thwart the fast mail bomber Nick, Niall

:100755 100755 cc78361... 61c73c0... M  .etckeeper
:000000 100644 0000000... e672809... A  fail2ban/filter.d/apache-fmb-plug.conf
:100644 100644 69f4902... 5db7991... M  fail2ban/jail.local

# fail2ban-client status apachefmb
# fail2ban-client get apachefmb actionunban 180.150.90.127

- ignore list is clear at startup; start and add two addresses:
# service fail2ban start;fail2ban-client set apachefmb addignoreip 180.150.90.127 ;fail2ban-client set apachefmb addignoreip 54.252.97.56

### Notes 2023-11-21 Nick + James

Date: Tue, 21 Nov 2023 11:50:56 +0800
From: Nick Bannon <[e-mail]>
To: [e-mail]
Subject: In-person ops meeting 2023-11-21 ... ops vs infra vs projects

[...]
Let's be practical and split up the tasks, making sure that:

- DONE: recent memberships/renewals are all processed
- a new mailing list, maybe "[e-mail]"? is tested and able to be preloaded with a list of current financial members
- also: checked nutmeg's scripts in UGMM
- minutes are published and deployed to the website
- EOY BBQ+hackday has a plan, I think that's mostly in hand
- PLUG-in-the-Pub January is pencilled in
- https://hackmd.io/@plug/pubs
- Second Tuesday February 2024 and March 2024 have some kind of rough plan: talks? Projects:AV? Pi Jam?
- anything else?

- test raspberrypi.com ID login
- query meetup user with default answers
- process recent membership renewals --2023-11-21
- `slapcat | extract-payments.py > payments.$(date +%Y%m%d)a.csv`
- https://www.plug.org.au/ugmm/ctte-members?expiredmembers=1
- http://lists.plug.org.au/mailman/listinfo

### Notes 2024-02-11 Nick, Ben
#### password-store

Moved to GitHub's repository README.md
https://github.com/asynthe/plug-pass

##### Google and Meetup Calendar updates

- February 2nd Tuesday event updated to reflect operations evening.
https://www.meetup.com/perth-linux-users-group-plug/events/298751581/

### Notes 2024-04-14 Niall, Nick, Ben
- backups: old AWS hosts, power, edison
- website deploy
- plus ugmm build
- checklist:
- was mostly automated as full machine build https://github.com/plugorgau/plug-services
- if you're confident to rebuild the whole machine in case of issues, it's easy! but if you're working it out, wanted to see what has changed, one can go deep
1. check website is working, before and after
- test URLs above: fangs/minilandl `webcheck.py`
- SSL expiring? ACME now running on `edison`: `ansible`: `tls-copy-edison2power`
2. check backups have been running OK, or do one now
3. build with `hugo` https://github.com/plugorgau/plugorgau.github.io/blob/master/README.md
- was built with CI: https://github.com/plugorgau/plugorgau.github.io/blob/master/build-to-gh-pages.sh
- copy or `rsync` to `power:/tmp`
4. `diff -ur /tmp/latest /home/plug/public_html `
5. `rsync -av /tmp/latest/. /home/plug/public_html/.`
- ownership/perms
- `--delete`
- check/diff for deleted/changed files against: `cp -al /home/plug/public_html web.$(date +%Y%m%d)`
6. https://github.com/plugorgau/ugmm v0.5 / manually-installed parts of v0.5.2 is part of the website
- build, check live URLs with `webcheck.py`, `diffoscope` against `plug-ugmm_0.5_all.deb`
- build in stretch chroot?
- `time mmdebstrap --mode=proot --variant=apt --include=build-essential stretch debian9-stretch.tar https://archive.debian.org/debian`
- `mkdir -p cache; chmod 1777 cache; time mmdebstrap --mode=proot --variant=apt --include=build-essential --skip=essential/unlink --setup-hook='mkdir -p ./cache "$1"/var/cache/apt/archives/' --setup-hook='sync-in ./cache /var/cache/apt/archives/' --customize-hook='sync-out /var/cache/apt/archives ./cache' stretch debian9-stretch3.tar https://archive.debian.org/debian`
- prune `power` backup objects: add lifecycle
- AWS - S3 bucket 'plug-us' created 2012 had no objects held in it. - Deleted by Niall
- vinyl/t-shirt printing

#### Ubuntu snaps
Distributor ID: Ubuntu
Description: Ubuntu 20.04.6 LTS
Release: 20.04
Codename: focal

- Open firefox, browse a few pages, surprise! an update is available.
- Can no longer open new tabs:

    Restart Required: Restart to Keep Using Firefox
    An update to Firefox started in the background. You’ll need to restart to finish the update.
    Your windows and tabs will be quickly restored, but private ones will not.
    [Restart Firefox]

- Normal apt-based upgrades also happening (including snapd):

    Software Updater: Updated software is available for this computer. Do you want to install it now?
    The computer also needs to restart to finish installing previous updates.

### Notes 2024-05-14 Nick, BenIDS, Sarah, MarkW, Owain, LawrenceL

- https://www.ato.gov.au/businesses-and-organisations/not-for-profit-organisations/your-organisation/in-detail/income-tax/mutuality-and-taxable-income-for-not-for-profits/lodgment-rules-and-tax-rates#ato-Taxrates
- Taxable income <$416? and Constitution Objects S3.2 ?
- https://www.ato.gov.au/businesses-and-organisations/not-for-profit-organisations/statements-and-returns/in-detail/reporting-requirements-to-self-assess-income-tax-exemption/who-doesnt-need-to-lodge

### Notes 2024-06-09 Nick, MarkW
- PLUG programming options?
- http://lists.plug.org.au/pipermail/plug/2024-May/084948.html has some including:
- IRC matrix bridge (in haskell!)
- pipermail archives:
- try adding https://public-inbox.org/ ?
- running Mailman2's pipermail (single python file) standalone on a new system, maybe Mailman3?
- https://mail.python.org/archives/list/[e-mail]/2002/10/?count=200
- https://mail.python.org/archives/list/[e-mail]/thread/PSXRAYZG6JC2KCW5CJ2DF74BEIRHFKZP/
- make http://lists.plug.org.au/pipermail/plug/recent work, like https://lists.debian.org/debian-vote/recent
- "report as spam" button, like https://lists.debian.org/debian-vote/2024/05/msg00000.html
- get inspired by https://wiki.debian.org/Teams/ListMaster/ListArchiveSpam and https://lists.debian.org/msgid-search
- expand eventcheckr / https://github.com/plugorgau/calendar-check
- AV project

### Notes 2024-09-10 Nick + MichaelC + JamesH + DanB
- Membership processing
- `edison` rebuild in AWS?
- events

- pcadmin has account on `power.plug.org.au` @AWS
- power is very old, login with RSA keys only (or old DSA), needs `ssh -oHostKeyAlgorithms=+ssh-rsa -oPubkeyAcceptedKeyTypes +ssh-rsa`

Host power.plug.org.au
    HostName power.plug.org.au
    User xxxx
    Port 22
    HostKeyAlgorithms ssh-rsa
    PubkeyAcceptedKeyTypes +ssh-rsa
    IdentityFile ~/.ssh/plug_2024_2
    IdentitiesOnly yes

- Create accounts for jamesh, pcadmin, dan on `plug.perthchat.org` / `plug.michael5ollins.com` @PC-HQ
- Create accounts for jamesh, pcadmin, dan on `edison.plug.org.au` ( @DigitalOcean )
- Create account for jamesh, pcadmin, dan on https://plug.signin.aws.amazon.com/console

plug.perthchat.org:/home/pcadmin/process_memberships $ ls -l
-rwxr-xr-x 1 pcadmin pcadmin 1843 Sep 12  2023 extract-payments.py
-rw-r--r-- 1 pcadmin pcadmin 1480 May 29 20:16 fetch.yml

### Agenda/minutes for next committee meeting:
- https://hackmd.io/@plug/committee-minutes-2024-09
- Dan, Nick: events

#### `edison` rebuild in AWS

- https://plug.signin.aws.amazon.com/console
- test instance t3a.micro : http://13.210.141.88/
- question: how do we bootstrap letsencrypt/certbot? how does it know what certs we want? read them out of nginx config?
- Nick, Michael: shutdown `edison` droplet, resize to USD$12/month droplet (cannot go smaller with the current 50GiB storage)

## 2024-10-13

- http://lists.plug.org.au/mailman/listinfo/

- PLUG Website build on Power:
- Set up a working copy of hugo on Power that can be run via `/home/admin/hugo/hugo.sh`
- Can build the website if I comment out `enableGitInfo = true` in `config.toml`: Hugo's invocation of Git uses arguments not supported by the ancient version.

  git clone https://github.com/plugorgau/plugorgau.github.io.git
  cd plugorgau.github.io
  sed -i 's/^enableGitInfo/#\0/' config.toml
  ../hugo/hugo.sh
  rsync -a public/ /home/plug/public_html/

- (plus 2025-02-18 updates, all done as `admin@power` user)

## 2024-11-10
- website deploy?
- `ssh power`, removed old users from admin,adm group
- mailman/mailing list web UI
- jayasekerakushan@gmail subscribing
- Brave, Chrome on Android, not Chrome Desktop?: http ignored went to https
- Mail sent from power->gmail, ended up in spam, reported not spam
- `Nov 10 14:40:27 power postfix/smtp[4870]: 3727D465C4: to=<[e-mail]>, relay=gmail-smtp-in.l.google.com[64.233.170.27]:25, delay=3, delays=0.01/0.01/1.6/1.4, dsn=2.0.0, status=sent (250 2.0.0 OK 1731221134 98e67ed59e1d1-2e9a5f5bc8fsi8807865a91.57 - gsmtp)`
- Saved from spam, self-confirm link, also http!
- **then** the request shows up in http://lists.plug.org.au/mailman/admindb/plug for moderator approval:
- Subscription Requests Address/name Defer Approve Reject Discard
- mailman issue: upgrade to header munge to avoid conflicts with tight DMARC policies

## 2024-11-10 Nick + Wyatt
- Wyatt checking UGMM - works!
- Wyatt checking committee mailing list
- http://lists.plug.org.au/mailman/listinfo/committee
- http://lists.plug.org.au/mailman/admin/committee
- DMARC: Wyatt (at gmail) has not seen Harry's most recent reply from decisions-and-designs.com.au Re: [plug-ctte] Old Laptops
- mailman not munging From:
- Michael Barker@Geraldton ex-school laptop recycling HP ProBooks with 2x SODIMM slots, 4th gen CPU
- 2024-01-09 previous AGM: https://www.meetup.com/perth-linux-users-group-plug/events/297310367
- cancel 2024-12-10 second Tuesday https://www.meetup.com/perth-linux-users-group-plug/events/302639149/

## 2025-01-07 Nick + Wyatt
- Infra (technical, specific small tasks)
- Issue with domains not appearing in emails: strict DMARC domain policies that prevent delivery
- `for DOMAINNAME in beckwith.net.au danscomp.net decisions-and-designs.com.au fnarfbargle.com iinet.net.au kenworthy.id.au mccormick.cx oranges.id.au stgeorge.com.au plug.org.au;do host -tTXT ${DOMAINNAME};host -tTXT _dmarc.${DOMAINNAME};done`
- mailman upgrade 2.x or 3.x to munge headers to come "from" us
- To try:
- turn on https for all links, run the link checker before and after
- deploy a website update: edit, git commit/push/pull, hugo build
- build of website with Github actions?
- Projects (larger)
- AV hardware (grants?)/software refresh (self-host BBB? self-starting OBS system?)
- Share infrastructure with LA or other LUGs
- Ops (Committee/organisational) for "What's on for PLUG in 2025-01": AGM prep, Committee meeting, PLUG-in-the-Pub
- Wyatt will phone/speak around Wednesday 2025-01-08 to the committee members that haven't responded yet, confirm AGM hosts
- State/Federal elections coming soon!

## 2025-02-18 Nick + Wyatt + JamesH
- Website upload: JamesH
- Wyatt login ssh to `power`, create SSH key, deploy website to live

## 2025-03-09 Nick + JamesH + MarkW

* Processed video for Jasper's talk from September
* Update to https://plugorgau.github.io/projects/video/network/
* subscribe TimL
* Update TLS cert with `ansible`: `tls-copy-edison2power`
* Fetch URLs: `webcheck.py` - test before/after:
* website deploy?
* full HTTPS config?

## 2025-04-13 Nick + MarkW + JamesH + TimL

* Fetch URLs: `webcheck.py` - test before/after:
* website deploy?
* full HTTPS config?

301 Moved Permanently,https://plug.org.au/membership -> https://plug.org.au/membership/,308

https://httpd.apache.org/docs/2.4/en/mod/mod_alias.html#redirect

- James: fast-mail-bomber: can we blackhole the http version of these mailman form pages?

../mailman/pending-subscription-request-policy-spam/github.com/juzeon/fast-mail-bomber/raw/master/data$ grep plug.org nodes.20230213.json 
    "http://lists.plug.org.au/mailman/subscribe/av",
    "http://lists.plug.org.au/mailman/subscribe/jobs",
    "http://lists.plug.org.au/mailman/subscribe/off-topic",
    "http://lists.plug.org.au/mailman/subscribe/plug",
    "http://lists.plug.org.au/mailman/subscribe/ugmm",
    "http://lists.plug.org.au/mailman/subscribe/userconf",

/etc/apache2/sites-available/lists.plug.org.au
/etc/apache2/sites-available/plug.linux.org.au

/etc/apache2/sites-available/plug.org.au.conf
/etc/apache2/sites-available/plug.org.au-ssl.conf

- working: http://lists.plug.org.au/pipermail/plug/
- 404 not found: http://lists.plug.org.au/pipermail/plug/

- add TLS for https://lists.plug... plus redirects
- check redirects for subscribe apachefmb bot activity:

power# grep 'lists.plug.org.au:443.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l
43
power:# grep 'lists.plug.org.au:80.*subscribe' /var/log/apache2/other_vhosts_access.log|wc -l
98

port80 -> 403, due to new "Deny from all" on "Location /mailman/subscribe/""
port443 -> 401, due to AuthType Basic on /mailman

- `fail2ban` apachefmb is scanning
- https://www.passwordstore.org/
- list accounts. services, passwords
- borgbackup -> rsync.net , pchq: email admin@plug needs new RSA? key, not ssh-dss
- `/root/bin/borgauto.sh`
- `/root/bin/borgauto-pchq.sh`

- problem: https://lists.plug.org.au/mailman/... is working on HTTPS, but mailman pages contain many pre-generated references to HTTP links, e.g. http://lists.plug.org.au/mailman/admin/plug/members?letter=m from https://lists.plug.org.au/mailman/admin/plug/members
- not fixed by s/http/https/ in:

# vi /etc/mailman/mm_cfg.py
...
DEFAULT_URL_PATTERN = 'https://%s/mailman/'
power# service mailman restart

some http link references in config text:
# /var/lib/mailman/bin/config_list -o - plug|less

<li><a href="http://lists.plug.org.au/pipermail/plug/">Go to list archives</a><br>&nbsp;<br>

- 2025-06-08 Nick + HarryMc + JamesH + JamesStewart + MarkWalker + DanB
- `borgbackup` on `power`, run, fix with `BORG_REMOTE_PATH=borg14` to rsync.net, test create and extract

- 2025-09-14 Nick + JamesStewart + JamesH
- https://search.google.com/search-console
- now domain-validated in DNS with `google-site-verification` TXT record on `plug.org.au`
- new round SVG logo PLUG-on-Tux for the youtube channel

- 2025-10-12 Nick + JohnM-D + JamesH + JamesStewart + MarkWalker
- add JohnM-D ssh key
- https://hackmd.io/@plug/server-upgrade-plan/edit

- 2025-11-09 Nick + Karl + Dan
- Laptop research - new and used, Linux supported
- Deploy website from git - thank you (Harry)
- AGM venue - signup to spacecubed?
- Post note from discussion with James .. to deply the website
1. login as [e-mail]
2. cd plugorgau.github.io
3. git pull origin
4. hugo -d _site
5. sudo ./deploy.sh
The git pull makes sure the local plugorgau folder is up to date
The hugo builds a site image in the _site folder
Only the deploy needs to be run as sudo to rsync out to /var/www

- 2025-11-10 Microbackup Pi0W workshop@UCC
- https://github.com/gmatht/btrnas
- Most interesting stuff is in btr_snap
directory.
- install_rpi... sh (or ps1) is the first thing to use but may be easier to do it manually if you don't like debugging.
- Task Sign-up Sheet: https://docs.google.com/spreadsheets/d/1NbVs-5cE906-JUI_UrSplXQveUMB5m-3tMEIoSKgmLI/edit?usp=drivesdk
- Temp Hotspot: GCon GenghisCon2024GenghisCon2024
- You can stop the rpi taking up the entire sd card with ext4, by creating a random partition on the sd card.

- 2025-12-14 Nick + Jamesh + JasperG + JamesStewart + DanBuzzard + BarkoteN
- JasperG: OpenVPN client
- AWS Cleanup:
- Deleted "multi" and "bayonet" EC2 VMs
- Tared up "LCA" VM and copied to ~glass/lca-vm.tar.zstd (938 MB) on iec, then deleted VM
- seems to have an RT4 install
- Many binaries seem to segfault when running from new kernel
- Deleted EBS volumes for above VMs
- Deleted old EBS snapshots
- to do: delete EFS volumes
- There's a copy of the data on iec, but it is not currently being backed up.