--- title: updates for week of 2025-11-17 --- --- title: updates for week of 2025-11-17 --- # Resources - [ ] - outage ticket: https://pagure.io/fedora-infrastructure/issue/????? - [ ] - status update: https://github.com/fedora-infra/statusfpo/pull/?? - [ ] - announcement sent to devel-announce/infrastructure lists # SOP-ish (put this in docs) - log into batcave01 - Put your name next to the machine here (with "in progress"). - run: `sudo rbac-playbook vhost_update_reboot.yml` - Also see: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/massupgrade/ updates for week of 2025-11-17 ## openqa - sometime - schedule with adamw * openqa-a64-worker01.rdu3.fedoraproject.org - done - adamw * openqa-a64-worker02.rdu3.fp.o - done - adamw * openqa-x86-worker01.rdu3.fp.o - done - adamw * openqa-x86-worker02.rdu3.fp.o - done - adamw * openqa-x86-worker03.rdu3.fp.o - done - adamw * openqa-x86-worker04.rdu3.fp.o - done - adamw * openqa-x86-worker05.rdu3.fp.o - done - adamw * qvmhost-x86-01.rdu3.fp.o - done - adamw ## copr_hypervisor - sometime - schedule / check with #buildsys:fedoraproject.org * vmhost-p09-copr01.rdu-cc.fedoraproject.org - * vmhost-p09-copr02.rdu-cc.fp.o - * vmhost-p09-copr01.rdu-cc.fp.o - * vmhost-x86-copr01.rdu-cc.fp.o - * vmhost-x86-copr02.rdu-cc.fp.o - * vmhost-x86-copr03.rdu-cc.fp.o - * vmhost-x86-copr04.rdu-cc.fp.o - * db.stg.aws.fedoraproject - ? # 2025-11-17 Monday - staging - done * buildhw-p10-02.stg.rdu3.fedoraproject.org - james - done * bvmhost-a64-01.stg.rdu3.fp.o - james - done * bvmhost-s390x-01.stg.s390.fp.o - james - done * bvmhost-x86-01.stg.rdu3.fp.o - james - done * bvmhost-x86-02.stg.rdu3.fp.o - james - done * bvmhost-x86-03.stg.rdu3.fp.o - james - done * vmhost-x86-01.stg.rdu3.fp.o - james - done * vmhost-x86-02.stg.rdu3.fp.o - james - done * vmhost-x86-03.stg.rdu3.fp.o - james - done * vmhost-x86-04.stg.rdu3.fp.o - james - done - ⚠el10: Weirdness on reboot, went away on reboot. * vmhost-x86-05.stg.rdu3.fp.o - james - done-ish - ⚠Mailman: `dnf rm --noautoremove gdal-libs libspatialite libgeotiff` -- due to epel to rhel migrations and dropped packages. # 2025-11-18 Tuesday - non outage causing production Can also be done a bit on Monday. * autosign01.rdu3.fedoraproject.org - ⚠ needs robosig pass - kevin - done * backup01.rdu3.fp.o - ⚠ needs backup pass - kevin - done * dedicatedsolutions01.fp.o - james - done * ibiblio02.fp.o - james - done * ibiblio05.fp.o - james - done - ⚠proxy12 had lvm2 pkg removed on 2025-10-08 so couldn't boot * internetx02.fp.o - james - done * osuosl02.fp.o - kevin - done * proxy05.fp.o - ⚠ bare metal - kevin - done * retrace03.rdu-cc.fp.o - ⚠ bare metal - kevin - done * sign-vault01.rdu3.fp.o - ⚠ needs vault pass - kevin - done * sign-vault02.rdu3.fp.o - ⚠ needs vault pass - kevin - done * storinator01.rdu-cc.fp.o - ⚠ bare metal - james - done * vmhost-x86-cc02.rdu-cc.fp.o - james - done * vmhost-x86-cc05.rdu-cc.fp.o - james - done * vmhost-x86-cc06.rdu-cc.fp.o - james - done # Package maintainer AWS hosts - done * aarch64-test01.fedorainfracloud.org - james - done * aarch64-test02.fc.o - james - done * el10-test.fc.o - james - done * el9-test.fc.o - james - done * f41-test.fc.o - james - done * f42-test.fc.o - james - done * f43-test.fc.o - james - done * ppc64le-test.fc.o - james - done * ppc64le-test02.fc.o - james - done * rawhide-test.fc.o - james - done --- # 2025-10-20 Thu - main event 22:00 UTC - **Don't** start before: ``` ================================ Day: Thursday ================================= 2025-11-20 14:00 PST US/Pacific 2025-11-20 17:00 EST --> US/Eastern <-- 2025-11-20 22:00 GMT Europe/London 2025-11-20 22:00 UTC UTC 2025-11-20 23:00 CET Europe/Berlin 2025-11-20 23:00 CET Europe/Paris ------------------------------- New Day: Friday -------------------------------- 2025-11-21 03:30 IST Asia/Calcutta 2025-11-21 06:00 HKT Asia/Hong_Kong 2025-11-21 06:00 +08 Asia/Singapore 2025-11-21 07:00 JST Asia/Tokyo 2025-11-21 08:00 AEST Australia/Brisbane ``` ## Checklist [x] Apply updates to log01 (**Do this 1.5h+ before**) - james - in progress [x] Apply updates to everything (**Do this 1h+ before**) - [x] Update status [x] Silence nagios [x] Disable backups on backup01 (**Do this 30m+ before**) - kevin - done [x] Disable updates pushes on bodhi-backend01 (**Do this 30m+ before**) - kevin - done [x] Stop koschei scheduler (**Do this a few hours before**) - kevin - done ### Extra [x] Do batcave01 / vmhost-x86-05.rdu3 first? [x] Do vmhost-x86-02.rdu3 early (to see if that "fixes" vHMC.rdu3) ### bkernel * buildhw-x86-01.rdu3.fedoraproject.org - needs pesignd passphrase - kevin - done ### builders * buildhw-a64-01.rdu3.fedoraproject.org - kevin - done * buildhw-a64-02.rdu3.fp.o - kevin - done * buildhw-a64-03.rdu3.fp.o - kevin - done * buildhw-a64-04.rdu3.fp.o - kevin - done * buildhw-a64-05.rdu3.fp.o - kevin - done * buildhw-a64-06.rdu3.fp.o - kevin - done * buildhw-a64-07.rdu3.fp.o - kevin - done * buildhw-x86-01.rdu3.fp.o - kevin - done * buildhw-x86-02.rdu3.fp.o - kevin - done * buildhw-x86-03.rdu3.fp.o - kevin - done * buildhw-x86-04.rdu3.fp.o - kevin - done * buildhw-x86-08.rdu3.fp.o - kevin - done * buildhw-x86-09.rdu3.fp.o - kevin - done * buildhw-x86-10.rdu3.fp.o - kevin - done * buildhw-x86-12.rdu3.fp.o - kevin - done * buildhw-x86-13.rdu3.fp.o - kevin - done * bvmhost-s390x-01.s390.fp.o - ⚠ statistically speaking likely will not come back up nicely - kevin - done * bvmhost-a64-01.rdu3.fp.o - james - done * bvmhost-a64-02.rdu3.fp.o - james - done - ⚠ buildvm-a64-17 shutdown timeout * bvmhost-a64-03.rdu3.fp.o - james - done - ⚠ bvmhost-a64-03 boot timeout * bvmhost-a64-04.rdu3.fp.o - james - done * bvmhost-a64-05.rdu3.fp.o - james - done * bvmhost-p09-05.rdu3.fp.o - james - done * bvmhost-x86-05.rdu3.fp.o - james - done - ⚠ buildvm-x86-09 shutdown timeout * bvmhost-x86-06.rdu3.fp.o - james - done * bvmhost-x86-riscv01.rdu3.fp.o - james - done ### builders - power10 lpars, managed by vHMC **Don't** update these unless vHMC is happy. * bvmhost-p10-01.rdu3.fp.o - no ipmi, ⚠ iscsi ordering with libvirt. - kevin - done * bvmhost-p10-02.rdu3.fp.o - no ipmi? - kevin - done * buildhw-p10-01.stg.rdu3.fp.o - done * buildhw-p10-02.stg.rdu3.fp.o - done ### non builders * bvmhost-x86-01.rdu3.fedoraproject.org - james - done * bvmhost-x86-02.rdu3.fp.o - james - done * bvmhost-x86-03.rdu3.fp.o - james - done * bvmhost-x86-04.rdu3.fp.o - james - done * vmhost-x86-01.rdu3.fp.o (bastion01/ipa01/noc01) - ⚠ nagios shush won't work - kevin - done * vmhost-x86-02.rdu3.fp.o (bastion02/ipa02), also -e nodns=true - kevin - done * vmhost-x86-03.rdu3.fp.o (log01) - james - done * vmhost-x86-04.rdu3.fp.o (db01) - james - done * vmhost-x86-05.rdu3.fp.o (batcave01) - kevin - done * vmhost-x86-cc01.rdu-cc.fp.o, after -e nodns=true - james - done - manual reboot: ⚠ reverse DNS * vmhost-x86-cc03.rdu-cc.fp.o - james - done * proxy3*-40 ( aws ) - kevin - done * proxy09 - kevin - done Possible things if there's time: * Upgrade builders to f43 - kevin - in progress * rhel10 reinstalls - no time * proxies to f43 (01/10/110/101) - kevin done * smtp-auth to f43 - james - done * buildvm-x86-riscv01/2 to f43 - james - done # Post checklist [x] start koschei scheduler [x] enable backups on backup01 [x] enable updates pushes on bodhi-backend01 [x] update status [x] unsilence nagios --- # Weird Errors * oci-candidate-registry01.rdu3 has weird network issues. * vmhost-x86-cc01.rdu-cc needs reverse DNS fixed.