--- # System prepended metadata title: updates for week of 2026-01-26 tags: [Updates] --- --- title: updates for week of 2026-01-26 --- # Resources - [x] - outage ticket: https://forge.fedoraproject.org/infra/tickets/issues/13067 - [x] - status update: done - [x] - announcement sent to devel-announce/infrastructure lists # SOP-ish (put this in docs) - log into batcave01 - Put your name next to the machine here (with "in progress"). - run: `sudo rbac-playbook vhost_update_reboot.yml` - Also see: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/massupgrade/ updates for week of 2026-01-26 ## openqa - sometime - schedule with adamw * openqa-a64-worker01.rdu3.fedoraproject.org - done - adamw * openqa-a64-worker02.rdu3.fp.o -done - adamw * openqa-x86-worker01.rdu3.fp.o - firmware staged - done - adamw * openqa-x86-worker02.rdu3.fp.o - firmware staged - done - adamw * openqa-x86-worker03.rdu3.fp.o - firmware staged - done - adamw * openqa-x86-worker04.rdu3.fp.o - firmware staged - done - adamw * openqa-x86-worker05.rdu3.fp.o - firmware staged - done - adamw * qvmhost-x86-01.rdu3.fp.o - firmware staged - done - adamw ## copr_hypervisor - sometime - schedule / check with #buildsys:fedoraproject.org * vmhost-p09-copr01.rdu3.fedoraproject.org - kevin - done (and upgraded to f43) * vmhost-p09-copr02.rdu3.fp.o - kevin - done * vmhost-p09-copr03.rdu3.fp.o - kevin - done * vmhost-p09-copr04.rdu3.fp.o - kevin - done * vmhost-x86-copr03.rdu3.fp.o - kevin - done * vmhost-x86-copr04.rdu3.fp.o - kevin - done * db.stg.aws.fedoraproject - ? :warning: Uptime: 68w5d+ # 2026-01-26 Monday/Tuesday - staging * buildhw-p10-01.stg.rdu3.fedoraproject.org - james - done * buildhw-p10-02.stg.rdu3.fp.o - james - done * bvmhost-a64-01.stg.rdu3.fp.o - james - done * bvmhost-s390x-01.stg.s390.fp.o - in progress - kevin - didn't come up, needs mainframe folks - managed to fix it - done * bvmhost-x86-01.stg.rdu3.fp.o - firmware staged - james - done * bvmhost-x86-02.stg.rdu3.fp.o - firmware staged - james - done * bvmhost-x86-03.stg.rdu3.fp.o - firmware staged - james - done * vmhost-x86-01.stg.rdu3.fp.o - firmware staged - james - done * vmhost-x86-02.stg.rdu3.fp.o - firmware staged - james - done * vmhost-x86-03.stg.rdu3.fp.o - firmware staged - james - done * vmhost-x86-04.stg.rdu3.fp.o - firmware staged - james - done * vmhost-x86-05.stg.rdu3.fp.o - firmware staged - james - done # 2026-01-27 Tuesday/Wednesday - non outage causing production Can also be done a bit on Monday. * autosign01.rdu3.fedoraproject.org - ⚠ needs robosig pass - done - kevin * backup01.rdu3.fp.o - ⚠ needs backup pass - done - kevin * dedicatedsolutions01.fp.o - kevin - done * ibiblio02.fp.o - staging firmware - kevin * ibiblio05.fp.o - (old ibm soon to be replaced, no firmware updates) - james - done * internetx02.fp.o - (no firmware) - james - done * osuosl02.fp.o - (no firmware) - kevin - done * proxy05.fp.o - ⚠ bare metal - james - done * retrace03.rdu-cc.fp.o - ⚠ bare metal - dead-ish? it is now retrace03.rdu3.fedoraproject.org - kevin - done * sign-vault01.rdu3.fp.o - ⚠ needs vault pass - done - kevin * sign-vault02.rdu3.fp.o - ⚠ needs vault pass - done - kevin * storinator01.rdu-cc.fp.o - ⚠ bare metal - skip - isn't in rdu3 yet. # Package maintainer AWS hosts - done * aarch64-test01.fedorainfracloud.org - james done * aarch64-test02.fc.o - james done * el9-test.fc.o - james done * el10-test.fc.o - james done * f42-test.fc.o - james done * f43-test.fc.o - james done * ppc64le-test.fc.o - dead-ish * ppc64le-test02.fc.o - james done * rawhide-test.fc.o - james done --- # 2026-01-29 Thu - main event 22:00 UTC - **Don't** start before: ``` ================================ Day: Thursday ================================= 2026-01-22 14:00 PST US/Pacific 2026-01-22 17:00 EST --> US/Eastern <-- 2026-01-22 22:00 GMT Europe/London 2026-01-22 22:00 UTC UTC 2026-01-22 23:00 CET Europe/Berlin 2026-01-22 23:00 CET Europe/Paris ------------------------------- New Day: Friday -------------------------------- 2026-01-23 03:30 IST Asia/Calcutta 2026-01-23 06:00 HKT Asia/Hong_Kong 2026-01-23 06:00 +08 Asia/Singapore 2026-01-23 07:00 JST Asia/Tokyo 2026-01-23 08:00 AEST Australia/Brisbane ``` ## Checklist [x] Apply updates to everything (**Do this 1h+ before**) - [x] Update status [x] Silence zabbix [x] Disable backups on backup01 (**Do this 30m+ before**) - [x] Disable updates pushes on bodhi-backend01 (**Do this 30m+ before**) - [x] Stop koschei scheduler (**Do this a few hours before**) - `ssh os-control01 oc -n koschei scale deployment scheduler --replicas=0` ### Extra [ ] Do batcave01 / vmhost-x86-01.rdu3 first? [x] Do firmware updates on some of the metal hosts [ ] Do some f42->f43 upgrades ### bkernel * buildhw-x86-01.rdu3.fedoraproject.org - needs pesignd passphrase - kevin - in progress ### builders * buildhw-a64-01.rdu3.fedoraproject.org - james - done * buildhw-a64-02.rdu3.fp.o - james - done * buildhw-a64-03.rdu3.fp.o - james - done * buildhw-a64-04.rdu3.fp.o - james - done * buildhw-a64-05.rdu3.fp.o - james - done * buildhw-a64-06.rdu3.fp.o - james - done * buildhw-a64-07.rdu3.fp.o - james - done * buildhw-x86-01.rdu3.fp.o - firmware staged - james - done * buildhw-x86-02.rdu3.fp.o - firmware staged - james - done * buildhw-x86-03.rdu3.fp.o - firmware staged - james - done * buildhw-x86-04.rdu3.fp.o - firmware staged - james - done * buildhw-x86-08.rdu3.fp.o - firmware staged - james - done * buildhw-x86-09.rdu3.fp.o - firmware staged - james - done * buildhw-x86-10.rdu3.fp.o - firmware staged - james - done * buildhw-x86-12.rdu3.fp.o - firmware staged - james - done * buildhw-x86-13.rdu3.fp.o - firmware staged - james - done * bvmhost-s390x-01.s390.fp.o - ⚠ statistically speaking likely will not come back up nicely - kevin - done * bvmhost-a64-01.rdu3.fp.o - james - done * bvmhost-a64-02.rdu3.fp.o - james - done * bvmhost-a64-03.rdu3.fp.o - james - done: needed luks passwd * bvmhost-a64-04.rdu3.fp.o - james - done * bvmhost-a64-05.rdu3.fp.o - james - done * bvmhost-p09-05.rdu3.fp.o - kevin - done (and upgraded to f43) * bvmhost-x86-05.rdu3.fp.o - firmware staged - james - done * bvmhost-x86-06.rdu3.fp.o - firmware staged - james - done * bvmhost-x86-riscv01.rdu3.fp.o - firmware staged - james - done ### builders - power10 lpars, managed by vHMC **Don't** update these unless vHMC is happy. * bvmhost-p10-01.rdu3.fp.o - no ipmi, ⚠ iscsi ordering with libvirt. - kevin - done * bvmhost-p10-02.rdu3.fp.o - no ipmi? - kevin - done * buildhw-p10-01.stg.rdu3.fp.o - already done * buildhw-p10-02.stg.rdu3.fp.o - already done ### non builders * bvmhost-x86-01.rdu3.fedoraproject.org - firmware staged - james - done * bvmhost-x86-02.rdu3.fp.o - firmware staged - james - done: kernels oops; then worked. * bvmhost-x86-03.rdu3.fp.o - firmware staged - james - done * bvmhost-x86-04.rdu3.fp.o - firmware staged - kevin - done * vmhost-x86-01.rdu3.fp.o (batcave01/bastion01/ipa01/noc01) - firmware staged - ⚠ nagios shush won't work - kevin - in progress * vmhost-x86-02.rdu3.fp.o (bastion02/ipa02), also -e nodns=true - firmware staged - kevin - in progress * vmhost-x86-03.rdu3.fp.o (log01) - firmware staged - kevin - in progress * vmhost-x86-04.rdu3.fp.o (db01) - firmware staged - kevn - done * vmhost-x86-05.rdu3.fp.o () - firmware staged - kevin - in progress * vmhost-x86-iso01.fp.o - firmware staged - james - done: needed new luks passwd * vmhost-x86-iso02.fp.o - firmware staged - james - done * vmhost-x86-iso03.fp.o - OLD iad2 passwd / old UI / firmware failed - james - done * vmhost-x86-iso04.fp.o - firmware staged - james - done * proxy3*-40 ( aws ) - james - done * proxy09 - james - done Possible things if there's time: * Firmware - https://docs.fedoraproject.org/en-US/infra/howtos/idrac-firmware-update/#_using_the_firmware_proxy_in_rdu3 * Needs to be done _before_ rebooting the hardware as they apply on reboot. Sometimes there's deps and it has to reboot more than once. * rhel10 reinstalls - * koji hubs upgrade to f43 ( koji01/02/01.stg/riscv01 )- kevin - done * bvmhost-p09-05 upgrade to f43 - kevin - done * bvmhost-s390x-01 upgrade to f43 - kevin - done * compose-* upgrade to f43 * ipsilon03 and ipsilon*stg upgrade to f43 * oci-*registries upgrade to f43 # Post checklist [x] start koschei scheduler [x] enable backups on backup01 [x] enable updates pushes on bodhi-backend01 [x] update status [x] unsilence zabbix --- # Weird Errors * Post quantum RPM GPG key problems on el10 -- need to update redhat-release before doing an update. * Multiple stg (8-16?) VMs showed "weirdness" in the updates phase. Shows an ansible error like: `fatal: [proxy02.fedoraproject.org]: FAILED! => {"ansible_job_id": "j366090400113.3990131", "changed": false, "failures": [], "finished": 1, "msg": "Unknown Error occurred: Could not run transaction.", "rc": 1, "results": [], "results_file": "/root/.ansible_async/j366090400113.3990131", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}` but when I look the results file isn't there, and dnf shows no updates. * Both prod. bvmhost-x86 05 and 06 had the same `"Could not run transaction"` for their VMs. * Seems to happen to most (all?) of the proxies, was a dnf failure. Interesting output: `>>> Running %post scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64 >>> Finished %post scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64 >>> Scriptlet output: >>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory >>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory >>> >>> Running %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64 >>> Finished %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64 >>> Scriptlet output: >>> Warning: The unit file, source configuration file or drop-ins of sshd.servic >>> >>> Running %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64 >>> Non-critical error in %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x >>> Scriptlet output: >>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory >>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory >>> >>> [RPM] %triggerin(freeipa-client-4.13.0-2.fc43.x86_64) scriptlet failed, exit And then at the end of the transaction it says: Transaction failed: Rpm transaction failed. `