---
title: updates for week of 2026-01-26

---

# Resources
- [x] - outage ticket: https://forge.fedoraproject.org/infra/tickets/issues/13067
- [x] - status update: done
- [x] - announcement sent to devel-announce/infrastructure lists

# SOP-ish (put this in docs)

- log into batcave01
- Put your name next to the machine here (with "in progress").
- run: `sudo rbac-playbook vhost_update_reboot.yml`
- Also see: https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/massupgrade/

updates for week of 2026-01-26

## openqa - sometime - schedule with adamw

* openqa-a64-worker01.rdu3.fedoraproject.org - done - adamw
* openqa-a64-worker02.rdu3.fp.o -done - adamw
* openqa-x86-worker01.rdu3.fp.o - firmware staged - done - adamw
* openqa-x86-worker02.rdu3.fp.o - firmware staged - done - adamw
* openqa-x86-worker03.rdu3.fp.o - firmware staged - done - adamw
* openqa-x86-worker04.rdu3.fp.o - firmware staged - done - adamw
* openqa-x86-worker05.rdu3.fp.o - firmware staged - done - adamw
* qvmhost-x86-01.rdu3.fp.o - firmware staged - done - adamw

## copr_hypervisor - sometime - schedule / check with #buildsys:fedoraproject.org

* vmhost-p09-copr01.rdu3.fedoraproject.org - kevin - done (and upgraded to f43)
* vmhost-p09-copr02.rdu3.fp.o - kevin - done
* vmhost-p09-copr03.rdu3.fp.o - kevin - done
* vmhost-p09-copr04.rdu3.fp.o - kevin - done
* vmhost-x86-copr03.rdu3.fp.o - kevin - done
* vmhost-x86-copr04.rdu3.fp.o - kevin - done
* db.stg.aws.fedoraproject - ? :warning: Uptime: 68w5d+

# 2026-01-26 Monday/Tuesday - staging

* buildhw-p10-01.stg.rdu3.fedoraproject.org - james - done
* buildhw-p10-02.stg.rdu3.fp.o - james - done
* bvmhost-a64-01.stg.rdu3.fp.o - james - done
* bvmhost-s390x-01.stg.s390.fp.o - in progress - kevin - didn't come up, needs mainframe folks - managed to fix it - done
* bvmhost-x86-01.stg.rdu3.fp.o - firmware staged - james - done
* bvmhost-x86-02.stg.rdu3.fp.o - firmware staged - james - done
* bvmhost-x86-03.stg.rdu3.fp.o - firmware staged - james - done
* vmhost-x86-01.stg.rdu3.fp.o - firmware staged - james - done
* vmhost-x86-02.stg.rdu3.fp.o - firmware staged - james - done
* vmhost-x86-03.stg.rdu3.fp.o - firmware staged - james - done
* vmhost-x86-04.stg.rdu3.fp.o - firmware staged - james - done
* vmhost-x86-05.stg.rdu3.fp.o - firmware staged - james - done

# 2026-01-27 Tuesday/Wednesday - non outage causing production

Can also be done a bit on Monday.

* autosign01.rdu3.fedoraproject.org - ⚠ needs robosig pass - done - kevin
* backup01.rdu3.fp.o - ⚠ needs backup pass - done - kevin
* dedicatedsolutions01.fp.o - kevin - done
* ibiblio02.fp.o - staging firmware - kevin
* ibiblio05.fp.o - (old ibm soon to be replaced, no firmware updates) - james - done
* internetx02.fp.o - (no firmware) - james - done
* osuosl02.fp.o - (no firmware) - kevin - done
* proxy05.fp.o - ⚠ bare metal - james - done
* retrace03.rdu-cc.fp.o - ⚠ bare metal - dead-ish? it is now retrace03.rdu3.fedoraproject.org - kevin - done
* sign-vault01.rdu3.fp.o - ⚠ needs vault pass - done - kevin
* sign-vault02.rdu3.fp.o - ⚠ needs vault pass - done - kevin
* storinator01.rdu-cc.fp.o - ⚠ bare metal - skip - isn't in rdu3 yet.

# Package maintainer AWS hosts - done
* aarch64-test01.fedorainfracloud.org - james done
* aarch64-test02.fc.o - james done
* el9-test.fc.o - james done
* el10-test.fc.o - james done
* f42-test.fc.o - james done
* f43-test.fc.o - james done
* ppc64le-test.fc.o - dead-ish
* ppc64le-test02.fc.o - james done
* rawhide-test.fc.o - james done

---

# 2026-01-29 Thu - main event

22:00 UTC - **Don't** start before:

================================ Day: Thursday =================================
2026-01-22 14:00 PST      US/Pacific        
2026-01-22 17:00 EST  --> US/Eastern         <--
2026-01-22 22:00 GMT      Europe/London     
2026-01-22 22:00 UTC      UTC               
2026-01-22 23:00 CET      Europe/Berlin     
2026-01-22 23:00 CET      Europe/Paris      
------------------------------- New Day: Friday --------------------------------
2026-01-23 03:30 IST      Asia/Calcutta     
2026-01-23 06:00 HKT      Asia/Hong_Kong    
2026-01-23 06:00 +08      Asia/Singapore    
2026-01-23 07:00 JST      Asia/Tokyo        
2026-01-23 08:00 AEST     Australia/Brisbane

## Checklist

[x] Apply updates to everything (**Do this 1h+ before**) -

[x] Update status

[x] Silence zabbix

[x] Disable backups on backup01 (**Do this 30m+ before**) -

[x] Disable updates pushes on bodhi-backend01 (**Do this 30m+ before**) -

[x] Stop koschei scheduler (**Do this a few hours before**) - `ssh os-control01 oc -n koschei scale deployment scheduler --replicas=0`

### Extra

[ ] Do batcave01 / vmhost-x86-01.rdu3 first?

[x] Do firmware updates on some of the metal hosts

[ ] Do some f42->f43 upgrades

### bkernel

* buildhw-x86-01.rdu3.fedoraproject.org - needs pesignd passphrase - kevin - in progress

### builders

* buildhw-a64-01.rdu3.fedoraproject.org - james - done
* buildhw-a64-02.rdu3.fp.o - james - done
* buildhw-a64-03.rdu3.fp.o - james - done
* buildhw-a64-04.rdu3.fp.o - james - done
* buildhw-a64-05.rdu3.fp.o - james - done
* buildhw-a64-06.rdu3.fp.o - james - done
* buildhw-a64-07.rdu3.fp.o - james - done
* buildhw-x86-01.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-02.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-03.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-04.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-08.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-09.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-10.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-12.rdu3.fp.o - firmware staged - james - done
* buildhw-x86-13.rdu3.fp.o - firmware staged - james - done

* bvmhost-s390x-01.s390.fp.o - ⚠ statistically speaking likely will not come back up nicely - kevin - done
* bvmhost-a64-01.rdu3.fp.o - james - done
* bvmhost-a64-02.rdu3.fp.o - james - done
* bvmhost-a64-03.rdu3.fp.o - james - done: needed luks passwd
* bvmhost-a64-04.rdu3.fp.o - james - done
* bvmhost-a64-05.rdu3.fp.o - james - done
* bvmhost-p09-05.rdu3.fp.o - kevin - done (and upgraded to f43)
* bvmhost-x86-05.rdu3.fp.o - firmware staged - james - done
* bvmhost-x86-06.rdu3.fp.o - firmware staged - james - done
* bvmhost-x86-riscv01.rdu3.fp.o - firmware staged - james - done

### builders - power10 lpars, managed by vHMC

**Don't** update these unless vHMC is happy.

* bvmhost-p10-01.rdu3.fp.o - no ipmi, ⚠ iscsi ordering with libvirt. - kevin - done
* bvmhost-p10-02.rdu3.fp.o - no ipmi? - kevin - done
* buildhw-p10-01.stg.rdu3.fp.o - already done
* buildhw-p10-02.stg.rdu3.fp.o - already done

### non builders

* bvmhost-x86-01.rdu3.fedoraproject.org - firmware staged - james - done
* bvmhost-x86-02.rdu3.fp.o - firmware staged - james - done: kernels oops; then worked.
* bvmhost-x86-03.rdu3.fp.o - firmware staged - james - done
* bvmhost-x86-04.rdu3.fp.o - firmware staged - kevin - done
* vmhost-x86-01.rdu3.fp.o (batcave01/bastion01/ipa01/noc01) - firmware staged - ⚠ nagios shush won't work - kevin - in progress
* vmhost-x86-02.rdu3.fp.o (bastion02/ipa02), also -e nodns=true - firmware staged - kevin - in progress
* vmhost-x86-03.rdu3.fp.o (log01) - firmware staged - kevin - in progress
* vmhost-x86-04.rdu3.fp.o (db01) - firmware staged - kevn - done
* vmhost-x86-05.rdu3.fp.o () - firmware staged - kevin - in progress
* vmhost-x86-iso01.fp.o - firmware staged - james - done: needed new luks passwd
* vmhost-x86-iso02.fp.o - firmware staged - james - done
* vmhost-x86-iso03.fp.o - OLD iad2 passwd / old UI / firmware failed - james - done
* vmhost-x86-iso04.fp.o - firmware staged - james - done
* proxy3*-40 ( aws ) - james - done
* proxy09 - james - done

Possible things if there's time:
* Firmware - https://docs.fedoraproject.org/en-US/infra/howtos/idrac-firmware-update/#_using_the_firmware_proxy_in_rdu3
* Needs to be done _before_ rebooting the hardware as they apply on reboot. Sometimes there's deps and it has to reboot more than once.
* rhel10 reinstalls -
* koji hubs upgrade to f43 ( koji01/02/01.stg/riscv01 )- kevin - done
* bvmhost-p09-05 upgrade to f43 - kevin - done
* bvmhost-s390x-01 upgrade to f43 - kevin - done
* compose-* upgrade to f43
* ipsilon03 and ipsilon*stg upgrade to f43
* oci-*registries upgrade to f43

# Post checklist

[x] start koschei scheduler

[x] enable backups on backup01

[x] enable updates pushes on bodhi-backend01

[x] update status

[x] unsilence zabbix

---
# Weird Errors

* Post quantum RPM GPG key problems on el10 -- need to update redhat-release before doing an update.
* Multiple stg (8-16?) VMs showed "weirdness" in the updates phase. Shows an ansible error like: `fatal: [proxy02.fedoraproject.org]: FAILED! => {"ansible_job_id": "j366090400113.3990131", "changed": false, "failures": [], "finished": 1, "msg": "Unknown Error occurred: Could not run transaction.", "rc": 1, "results": [], "results_file": "/root/.ansible_async/j366090400113.3990131", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}` but when I look the results file isn't there, and dnf shows no updates.
* Both prod. bvmhost-x86 05 and 06 had the same `"Could not run transaction"` for their VMs.
* Seems to happen to most (all?) of the proxies, was a dnf failure. Interesting output:
`>>> Running %post scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64
>>> Finished %post scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64
>>> Scriptlet output:
>>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory
>>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory
>>>
>>> Running %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64
>>> Finished %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64
>>> Scriptlet output:
>>> Warning: The unit file, source configuration file or drop-ins of sshd.servic
>>>
>>> Running %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x86_64
>>> Non-critical error in %triggerin scriptlet: freeipa-client-0:4.13.0-2.fc43.x
>>> Scriptlet output:
>>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory
>>> sed: can't read /etc/ssh/ssh_config.d/04-ipa.conf: No such file or directory
>>>
>>> [RPM] %triggerin(freeipa-client-4.13.0-2.fc43.x86_64) scriptlet failed, exit

And then at the end of the transaction it says:

Transaction failed: Rpm transaction failed.
`