---
title: updates for week of 2025-05-19

---

# Resources
- [ ] - outage ticket: https://pagure.io/fedora-infrastructure/issue/12455
- [ ] - status update: https://github.com/fedora-infra/statusfpo/pull/59
- [ ] - announcement sent to devel-announce/infrastructure lists

# SOP-ish (put this in docs)

- log into batcave01
- Put your name next to the machine here (with "in progress").
- run: `sudo rbac-playbook vhost_update_reboot.yml`
- - If it's bare metal (You'll find out because libvirt will give an erorr and the vhost_* playbook will fail) Kevin needs to do it.
- - Authenticate (should be 2factor)
- - Put the full hostname into the prompt, to start the update
- - Monitor the output, and in another window you might want to monitor the machine (see below)
- - After the update has finished, you'll be prompted a second time for a host to reboot. Put **the same hostname** into this prompt
- To monitor a machine you are update/rebooting in another window (also from batcave01) you can run: `mtr --displaymode 1 -i 4 <host>`
- - The main thing to look for is if it goes away during the upgrade part (very bad), or if it takes too long to reboot (all hosts are different, but 5m is fine and over 10m is usually bad).
- If anything weird happens, speak to Kevin
- When a host fails to come back after the reboot it's usually due to luks passwds needed
- - Which requires console access via (without the -stg if not in staging) **noc01**: `ipmitool -U admin -H <host>-stg.mgmt.iad2.fedoraproject.org -I lanplus shell` ... you'll need the passwds if you are doing this bit. (and run `sol activate`)
- - If `ipmitool` doesn't work you can use: `sshuttle 172.23.1.0/24 -r noc-cc01.rdu-cc.fedoraproject.org -v`

updates for week of 2025-05-22

## openqa - sometime - schedule with adamw

* openqa-a64-worker01.iad2.fedoraproject.org -
* openqa-a64-worker02.iad2.fedoraproject.org -
* openqa-a64-worker03.iad2.fedoraproject.org -
* openqa-a64-worker04.iad2.fedoraproject.org -
* openqa-p09-worker01.iad2.fedoraproject.org -
* openqa-p09-worker02.iad2.fedoraproject.org -
* openqa-x86-worker01.iad2.fedoraproject.org -
* openqa-x86-worker02.iad2.fedoraproject.org -
* openqa-x86-worker03.iad2.fedoraproject.org -
* openqa-x86-worker04.iad2.fedoraproject.org -
* openqa-x86-worker05.iad2.fedoraproject.org -
* openqa-x86-worker06.iad2.fedoraproject.org -
* qvmhost-x86-01.iad2.fedoraproject.org -
* qvmhost-x86-02.iad2.fedoraproject.org -

## copr - sometime - schedule / check with #buildsys:fedoraproject.org

* vmhost-p08-copr01.rdu-cc.fedoraproject.org
* vmhost-p08-copr02.rdu-cc.fedoraproject.org
* vmhost-p09-copr01.rdu-cc.fedoraproject.org
* vmhost-x86-copr01.rdu-cc.fedoraproject.org
* vmhost-x86-copr02.rdu-cc.fedoraproject.org
* vmhost-x86-copr03.rdu-cc.fedoraproject.org
* vmhost-x86-copr04.rdu-cc.fedoraproject.org

# 2025-05-19 Monday - staging

* bvmhost-a64-01.stg.iad2.fedoraproject.org - james - done
* bvmhost-p09-01.stg.iad2.fedoraproject.org - james - done
* bvmhost-s390x-01.stg.s390.fedoraproject.org - james - done
* bvmhost-x86-01.stg.iad2.fedoraproject.org - james - done
* bvmhost-x86-02.stg.iad2.fedoraproject.org - james - done
* bvmhost-x86-03.stg.iad2.fedoraproject.org - james - done
* bvmhost-x86-05.stg.iad2.fedoraproject.org - james - done - sign-vault01.stg doesn't come back on it's own
* vmhost-x86-01.stg.iad2.fedoraproject.org - james - done
* vmhost-x86-02.stg.iad2.fedoraproject.org - james - done
* vmhost-x86-05.stg.iad2.fedoraproject.org - james - done - ocp01 unreachable
* vmhost-x86-06.stg.iad2.fedoraproject.org - james - done - ocp02 unreachable
* vmhost-x86-07.stg.iad2.fedoraproject.org - james - done - ocp03 unreachable
* vmhost-x86-08.stg.iad2.fedoraproject.org - james - done
* vmhost-x86-09.stg.iad2.fedoraproject.org - james - done
* vmhost-x86-11.stg.iad2.fedoraproject.org - james - done - centos-ipa-client01, 03, 04
* vmhost-x86-12.stg.iad2.fedoraproject.org - james - done - centos-ipa-client02

# 2025-05-20/21 Tuesday/Wednesday - non outage causing production

* autosign02.iad2.fedoraproject.org - needs robosig pass - kevin - done
* backup01.iad2.fedoraproject.org - needs backup pass - kevin - done
* dedicatedsolutions01.fedoraproject.org - james - done
* ibiblio02.fedoraproject.org - james - done
* ibiblio05.fedoraproject.org - james - done
* internetx02.fedoraproject.org - james - done
* osuosl02.fedoraproject.org - james - done
* proxy05.fedoraproject.org - bare metal - ⚠
* retrace03.rdu-cc.fedoraproject.org - bare metal - ⚠
* sign-vault02.iad2.fedoraproject.org - needs vault pass - kevin - done
* storinator01.rdu-cc.fedoraproject.org - bare metal - ⚠
* bvmhost-x86-riscv01.iad2.fedoraproject.org - james - done
* vmhost-x86-cc02.rdu-cc.fedoraproject.org - james - done: luks needed
* vmhost-x86-cc05.rdu-cc.fedoraproject.org - james - done
* vmhost-x86-cc06.rdu-cc.fedoraproject.org - james - done
* maintainer_test group - ??

---

# 2025-03-22 Thursday - main event

21:00 UTC - **Don't** start before:

================================ Day: Thursday =================================
2025-05-22 14:00 PDT      US/Pacific        
2025-05-22 17:00 EDT  --> US/Eastern         <--
2025-05-22 21:00 UTC      UTC               
2025-05-22 22:00 BST      Europe/London     
2025-05-22 23:00 CEST     Europe/Berlin     
2025-05-22 23:00 CEST     Europe/Paris      
------------------------------- New Day: Friday --------------------------------
2025-05-23 02:30 IST      Asia/Calcutta     
2025-05-23 05:00 HKT      Asia/Hong_Kong    
2025-05-23 05:00 +08      Asia/Singapore    
2025-05-23 06:00 JST      Asia/Tokyo        
2025-05-23 07:00 AEST     Australia/Brisbane

## Checklist

[x] Apply updates to everything (pre outage)

[x] Update status

[x] Silence nagios

[x] Disable backups on backup01

[x] Disable updates pushes on bodhi-backend01

[x] Stop koschei scheduler

### Extra

[x] Add CPUs to db01
[x] Do batcave01 and vmhost-x86-08.iad2 first.
[x] Take down wiki01/02 and save xml / disk in case

### builders

* bkernel01.iad2.fedoraproject.org - needs pesignd passphrase - kevin - done
* bkernel02.iad2.fedoraproject.org - needs pesignd passphrase - kevin - done
* buildhw-a64-03.iad2.fedoraproject.org - james - done
* buildhw-a64-04.iad2.fedoraproject.org - james - done
* buildhw-a64-05.iad2.fedoraproject.org - james - done
* buildhw-a64-06.iad2.fedoraproject.org - james - done
* buildhw-x86-01.iad2.fedoraproject.org - james - done
* buildhw-x86-02.iad2.fedoraproject.org - james - done
* buildhw-x86-03.iad2.fedoraproject.org - james - done
* buildhw-x86-04.iad2.fedoraproject.org - james - done
* buildhw-x86-05.iad2.fedoraproject.org - james - done
* buildhw-x86-06.iad2.fedoraproject.org - james - done
* buildhw-x86-07.iad2.fedoraproject.org - james - done
* buildhw-x86-08.iad2.fedoraproject.org - james - done
* buildhw-x86-09.iad2.fedoraproject.org - james - done
* buildhw-x86-10.iad2.fedoraproject.org - james - done
* buildhw-x86-11.iad2.fedoraproject.org - james - done
* buildhw-x86-12.iad2.fedoraproject.org - james - done
* buildhw-x86-13.iad2.fedoraproject.org - james - done
* buildhw-x86-14.iad2.fedoraproject.org - james - done
* buildhw-x86-15.iad2.fedoraproject.org - james - done
* buildhw-x86-16.iad2.fedoraproject.org - james - done

* bvmhost-a64-01.iad2.fedoraproject.org - james - done
* bvmhost-a64-02.iad2.fedoraproject.org - james - done: grub: Can't find cmd: linuxefi
* bvmhost-a64-03.iad2.fedoraproject.org - james - done
* bvmhost-a64-04.iad2.fedoraproject.org - james - done: grub: fix:`chassis bootdev disk` and `power cycle`
* bvmhost-x86-06.iad2.fedoraproject.org - james - done
* bvmhost-x86-07.iad2.fedoraproject.org - james - done
* bvmhost-p09-01.iad2.fedoraproject.org - james - done
* bvmhost-p09-02.iad2.fedoraproject.org - james - done
* bvmhost-p09-03.iad2.fedoraproject.org - james - done
* bvmhost-p09-04.iad2.fedoraproject.org - james - done
* bvmhost-p09-05.iad2.fedoraproject.org - james - done: brought up wired network
* bvmhost-s390x-01.s390.fedoraproject.org - nirik - done - ⚠ statistically speaking likely will not come back up nicely - it didn't

### non builders

* bvmhost-x86-01.iad2.fedoraproject.org - james - done
* bvmhost-x86-02.iad2.fedoraproject.org - james - done
* bvmhost-x86-03.iad2.fedoraproject.org - james - done
* bvmhost-x86-04.iad2.fedoraproject.org - james - done
* bvmhost-x86-05.iad2.fedoraproject.org - james - done
* vmhost-x86-01.iad2.fedoraproject.org (bastion01/batcave01) - kevin - done
* vmhost-x86-02.iad2.fedoraproject.org - james - done
* vmhost-x86-03.iad2.fedoraproject.org - james - done
* vmhost-x86-04.iad2.fedoraproject.org - IPA! Also contains noc01, and thus. nagios shush does't work anymore. - kevin - done
* vmhost-x86-05.iad2.fedoraproject.org - james - done
* vmhost-x86-06.iad2.fedoraproject.org - IPA! - james - done
* vmhost-x86-08.iad2.fedoraproject.org - db01 - kevin - done
* vmhost-x86-cc01.rdu-cc.fedoraproject.org - james - DNS push failed ⚠
* vmhost-x86-cc03.rdu-cc.fedoraproject.org - james - done
* proxy3*-40 ( aws ) - kevin - done

## Wiki

For both wiki01 and wiki02:

[x] Change ansible playbooks to be F42
[x] Shutdown and save VM
[x] Run wiki playbook (maybe a 2nd time if `nmcli c up eth0` is needed) (with -l wiki01.iad2.fedoraproject.org to only bring up 01 first)
[x] Run upgrade: `/srv/web/fp-wiki/maintenance/run.php --conf /srv/web/fp-wiki/LocalSettings.php update --skip-config-validation`
[x] Look at wiki and see that it's running (cry if not).

# Post checklist

[x] start koschei scheduler

[x] enable backups on backup01

[x] enable updates pushes on bodhi-backend01

[x] update status

[x] unsilence nagios

---
# Weird Errors