updates for week of 2025-01-27 = openqa - sometime * openqa-a64-worker01.iad2.fedoraproject.org * openqa-a64-worker02.iad2.fedoraproject.org * openqa-a64-worker03.iad2.fedoraproject.org * openqa-a64-worker04.iad2.fedoraproject.org * openqa-p09-worker01.iad2.fedoraproject.org * openqa-p09-worker02.iad2.fedoraproject.org * openqa-x86-worker01.iad2.fedoraproject.org * openqa-x86-worker02.iad2.fedoraproject.org * openqa-x86-worker03.iad2.fedoraproject.org * openqa-x86-worker04.iad2.fedoraproject.org * openqa-x86-worker05.iad2.fedoraproject.org * openqa-x86-worker06.iad2.fedoraproject.org * qvmhost-x86-01.iad2.fedoraproject.org * qvmhost-x86-02.iad2.fedoraproject.org = copr - sometime * vmhost-p08-copr01.rdu-cc.fedoraproject.org * vmhost-p08-copr02.rdu-cc.fedoraproject.org * vmhost-p09-copr01.rdu-cc.fedoraproject.org * vmhost-x86-copr01.rdu-cc.fedoraproject.org * vmhost-x86-copr02.rdu-cc.fedoraproject.org * vmhost-x86-copr03.rdu-cc.fedoraproject.org * vmhost-x86-copr04.rdu-cc.fedoraproject.org = 2025-01-27 Monday - staging * bvmhost-a64-01.stg.iad2.fedoraproject.org - kevin - done * bvmhost-p09-01.stg.iad2.fedoraproject.org - kevin - done * bvmhost-s390x-01.stg.s390.fedoraproject.org - kevin - done * bvmhost-x86-01.stg.iad2.fedoraproject.org - kevin - done * bvmhost-x86-02.stg.iad2.fedoraproject.org - kevin - done * bvmhost-x86-03.stg.iad2.fedoraproject.org - kevin - done * bvmhost-x86-05.stg.iad2.fedoraproject.org - kevin - done * vmhost-x86-01.stg.iad2.fedoraproject.org - kevin - done * vmhost-x86-02.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-05.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-06.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-07.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-08.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-09.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-11.stg.iad2.fedoraproject.org - jantill - done * vmhost-x86-12.stg.iad2.fedoraproject.org - jantill - done = 2025-01-28 Tuesday - non outage causing * autosign02.iad2.fedoraproject.org - needs robosig pass - kevin - done * backup01.iad2.fedoraproject.org - needs backup pass - kevin - done * buildhw-a64-01.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-02.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-07.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-08.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-09.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-10.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-11.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-12.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-13.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-19.iad2.fedoraproject.org - perhaps we just remove these now * buildhw-a64-20.iad2.fedoraproject.org - perhaps we just remove these now * dedicatedsolutions01.fedoraproject.org - kevin - done * ibiblio02.fedoraproject.org - kevin - done * ibiblio05.fedoraproject.org - kevin - done * internetx02.fedoraproject.org - kevin - done * osuosl02.fedoraproject.org - kevin - done * proxy05.fedoraproject.org - kevin - done * retrace03.rdu-cc.fedoraproject.org - jantill - fail libvirt import * sign-vault02.iad2.fedoraproject.org - needs vault pass - kevin - done * storinator01.rdu-cc.fedoraproject.org - jantill - failed libvirt import * bvmhost-x86-riscv01.iad2.fedoraproject.org - jantill - done * vmhost-x86-cc02.rdu-cc.fedoraproject.org - jantill - done * vmhost-x86-cc05.rdu-cc.fedoraproject.org - jantill - done * vmhost-x86-cc06.rdu-cc.fedoraproject.org - jantill - done * maintainer_test group - kevin - done = 2025-01-29 Wed [x] Apply updates to everything - kevin - in progress [x] update status [x] silence nagios [x] disable backups on backup01 [x] disable updates pushes on bodhi-backend01 [ ] stop koschei scheduler] builders: bkernel01.iad2.fedoraproject.org - needs pesignd passphrase - kevin - in progress bkernel02.iad2.fedoraproject.org - needs pesignd passphrase buildhw-a64-03.iad2.fedoraproject.org - kevin - done buildhw-a64-04.iad2.fedoraproject.org - kevin - done buildhw-a64-05.iad2.fedoraproject.org - kevin - done buildhw-a64-06.iad2.fedoraproject.org - kevin - done buildhw-x86-01.iad2.fedoraproject.org - kevin - done buildhw-x86-02.iad2.fedoraproject.org - kevin - done buildhw-x86-03.iad2.fedoraproject.org - kevin - done buildhw-x86-04.iad2.fedoraproject.org - kevin - done buildhw-x86-05.iad2.fedoraproject.org - kevin - done buildhw-x86-06.iad2.fedoraproject.org - kevin - done buildhw-x86-07.iad2.fedoraproject.org - kevin - done buildhw-x86-08.iad2.fedoraproject.org - kevin - done buildhw-x86-09.iad2.fedoraproject.org - kevin - done buildhw-x86-10.iad2.fedoraproject.org - kevin - done buildhw-x86-11.iad2.fedoraproject.org - kevin - done buildhw-x86-12.iad2.fedoraproject.org - kevin - done buildhw-x86-13.iad2.fedoraproject.org - kevin - done buildhw-x86-14.iad2.fedoraproject.org - kevin - done buildhw-x86-15.iad2.fedoraproject.org - kevin - done buildhw-x86-16.iad2.fedoraproject.org - kevin - done bvmhost-a64-01.iad2.fedoraproject.org - kevin - done bvmhost-a64-02.iad2.fedoraproject.org - kevin - done bvmhost-a64-03.iad2.fedoraproject.org - kevin - done bvmhost-a64-04.iad2.fedoraproject.org - kevin - done bvmhost-x86-06.iad2.fedoraproject.org - kevin - done bvmhost-x86-07.iad2.fedoraproject.org - kevin - done bvmhost-p09-01.iad2.fedoraproject.org - kevin - done bvmhost-p09-02.iad2.fedoraproject.org - kevin - done bvmhost-p09-03.iad2.fedoraproject.org - kevin - done bvmhost-p09-04.iad2.fedoraproject.org - kevin - done bvmhost-p09-05.iad2.fedoraproject.org - kevin - done bvmhost-s390x-01.s390.fedoraproject.org - statistically speaking likely will not come back up nicely - kevin - done non builders: bvmhost-x86-01.iad2.fedoraproject.org - kevin - done bvmhost-x86-02.iad2.fedoraproject.org - kevin - done bvmhost-x86-03.iad2.fedoraproject.org - kevin - done bvmhost-x86-04.iad2.fedoraproject.org - kevin - done bvmhost-x86-05.iad2.fedoraproject.org - kevin - done vmhost-x86-01.iad2.fedoraproject.org - kevin - done ^ might reinstall with rhel9 but need to migrate batcave01/bastion01 off while I do vmhost-x86-02.iad2.fedoraproject.org - kevin - done vmhost-x86-03.iad2.fedoraproject.org - kevin - done vmhost-x86-04.iad2.fedoraproject.org - kevin - done vmhost-x86-05.iad2.fedoraproject.org - kevin - done vmhost-x86-06.iad2.fedoraproject.org - kevin - done vmhost-x86-08.iad2.fedoraproject.org - kevin - done vmhost-x86-cc01.rdu-cc.fedoraproject.org - kevin - done vmhost-x86-cc03.rdu-cc.fedoraproject.org - kevin - done proxy3*-40 ( aws ) - kevin - in progress - 35,36,39 didn't come back - fixed them by rebooting so they would see nvme drives in the right order possible 40/41 upgrades: kojipkgs01.iad2.fedoraproject.org - kevin - done kojipkgs02.iad2.fedoraproject.org - kevin - done oci-candidate-registry01.iad2.fedoraproject.org - kevin - done oci-registry01.iad2.fedoraproject.org - kevin - done oci-registry02.iad2.fedoraproject.org - kevin - done proxies - will see how proxy02.stg looks smtp-auth-cc-rdu01.fedoraproject.org - kevin - done [ ] start koschei scheduler [x] enable backups on backup01 [x] enable updates pushes on bodhi-backend01 [x] update status [ ] unsilence nagios