# mass update/reboots - week of 2022-02-14

## monday - 2022-02-14

Staging hosts:

- [x] overall update * - in progress - kevin

- [x] new staging mgmt admin password, see bitwarden or talk to kevin
- [x] update drac and bios and password on each dell machine, make sure root account is disabled.

- [x] bvmhost-a64-01.stg.iad2.fedoraproject.org
- [x] bvmhost-p08-01.stg.iad2.fedoraproject.org
- [x] bvmhost-x86-01.stg.iad2.fedoraproject.org
- [x] bvmhost-x86-02.stg.iad2.fedoraproject.org
- [x] bvmhost-x86-03.stg.iad2.fedoraproject.org
- [x] bvmhost-x86-04.stg.iad2.fedoraproject.org - kevin done
- [x] bvmhost-x86-05.stg.iad2.fedoraproject.org - mobrien
- [x] vmhost-x86-01.stg.iad2.fedoraproject.org
- [x] vmhost-x86-02.stg.iad2.fedoraproject.org
- [x] vmhost-x86-03.stg.iad2.fedoraproject.org
- [x] vmhost-x86-04.stg.iad2.fedoraproject.org
- [x] vmhost-x86-05.stg.iad2.fedoraproject.org
- [x] vmhost-x86-06.stg.iad2.fedoraproject.org
- [x] vmhost-x86-07.stg.iad2.fedoraproject.org
- [x] vmhost-x86-08.stg.iad2.fedoraproject.org
- [x] vmhost-x86-09.stg.iad2.fedoraproject.org
- [x] vmhost-x86-10.stg.iad2.fedoraproject.org - Everything done, drac again
- [x] vmhost-x86-11.stg.iad2.fedoraproject.org
- [x] vmhost-x86-12.stg.iad2.fedoraproject.org - done - kevin, drac gateway problem, host down

## tuesday - 2022-02-15

Non outage causing hosts (proxies, colovirt, vmhosts that don't cause outage when things are taken out of dns)

Will do vhost_update then vhost_reboot on each

- [x] dedicatedsolutions01.fedoraproject.org
- [x] ibiblio01.fedoraproject.org - done kevin
- [x] ibiblio05.fedoraproject.org - done kevin
- [x] internetx01.fedoraproject.org - done - kevin
- [-] osuosl01.fedoraproject.org - in progress - kevin ( waiting on vpn access for mgmt )
- [-] osuosl02.fedoraproject.org - in progress - kevin ( waiting on vpn access for mgmt )
- [x] virthost-cc-rdu01.fedoraproject.org - done - kevin (no mgmt access?)
- [x] virthost-cc-rdu03.fedoraproject.org - in progress - kevin (no mgmt access?)
- [x] virthost-rdu01.fedoraproject.org - in progress - kevin (has mgmt, but can't login to it)
- [x] vmhost-x86-cc05.rdu-cc.fedoraproject.org (no mgmt)
- [x] vmhost-x86-02.iad2.fedoraproject.org
- [x] vmhost-x86-04.iad2.fedoraproject.org
- [x] vmhost-x86-05.iad2.fedoraproject.org
- [x] vmhost-x86-06.iad2.fedoraproject.org
- [x] vmhost-x86-07.iad2.fedoraproject.org (finished, but can't reach mgmt/gw problem)
- [x] bvmhost-x86-02.iad2.fedoraproject.org
- [x] bvmhost-x86-03.iad2.fedoraproject.org
- [x] bvmhost-x86-04.iad2.fedoraproject.org

## wed - 2022-02-16

The rest:

- [x] mass update * run - in progress - kevin
- [x] update status for outage
- [x] silence nagios
- [ ]
- [ ] buildvmhost-s390x-01.s390.fedoraproject.org
I updated/rebooted all the bvmhost-a64's last week when re-installing armv7 builders
They all need mgmt password changed though (m indicates mgmt needs password changed)
- [x] bvmhost-a64-01.iad2.fedoraproject.org
- [x] bvmhost-a64-02.iad2.fedoraproject.org
- [x] bvmhost-a64-03.iad2.fedoraproject.org
- [x] bvmhost-a64-04.iad2.fedoraproject.org
- [x] bvmhost-a64-05.iad2.fedoraproject.org
- [x] bvmhost-a64-06.iad2.fedoraproject.org
- [x] bvmhost-a64-07.iad2.fedoraproject.org
- [x] bvmhost-a64-08.iad2.fedoraproject.org
- [x] bvmhost-a64-09.iad2.fedoraproject.org
- [x] bvmhost-a64-10.iad2.fedoraproject.org
- [x] bvmhost-a64-11.iad2.fedoraproject.org

These can be done out of outage if needed:
- [ ] bvmhost-a64-osbs-01.iad2.fedoraproject.org
- [ ] bvmhost-p09-01.iad2.fedoraproject.org
- [ ] bvmhost-p09-02.iad2.fedoraproject.org
- [ ] bvmhost-p09-03.iad2.fedoraproject.org
- [m] bvmhost-p09-04.iad2.fedoraproject.org - done kevin

- [x] bvmhost-x86-01.iad2.fedoraproject.org (mgmt pass changed/firmware prepped for next reboot)
- [x] bvmhost-x86-05.iad2.fedoraproject.org (mgmt pass changed/firmware prepped for next reboot)
- [x] bvmhost-x86-06.iad2.fedoraproject.org (ditto)
- [x] bvmhost-x86-07.iad2.fedoraproject.org (ditto)
- [x] bvmhost-x86-08.iad2.fedoraproject.org (ditto)
- [x] vmhost-x86-copr01.rdu-cc.fedoraproject.org - done
- [x] vmhost-x86-copr02.rdu-cc.fedoraproject.org - done
- [x] vmhost-x86-copr03.rdu-cc.fedoraproject.org
- [x] vmhost-x86-copr04.rdu-cc.fedoraproject.org
- [x] vmhost-p08-copr01.rdu-cc.fedoraproject.org (no need)
- [x] vmhost-p08-copr02.rdu-cc.fedoraproject.org (no need)
- [x] qvmhost-x86-01.iad2.fedoraproject.org
- [x] qvmhost-x86-02.iad2.fedoraproject.org - mobrien
- [x] virthost-cc-rdu02.fedoraproject.org - mobrien rebooted, couldn't find mgmt
- [x] vmhost-x86-01.iad2.fedoraproject.org - kevin in progress
- [x] vmhost-x86-03.iad2.fedoraproject.org - kevin in progress
- [ ] buildhw-x86-01 to 16
- [ ] buildhw-a64-* - mobrien all rebooted, 19+ need passwd changed

- [x] check nagios for issues and fix them
- [x] unsilence nagios
- [x] clear status

Nagios actually has a pretty good list of both these things:

## Dead hosts

- ~~~~buildhw-a64-03.iad2.fedoraproject.org (mustang) - FIXED
- buildhw-a64-05.iad2.fedoraproject.org (mustang)
- buildhw-a64-08.iad2.fedoraproject.org (mustang)
- vmhost-x86-06.stg.iad2.fedoraproject.org
- vmhost-x86-08.stg.iad2.fedoraproject.org
- worker06.ocp.iad2.fedoraproject.org

## hosts where gw needs resetting
- ~~~~bvmhost-x86-01.stg.iad2.fedoraproject.org - FIXED
- vmhost-x86-07.stg.iad2.fedoraproject.org
- ~~~~vmhost-x86-10.stg.iad2.fedoraproject.org - FIXED
- vmhost-x86-04.iad2.fedoraproject.org
- vmhost-x86-07.iad2.fedoraproject.org
- ~~~~vmhost-x86-12.stg.iad2.fedoraproject.org - FIXED
- ~~~~bvmhost-x86-02.iad2.fedoraproject.org - FIXED