Try   HackMD

Current Live updated schedule for PHX2 -> IAD2 move

tags: IAD2 Datacenter

Acronyms Used in Document

MVFE - Minimal Viable Fedora Environment PHX2 - Chandler Arizona datacentre RDU-CC - Morrisville NC datacentre community cage IAD2 - Ashburn Equinix datacentre AAA - Authentication, Authorization, Auditing IPA - Identity, Policy, Audit FAS - Fedora Account System (older AAA solution) FreeIPA - Newer account system

Shipment Equipment

The following lists cover the hardware which is being shipped at specific dates and where the hardware is going.

June 15th Fedora to IAD

note: it looks like it will take 2 to 3 days to unrack the systems in PHX2. it will also take 1 week for the hardware to travel across the country.

Serial No Model Current Hostname Current Rack New Rack New Hostname
??? Ampere ??? Rack 150 101 ???
??? Ampere ??? Rack 147 101 ???
??? Dell R630 fed-cloud13 Rack 157 101 ???
??? Dell R430 sign02 Rack 148 01 ???
??? Dell R430 bkernel04 Rack 150 101 ???
??? Mustang 3PBC-A0000Z Rack 146 101 ???
??? Mustang 3PBD-A0004P Rack 146 101 ???
??? IBM Power 9 ??? Rack 147 101 ???
??? IBM Power 8 ??? Rack 147 101 ???
??? Dell R630 virthost02.stg Rack 153 102 ???
??? Dell R630 virthost01.stg Rack 153 102 ???
??? Dell R640 virthost05 Rack 154 102 ???
??? Dell R640 virthost04 Rack 154 102 ???
??? Dell R630 virthost06 Rack 153 102 ???
??? Dell R630 virthost02 Rack 153 102 ???
??? Dell R430 sign01 Rack 148 102 ???
??? Dell R630 backup01 Rack 155 102 ???
??? Dell R430 sign01 Rack 148 102 ???
??? Dell R730 virthost01 Rack 153 102 ???
??? Dell R630 virthost12 Rack 155 102 ???
??? Dell R630 virthost22 Rack 154 102 ???
??? Dell R630 fed-cloud15 Rack 157 102 ???
??? Dell R630 bvirthost01 Rack 148 102 ???
??? Dell R630 bvirthost04 Rack 149 102 ???
??? Dell R630 bvirhost05 Rack 149 102 ???
??? Dell R630 bvirthost13 Rack 149 102 ???
??? Dell R630 bvirthost14 Rack 149 102 ???
??? Dell R630 bvirthost15 Rack 149 102 ???
??? Dell R430 bkernel03 Rack 150 102 ???
??? Dell R630 virthost14 Rack 154 102 ???
??? Dell R630 virthost19 Rack 154 102 ???
??? Dell R630 virthost21 Rack 154 102 ???
??? Dell R630 autosign Rack 149 102 ???
??? Dell FX2 Builders FX Rack 150 102 ???
??? IBM Power 9 ??? Rack 152 102 ???
??? Ibm Power 8 ??? Rack 147 102 ???
??? Ibm Power 9 ??? Rack 147 103 ???
??? Ampere ??? Rack 147 103 ???
??? Ampere ??? Rack 157 103 ???
??? Ampere ??? Rack 149 103 ???
??? Cavium ??? Rack 146 103 ???
??? Ampere ??? Rack 147 103 ???
??? Ampere ??? Rack 147 103 ???
?RT Ampere ?RT Rack 147 103 ???
?LT Ampere ?LT Rack 149 103 ???
?RX Ampere ?RX Rack 149 103 ???
?NG Ampere ?NG Rack 150 105 ???
?LV Ampere ?LV Rack 150 105 ???
?NT ?NT Rack 150 105 ???
?RH Ampere ?RH Rack 150 104 ???
??? Dell R630 virthost04.stg Rack 154 104 ???
??? Dell R630 virthost03.stg Rack 153 104 ???
?V2 Ampere ?V2 Rack 149 104 ???
??? Dell R630 vhost-comm01 Rack 152 104 ???
??? Dell R640 qa01 Rack 151 104 ???
?MN Ampere ?MN Rack 147 104 ???
?P4 Ampere ?P4 Rack 147 104 ???
?R6 Ampere ?R6 Rack 147 104 ???
??? Ampere ??? Rack 150 10104 ???
??? Dell FX2 Builders FX Rack 148 104 ???
??? IBM Power 9 ??? Rack 152 104 ???
??? IBM Power 8 ??? Rack 147 104 ???
??? OpenGear Serial Rack 146 105 ???
??? Mustang 3PBD-A00053 Rack 146 105 ???
??? Mustang 3PBD-A0009P Rack 146 105 ???
??? Mustang 3PBD-A0009B Rack 146 105 ???
??? Mustang 3PBD-A0002B Rack 146 105 ???
??? Mustang 3PBD-A0008J Rack 146 105 ???
??? Mustang 3PBD-A0006W Rack 146 105 ???
??? Mustang 3PBD-A0004G Rack 146 105 ???
??? Mustang 3PBD-A0007T Rack 146 105 ???
??? IBM Power 8 ??? Rack 147 105 ???
??? IBM Power 8 ??? Rack 152 105 ???

Phase 1

Week 00 (2020-03-02 -> 2020-03-08)

  • Hardware shipping needs to be planned out
  • Rack layouts for IAD2 and RDU-CC need to be finalized
  • Work with RH IT on what they need in network diagrams

Week 01 (2020-03-09 -> 2020-03-15)

  • Get RHEL-8 virthost built
  • Deliver network diagrams to RHIT
  • Announce at downtime for communishift
  • Try to find a place for communishift to temp run? - no

Week 02 (2020-03-16 -> 2020-03-22)

  • 2020-03-17 Fedora 32 Beta Date

Week 03 (2020-03-23 -> 2020-03-29)

  • Re-Announce downtime for communishift
  • Get updated tasks and timeline set out
  • Work out COVID-19 Contingencies

Week 04 (2020-03-30 -> 2020-04-05)

  • Virthost03 skunkworks (kevin)
    • RHEL-8 install
    • test encryption over bridge
    • test TEAM network
  • Virthost05 skunkworks (kevin)
    • uefi EL8 install instructions
    • secure boot.
  • Re-Announce downtime for communishift
  • Set up noggin instance in AWS to replace communishift (kevin)

Week 05 (2020-04-06 -> 2020-04-12)

  • Create DNS templates for IAD2
  • Create template DHCP for mgmt hosts
  • Add more items to ship to IAD2
  • Get network layout for RDU-CC finalized
    • public ip address count and space needed
    • mgmt network 172.23.1.??/24
    • private network for openshift backnodes 172.23.2.??/24
  • Collect all mgmt mac addresses
  • Collect all hardware mac addresses for debugging
  • Set mgmt interfaces to DHCP before getting shipped
    • opengear
    • amperes
    • power hardware
  • 2020-04-09 Fedora 32 Final Freeze
  • Power needs to be ready in RDU-CC cage
  • Option A/B meeting for Phase 2
    • Option A - We ship items and continue with A
    • Option B - We go with planning out move of items within PHX2 and look at the larger move to IAD2 next year

Phase 2

Week 06 (2020-04-13 -> 2020-04-19)

  • Put move details into status.fedoraproject.org so people can see what is going on.
  • Take down and ship communishift hardware to RDU-CC
  • Take down and ship extra hardware to IAD2
  • Begin takedown of communishift hardware
  • Communishift 13th April - 1st May
    • Power off systems in racks
    • Work with logistics for pack and move
    • Power should be on by 17th - this is power to RDU-CC

Week 07 (2020-04-20 -> 2020-04-26)

  • Hardware should arrive at RDU-CC
    • Rerack from 27th 1st may
  • Hardware should arrive at IAD2
    • work with Shauns team to get systems racked/stacked in 101

Week 08 (2020-04-27 -> 2020-05-03)

  • Work out temporary root password for installs
  • 2020-04-28 Fedora 32 release
    • [ ] set ip address in mac
    • [ ] set up admin user
    • [ ] set ipmi and serial over lan access

Week 09 (2020-05-04 -> 2020-05-10)

  • Write howto on Dell mgmt setup (smooge)
  • IAD2 work (see IAD2 bootstrap) [x] find all expected hardware [x] Install new hardware

Week 10 (2020-05-11 -> 2020-05-17)

  • IAD2 work (see IAD2 bootstrap)

Week 11 (2020-05-18 -> 2020-05-24)

  • IAD2 work (see IAD2 bootstrap)
  • RDU Bootstrap
  • Do any items in RDU-CC that time allows
  • Work with IT on any network layout issues left for RDU2 site.
    • communishift proxies with private + external interfaces
  • Give internal IT new DNS servers ip address
  • Give internal IT new SMTP server ip address
  • Move final virthost-cc boxes into new RDU-CC racks

Week 12 (2020-05-25 -> 2020-05-31)

  • IAD2 work (see IAD2 bootstrap)
  • Rack 103 must be up. Get openqa and other systems in.

Phase 4

Week 13 (2020-06-01 -> 2020-06-07)

  • Final checklist and approval of IAD2 MVFE
    • Test email routing through IAD2 proxies
    • Test www proxies
    • Test builds
    • Test openvpn
    • Test rsync
    • Test route to s390x
    • Test route to bugzilla STOMP message bus
  • make sure nagios says everything is green
  • Bring up openqa in IAD2
  • Change Fedora DNS to shorter times for major change
  • GO/NO-GO meeting:
    • Option A: Ship everything left in PHX2 to IAD2 on June 15th
    • Option B: Look at internal move of PHX2 equipment

Fedora Datacenter Move week (and week before) - detailed tasks

Notes:

  • db switcharoo means: take service down in phx2, dump db in phx2, copy dump to iad2, load dump in iad2, bring up service in iad2, test, switch dns.
  • snapmirror switcharoo means: take service down in phx2, umount all users of volume, sync final data via snapmirror, break snapmirror and make rw in iad2.

oustanding questions:

  • Did we miss any applications/services?
  • When do we want to move openqa? It's pretty self contained.

2020-06-01 - monday

Testing day Fedora Datacenter Test Plan

2020-06-02 - tuesday

Penaultimate Testing day Fedora Datacenter Test Plan

2020-06-03 - wed

Last testing day

  • 16:00 UTC: move dl.fedoraproject.org over to iad2
    • change dns
    • disable rsync/httpd on phx2 download servers
    • destroy all phx2 download vm's.
    • send note to mirror-admins list in case someone finds problems
  • Test everything we can with a local /etc/hosts file and vpn pointing to iad bastions.
  • 21:00 UTC: Add all iad2 vpn hosts to vpn.fedoraproject.org dns with a -iad2 to the name and in the 192.168.20.x net.

2020-06-04 - thursday

Batcave and wiki day

( testing after this point will be difficult as vm's are connected to the phx2 vpn)

  • 15:00 UTC(start) switch to using batcave01.iad2

    • shutdown access to batcave01.phx2
    • final sync of all data
      • /home
      • /git
      • /srv/* –exclude pub
      • /root
    • snapmirror switcharoo: fedora_app WARNING: builds need this for rhel things, no epel builds while it's changing, wiki needs this for attachements
    • remount old volume in phx2 on kojipkgs*.phx2 to keep epel builds working
    • test ansible-playbook runs from batcave01.iad2
    • change dns for infrastructure.fedoraproject.org to batcave01.iad2
    • power off batcave01.phx2 vm

  • have all iad2 vpn hosts reconnect to bastion01.phx2 so they can take traffic as we switch applications over to them.

  • [/] change the "IAD2" view of the fedoraproject.org zone to use PHX2/external things for rabbitmq/etc. ie, where we have if IAD2, change to match PHX2 if This allows iad2 services to connect to phx2 ones until we move those.

  • set all dns zones to 5m ttl

  • wiki initial attempt - rolled back due to auth issue

2020-06-05 - friday

Zodbot / meeting day

  • wiki (needs fedora_app for attachments)
    • take wiki*phx2 vms down
    • db switcharoo: db03/fpo-mediawiki (note this is mariadb)
    • upgrade run (wiki01.iad2 is newer than wiki01.phx2)
    • test via local curl and the like
    • change haproxy to use wiki01-iad2 (so in proxies -> bastion01.phx2 -> wiki01.iad2)
  • grobbisplitter - move to batcave01.iad2

2020-06-06 - saturday

  • migrate value01 / zodbot
    • pick time with no meetings planned
    • stop zodbot/httpd on value01.phx2
    • rsync data from value01.phx2 to value01.iad2
    • bring up zodbot / fedmsg-irc from value01.iad2
    • change valu01 in proxies to value01-iad2

2020-06-07 - sunday

  • disable rawhide composes after sundays completes
  • confirm no new packages will be added, no unretirements processed. (mail sent to all processors)
  • disable backups on backup01.phx2

Week 14 (2020-06-08 -> 2020-06-14)

  • Move logical infrastructure to IAD2 MVFE

2020-06-08 - monday

Fedora messaging, PDC, mirrormanager and authentication day

  • 15:00 UTC: *.stg.fedoraproject.org shut down and commented in ansible inventory.
  • 15:00 UTC: *.stg.fedoraproject.org virthosts reconfigured for iad2 and powered off.
  • fedora-messaging/fedmsg buses
    • make sure iad2 cluster has all vhosts and queues needed
    • shutdown phx2 cluster
    • change rabbitmq.fedoraproject.org in dns to point to iad2 cluster
    • make sure openshift iad2 messaging-bridges are working
    • github2fedmsg
    • bugzilla2fedmsg
    • x ] take down vm's, remove from ansible and clean from nagios
  • migrate notifs-backend01/notifs-web01 vms
    • shut down vms
    • copy storage and libvirt xml over
    • db switcharoo: db01/notifs
    • bring up and reconfigure for new network
  • migrate pdc instances !WARNING: commits will fail while pdc is down, do this fast
    • stop services in phx2
    • dump database / restore database
    • bring up services in iad2 and test
    • switch haproxy to point to pdc-web01-iad2
  • mirrormanager
    • shut down service in phx2
    • db switcharoo: db01/mirrormanager2
    • start things in iad2
    • test
    • point service in haproxy to iad2
    • stop vm's and remove from inventory, update nagios
  • authentication stack
    • stop services in phx2: fas, ipsilon, ipa
    • bring up fas in iad2 openshift
    • bring up ipsilon in iad2 openshift
    • bring up ipa vm's in iad2
    • db switcharoo db-fas01/*
    • switch httpd proxy config to point to -iad2 versions
    • disable ipa replication to/from ipa01.phx2 and ipa02.phx2 (can be done at any time, just power off phx2 IPA boxes for now)

Due to some entanglements, we needed to move our entire openshift cluster today. Some apps are still operating ok in phx2, but aren't reachable via the web anymore.

  • bodhi database
    • dump bodhi2 db on db01.phx2 time:
    • transfer db dump to db01.iad2 time:
    • load dump into db01.iad2 time:
  • bodhi db dump/restore done bring up bodhi for testing.
    • bodhi openshift pods bring up
    • coreos* openshift pods bring up
    • fedora-ostree-pruner
    • compose-tracker
  • openshift apps
    • asknot
    • distgit-bugzilla-sync
    • greenwave
    • koschei (scheduling off)
    • mdapi
      • switcharoo fedora_prod_mdapi
    • message-tagging-service
    • monitor-gating
    • release-monitoring
    • the-new-hotness
    • waiverdb
  • power off misc hardware that is not being shipped.
    • kernel02
    • others?

2020-06-09 - tuesday

Buildsystem and it's friends day:

    • 00:01 UTC: disable bodhi push after it runs
    • 15:00 UTC: disable all builders and show koji offline message.
    • 15:00 UTC: cancel all builds still in progress
    • 15:00 UTC: disable all builders.
    • 15:00 UTC: build system goes down. (src/koji/kojipkgs/bodhi/master mirrors/registry/builders)
    • 15:00 UTC: begin netapp transition for:
      • fedora_koji
      • fedora_koji_archive*
      • fedora_odcs
      • fedora_ostree_content
      • fedora_ftp
      • fedora_sourcecache
      • oci-registry
    • db-koji01 database
      • start pg_dumpall on db-koji01.phx2 time: 2020-06-09 15:30
      • transfer to db-koji01.iad2 time: 2020-06-09 22:45
      • load pg_dump into db-koji01.iad2 time:
    • db-koji01.iad2 is loaded and fedora_koji is ready, bring up koji for testing.
      • run 1.21.0 koji migrations on db
      • run sql modify script to clean things up
      • add all new iad2 builders hosts to the db
      • adjust hub config for channels and adjust builders for channels.
      • set hub to LockOut = true and test with some admin commands
      • unset lockout
      • test via /etc/hosts and iad2 proxies.
    • src.fedoraproject.org
        • rsync of pkgs02.phx2 git data to pkgs01.iad2 time:
        • fedora_sourcecache is ready and pkgs sync is done bring up pagure on pkgs for testing.
        • test via /etc/hosts
        • src.fp.o dns (for https/main service)
    • signing
      • autosign
      • sigul rpm ready from patrick
      • sign-bridge01.iad2 playbook run and config, bridge running
      • sign-vault01.iad2
          • config adjusted for new sigul
          • playbook run
          • Old vault data loaded and unlocked.
    • odcs
        • odcs db dump/restore done
        • fedora_ftp done koji done,
        • bring up odcs for testing.
    • osbs
    • resultsdb
        • resultsdb database
          • transfer db dump to db01.iad2 time:
          • load dump into db01.iad2 time:
          • reconfigure to use db01 and bring up service.
  • registry
    • oci_registry cut over: mount as needed, switch dns to iad2.
  • mbs
    • mbs db dump/restore done
    • bring up mbs for testing.
    • repoint s390x builders to new koji hub.
    • upgrade s390x builders to f32
  • fedora_koji_archive* cut over: mount needed hosts in iad2.
  • fedora_ftp cut over: mount ro on downloads and rw on composers/bodhi

validation:

    • test koji scratch build
    • test koji real build
    • test koji container build
    • test koji module build
    • test signing
    • test bodhi updates push
    • switch proxy httpd balancer to point to koji to iad2
    • switch proxy httpd balancer to point to dns for pkgs/src to iad2
    • switch proxy httpd balancer to point to bodhi to iad2
    • fire off a rawhide compose on compose-rawhide01.iad2 (keep cron off)

2020-06-10 - wed

Openshift apps, mailman/lists and datagrepper/datanommer day

  • 16:00 UTC backups

    • switcharoo fedora_backups volume
    • switcharoo openshift_prod_mdapi

  • mailman/lists

    • take down mailman01.phx2
    • Migrate it and all data to iad2
    • move pointer to iad2.

  • datagrepper / datanommer

    • take down service in phx2.
    • switch busgateway over to iad2
    • dump db and reload in iad2
    • switch dns to iad2 service.

2020-06-11 - thursday

Website builders, blockerbugs, and elections day:

  • openqa

    • switcharoo volume
    • db switcharoo
    • [/] rsync assets from old server
    • switch in dns

  • docsbuilding

    • switch fedora_prod_docs

  • websites building

    • switch fedora_prod_websites

  • reviewstats

    • switch fedora_prod_reviewstats

  • fedimg

  • blockerbugs

    • dump/restore blockerbugs db
    • bring up blockerbugs for testing in iad2.
    • switch blockerbugs dns to iad2.

  • kerneltest vm/app

    • dump/restore db
    • bring up in iad2 and test
    • switch in vpn/dns

  • if all looks well, switch all vpn hosts to point to bastion01.iad2

  • [/] shutdown bastion hosts in phx2.

2020-06-12 - friday

Stomp out fires day:

  • fix bugs / issues as found
  • process new packages
  • allow retirements
  • make sure everything in phx2 is off and ready to ship next week
  • test and re-enable backups on backup01.iad2.

Week 15 (2020-06-15 -> 2020-06-21)

  • Shutdown of PHX2 racks
  • Removal of systems from PHX2 and shipment to IAD2
  • Travel of equipment to IAD2
  • PHX2 – go over remaining hardware to recycle

Week 16 (2020-06-22 -> 2020-06-28)

  • Most likely time for hardware arrival
  • Racking and stacking of IAD2 equipment
  • Set up mgmt interfaces
  • Do initial hardware installs to RHEL8

Week 17 (2020-06-29 -> 2020-07-05)

  • Finish initial hardware installs
  • Bring up additional builders

Week 18 (2020-07-06 -> 2020-07-12)

  • Bring up additional services
  • Move mgmt interfaces back to static

Week 19 (2020-07-13 -> 2020-07-19)

  • Probably more work at data centre

Week 20 (2020-07-20 -> 2020-07-26)

  • Sign off on work completed

Week 21 (2020-07-27 -> 2020-08-02)

  • A miracle occurs Several in fact

Week 22 (2020-08-03 -> 2020-08-09) ???

  • Mass Rebuild for Fedora 33 starts
    • All systems must be up and running
    • Production needs to be normal
  • FlockToFedora 2020 (now virtual)
  • profit

HISTORICAL SECTION BELOW HERE

Order of Bootstrapping in RDU-CC (Deferred Until After IAD2)

  1. Get power setup for racks
  2. Get items shipped from PHX2
    1. Racks
    2. Systems
  3. Get systems installed
    1. Racks
    2. Systems
    3. Inventory systems and confirm
    4. Write up wire spreadsheet for networks
  4. SPIKE: Get switches wired into master router.
    1. ex3400 to mgmt vlan
    2. ex4300 ports 1-24 to production vlan
    3. ex4300 ports 25-36 to mgmt vlan
    4. ex4300 ports 37-48 to storage vlan
    5. ex5??? 10 gig ports to prod vlan
  5. Prepare bootstrap
    1. make RHEL-8.2 usb stick
    2. make CentOS-8.1 usb stick
    3. get mask and gloves for datacenter visit
  6. SPIKE: Build out bastion server (old vhost-s390)
    1. wire idrac to mgmt vlan
    2. configure idrac with ip address
    3. wire eth0 to prod network
    4. wire eth1 to mgmt vlan
    5. install RHEL-8.1 onto hardware
    6. build a openvpn network
  7. SPIKE: Mgmt interfaces
    1. wire mgmt to top switch.
    2. power on hardware
    3. go into bios and configure the ip address
    4. test that mgmt is reachable from bastion-rdu-cc
  8. SPIKE: Bring up vger
    1. wire hardware to front switch
    2. see if hardware works.
    3. give mgmt login to
    4. call in hardware repairs
  9. SPIKE: Bring up retrace
    1. wire hardware into top and back switch
    2. see if hardware works
    3. log in via physical console and configure ip address
    4. test and fix ansible
  10. Bring up storinator01 up
    1. log into console
    2. change ip addresses to proper ones
    3. test storage and data
  11. Bring up vmhost-rdu-cc-05 up
    1. Reinstall hardware with RHEL-8.1
    2. Start deploying guests as needed
  12. Move over vmhost-rdu-cc-01 -> vmhost-rdu-cc-04 from other racks to A06
    1. Connect mgmt to mgmt network
    2. Give mgmt ip address via BIOS
    3. Connect eth0 to external network
    4. Connect eth1 to internal network
    5. Configure host to have br1 ip address
  13. Build noc03 for rdu03
    1. Configure out a dhcp on eth1
    2. Configure out a tftp
    3. Mirror rhel8 and openshift bits
  14. OpenShift 4.x Install
    1. Bring up proxy front ends
    2. Bring up etcd systems
    3. Begin install of dell fx systems
    4. Test initial loads
  15. Additional Buildout changes?
    1. Add here as found.

Order of Bootstrapping in IAD2

  1. IP space: we need to know what networks we have internally and externally.
    • Internal: 10.3.160->10.3.176
    • External: 38.145.60.0/24
  2. Setup DNS space for the zones.
    • Internal reverse zones.
    • Internal forward zones.
    • External reverse zones.
    • External forward zones.
  3. Map internal network port allowances
    • Prod to Build/Build to Prod
    • Prod to QA/QA to Prod
  4. Map external network ports to internal
    • External to Prod / Prod to External
    • External to QA / QA to External
  5. Back ground network setup
    • Networking sets up vlans and wiring in top racks.
    • IAD2 firewall rules need to be setup for bastion host.
      • ssh
      • https
      • chrony
      • unbound/DNS
      • openvpn
    • IAD2 firewall rules need to be setup for general outbound access.
      • https
      • unbound/DNS
      • fedmsg
      • other outbound items?
    • Set mgmt router to have dhcp using 10.3.160.* space
  6. Get access to PDU's in racks {{Deferred}}
    • get ip addresses for rack 101 103
    • get account and password
    • test login
    • power on systems
  7. Install initial hardware
    • power on first virthost
    • determine its dhcp address
    • log into system idrac and change base password
    • create admin account and give additional rights to it.
    • give permanent ip address of 10.3.160.10
    • test that system goes to new ip address and works.
    • install rhel-8
    • give host secondary ip address to box of bastion01.iad2.
    • test external login abilities to this host
    • test routing via this host from phx2 facility.
    • set up any other temporary services on this host
  8. Install additional hardware
    • power on next host
    • determine its dhcp address
    • log into system idrac and change base password
    • create admin account and give additional rights to it.
    • give permanent ip address for host in dns
    • test that system goes to new ip address and works.
    • install rhel-8
  9. Install guests
    • DHCP/TFTP from all Build/QA/etc networks routes to 10.3.163.10

IAD2 Build list:

  1. Remaining virtual servers
    1. centos01
    2. centos02
    3. vmhost-x86-01
    4. vmhost-x86-02
    5. vmhost-x86-03
    6. vmhost-x86-04
    7. vmhost-x86-05
    8. vmhost-x86-06
    9. vmhost-x86-07
    10. bvmhost-x86-01
    11. bvmhost-x86-02
    12. bvmhost-x86-03
    13. bvmhost-x86-04
    14. bvmhost-x86-05
    15. bvmhost-x86-06
    16. bvmhost-a64-01 (ampere01)
    17. bvmhost-a64-02 (ampere02)
    18. bvmhost-x86-07
    19. autosign01 (fed-cloud12)
    20. sign-box (sign06)
    21. bkernel (bkernel05?)
    22. mustang01
    23. mustang02 (needs serial)
    24. power08-01 [hardware broken]
    25. power09-01
    26. power08-02 [hardware broken]
    27. power09-02
    28. qvmhost-x86-01 [hardware broken]
    29. qvmhost-x86-02 (qa r640)
    30. qa-x86-01 (qa r640)
    31. qa-a64-01 (ampere)
    32. bvmhost-a64-03 (ampere)
    33. bvmhost-a64-04 (ampere)
  2. Critical infrastructure services (MVF list)
    • bastion2
    • config-mgmt (batcave)
    • rebuild bastion1 as proper virt-guest
    • dns
    • noc/dhcp
    • log-server
    • tang
    • sign-vault
    • sign-bridge
    • autosign
    • certgetter
    • ipa cluster (rhel8 from rhel7)
    • loopabull
    • mirrormanager vm's
    • noc01
  3. SPIKE: database servers
    • db01 postgresql 12 / rhel8
    • db-koji01 postgresql 12 / rhel8
    • db-fas01 postgresql 12 / rhel8
    • db02 (mariadb for wiki)
  4. SPIKE: rabbitmq setup
  5. SPIKE: bring up openshift cluster
  6. SPIKE: bring up koji and build infra as a temp staging environment to test the list of MVF builds to test to make sure that stuff works. [x] koji hubs [ ] koji builders [x] kojipkgs [x] bodhi-backend [ ] grobbisplittr [x] mbs [x] rawhide/branched composers [x] compose-x86-01 [x] compose-iot [ ] osbs [x] odcs [x] registry [x] pkgs [x] downloads [x] pdc
    1. set partition on the netapp
    2. mount them on the box
    3. run the services here for testing
  7. SPIKE: bring up additional non-build service
    1. proxies
    2. mailing lists
    3. backups
    4. download servers
    5. sundries
    6. value
    7. wiki
    8. bugzilla2fedmsg
    9. datagrepper
    10. datanommer-db
    11. FMN (might just sync this over and adjust it)
  8. SPIKE: openqa setup and testing
  9. Evaluate the MVF with community member testing by sending to the lists with a feedback loop & closeout time

vm / application install / initial configuration checklist

non build: week of 2020-06-01

build: week of 2020-06-08

qa vms:

Old data

April 13 Cloud to RDU-CC

  • Hardware was deracked and removed from PHX2 data centre 2020-04-14
  • Hardware arrived at data-centre 2020-04-20
  • Hardware was rack/stacked 2020-??-??
  • Hardware was reinstalled 2020-??-??
Serial No Model Mac Address Current Hostname New Hostname
??? juniper ???
??? juniper ???
??? OpenGear ??? rack 156 open opengear02.rdu-cc.
??? Mustang ??? ??? ???
??? Mustang ??? ??? ???
??? Power8 ??? ??? ???
??? Dell R6550 ??? copr-vmhost01 copr-vmhost01
??? Dell R6550 ??? copr-vmhost02 copr-vmhost02
??? Dell R6550 ??? copr-vmhost03 copr-vmhost03
??? Dell R6550 ??? copr-vmhost04 copr-vmhost04
??? Len Ampr ??? cloud-a64 cloud-a64
??? Dell R7550 ??? retrace03 ???
??? Dell R630 ??? ??? cloudvmhost-x86_64-01
??? Dell R630 ??? ??? cloudvmhost-x86_64-02
??? Dell FX ??? lots-o-stuff lots-o-stuff
??? juniper ???
??? juniper ???
??? 10g switch ???
??? Power8 ??? ??? ???
??? Cavium 1 ??? ??? ???
??? Cavium 2 ??? ??? ???
??? Dell FX ??? lots-o-stuff lots-o-stuff
??? Storinator ??? storinator01 storinator01

April 13 MVF to IAD

  • Hardware was deracked and removed from PHX2 data centre 2020-04-14
  • Hardware arrived at data-centre 2020-04-20
  • Hardware was rack/stacked 2020-??-??
  • Hardware was reinstalled 2020-??-??
Serial No Model Mac Address Current Hostname New Hostname
??? OpenGear ??? rack 146 open opengear01.rdu-cc.
??? Dell R630 B8:2A:72:FC:ED:22 fed-cloud12 vmh-x64-09.iad
??? Dell R630 B8:2A:72:FC:F2:2E fed-cloud13 vmh-x64-10.iad
??? Len Ampr E8:6A:64:39:18:99 vhm-a-22 vmh-a64-01.iad
??? Len Ampr E8:6A:64:39:18:85 vhm-a-21 vmh-a64-02.iad
??? Dell R430 ??? sign02 sign02
??? Dell R430 ??? bkernel04 bkernel04
??? Mustang ??? ??? ???
??? Mustang ??? ??? ???
??? Power9 ??? ??? ???
??? Power8 ??? ??? ???
??? Len Ampr E8:6A:64:97:6B:49 oqa-a-02 oqa-a-01.iad
??? Dell R640 ??? ??? vhm-qa-01.qa
??? Dell R640 ??? ??? vhm-qa-02.qa
??? Dell R640 ??? ??? oqa-x86-01.qa
??? Dell R630 ??? fed-cloud-15 oqa-x86-02.qa
??? Power 9 ??? ??? oqa-p64
??? Dell R630 ??? virthost05 bvhm-02
Last changed by 
 
Fedora_Infrastructure
A team which covers items with fedora
0
6907

Read more

Updates for week of 2025-03-24

updates for week of 2025-03-24

Mar 27, 2025
IAD2 -> RDU3 overview

Overview of the IAD2->RDU3 datacenter move

Mar 25, 2025
updates for week of 2025-01-27

updates for week of 2024-11-18

Mar 20, 2025
riscv secondary koji planning

== overview

Feb 28, 2025
Read more from Fedora_Infrastructure
Published on HackMD