tags: data centre move

Current Live updated schedule for PHX2 -> IAD2 move

Acronyms Used in Document

MVFE - Minimal Viable Fedora Environment
PHX2 - Chandler Arizona datacentre
RDU-CC - Morrisville NC datacentre community cage
IAD2 - Ashburn Equinix datacentre
AAA - Authentication, Authorization, Auditing
IPA - Identity, Policy, Audit
FAS - Fedora Account System (older AAA solution)
FreeIPA - Newer account system

June 15th Fedora to IAD

note: it looks like it will take 2 to 3 days to unrack the systems in PHX2. it will also take 1 week for the hardware to travel across the country.

Phase 1

Week 00 (2020-03-02 -> 2020-03-08)

• Hardware shipping needs to be planned out
• Rack layouts for IAD2 and RDU-CC need to be finalized
• Work with RH IT on what they need in network diagrams

Week 01 (2020-03-09 -> 2020-03-15)

• Get RHEL-8 virthost built
• Deliver network diagrams to RHIT
• Announce at downtime for communishift
• Try to find a place for communishift to temp run? - no

Week 02 (2020-03-16 -> 2020-03-22)

• 2020-03-17 Fedora 32 Beta Date

Week 03 (2020-03-23 -> 2020-03-29)

• Re-Announce downtime for communishift
• Get updated tasks and timeline set out
• Work out COVID-19 Contingencies

Week 04 (2020-03-30 -> 2020-04-05)

• Virthost03 skunkworks (kevin)
  • RHEL-8 install
  • test encryption over bridge
  • test TEAM network
• Virthost05 skunkworks (kevin)
  • uefi EL8 install instructions
  • secure boot.
• Re-Announce downtime for communishift
• Set up noggin instance in AWS to replace communishift (kevin)

Week 05 (2020-04-06 -> 2020-04-12)

• Create DNS templates for IAD2
• Create template DHCP for mgmt hosts
• Add more items to ship to IAD2
• Get network layout for RDU-CC finalized
  • public ip address count and space needed
  • mgmt network 172.23.1.??/24
  • private network for openshift backnodes 172.23.2.??/24
• Collect all mgmt mac addresses
• Collect all hardware mac addresses for debugging
• Set mgmt interfaces to DHCP before getting shipped
  • opengear
  • amperes
  • power hardware
• 2020-04-09 Fedora 32 Final Freeze
• Power needs to be ready in RDU-CC cage
• Option A/B meeting for Phase 2
  • Option A - We ship items and continue with A
  • Option B - We go with planning out move of items within PHX2 and look at the larger move to IAD2 next year

Phase 2

Week 06 (2020-04-13 -> 2020-04-19)

• Put move details into status.fedoraproject.org so people can see what is going on.
• Take down and ship communishift hardware to RDU-CC
• Take down and ship extra hardware to IAD2
• Begin takedown of communishift hardware
• Communishift 13th April - 1st May
  • Power off systems in racks
  • Work with logistics for pack and move
  • Power should be on by 17th - this is power to RDU-CC

Week 07 (2020-04-20 -> 2020-04-26)

• Hardware should arrive at RDU-CC
  • Rerack from 27th 1st may
• Hardware should arrive at IAD2
  • work with Shauns team to get systems racked/stacked in 101

Week 08 (2020-04-27 -> 2020-05-03)

• Work out temporary root password for installs
• 2020-04-28 Fedora 32 release
  • [ ] set ip address in mac
  • [ ] set up admin user
  • [ ] set ipmi and serial over lan access

Week 09 (2020-05-04 -> 2020-05-10)

• RDU Bootstrap
• Write howto on Dell mgmt setup (smooge)
• IAD2 work (see IAD2 bootstrap)
• Work with IT on any network layout issues left for RDU2 site.
  • communishift proxies with private + external interfaces
• Move final virthost-cc boxes into new RDU-CC racks
• Give internal IT new DNS servers ip address
• Give internal IT new SMTP server ip address

Week 10 (2020-05-11 -> 2020-05-17)

• IAD2 work (see IAD2 bootstrap)
• Do any items in RDU-CC that time allows

Week 11 (2020-05-18 -> 2020-05-24)

• IAD2 work (see IAD2 bootstrap)
• [ ]Checkin meeting:
  • Option A: Ship everything left in PHX2 to IAD2 on June 15th
  • Option B: Need more time and ship on June 30

Week 12 (2020-05-25 -> 2020-05-31)

• IAD2 work (see IAD2 bootstrap)
• Rack 103 must be up. Get openqa and other systems in.

Phase 4

Week 13 (2020-06-01 -> 2020-06-07)

• Final checklist and approval of IAD2 MVFE
  • Test email routing through IAD2 proxies
  • Test www proxies
  • Test builds
  • Test openvpn
  • Test rsync
  • Test route to s390x
  • Test route to bugzilla STOMP message bus
• Bring up openqa in IAD2
• Change Fedora DNS to shorter times for major change
• GO/NO-GO meeting:
  • Option A: Ship everything left in PHX2 to IAD2 on June 15th
  • Option B: Look at internal move of PHX2 equipment

Week 14 (2020-06-08 -> 2020-06-14)

• Move logical infrastructure to IAD2 MVFE
  • src, koji hubs, koji db, bodhi, downloads all move in one day outage
  • particular applications can move one at a time with their deps
  • Shutdown PHX2 koji and related dbs
  • Do final dump
  • Import data into IAD2 dbs
  • Turn DNS to point to IAD2
  • Move openvpn to IAD2
• Begin cleaning PHX2 systems as possible
  • Reinstall with base EL8?
• Power off PHX2 systems as possible

Week 15 (2020-06-15 -> 2020-06-21)

• Shutdown of PHX2 racks
• Removal of systems from PHX2 and shipment to IAD2
• Travel of equipment to IAD2
• PHX2 – go over remaining hardware to recycle

Week 16 (2020-06-22 -> 2020-06-28)

• Most likely time for hardware arrival
• Racking and stacking of IAD2 equipment
• Set up mgmt interfaces
• Do initial hardware installs to RHEL8

Week 17 (2020-06-29 -> 2020-07-05)

• Finish initial hardware installs
• Bring up additional builders

Week 18 (2020-07-06 -> 2020-07-12)

• Bring up additional services
• Move mgmt interfaces back to static

Week 19 (2020-07-13 -> 2020-07-19)

• Probably more work at data centre

Week 20 (2020-07-20 -> 2020-07-26)

• Sign off on work completed

Week 21 (2020-07-27 -> 2020-08-02)

• A miracle occurs Several in fact

Week 22 (2020-08-03 -> 2020-08-09) ???

• Mass Rebuild for Fedora 33 starts
  • All systems must be up and running
  • Production needs to be normal
• FlockToFedora 2020