IAD2 Datacenter
last updated: 2020-05-27 01:17UTC
This document serves to provide a checklist/working document of services/applications setup in the new IAD2 datacenter. As services/applications/vm's are tested by their owners or others they can be checked off here and we know they are ready for the switch over from PHX2 datacenter on the week of 2020-06-08.
issues here should have a infra ticket opened on them and/or discussed on list or irc. https://pagure.io/fedora-infrastructure/issues?tags=iad2
Please test if your application works, if anything is missing, if you can see anything that would break.
Please also note if you are missing any command line history or files in your home dir that you expect/need, or cannot access things you normally do (sudo should work if it works in phx2, etc)
I have dumped/restored databases for all services as of last week (2020-05-24) and restored the last db before bringing up the iad2 version.
Some hosts have local data (ex: pkgs02.phx2 has git repos on local disk). We will be rsyncing that data, and at cutover shutting down phx2 service, sync and bring up.
With only a few exceptions (where noted) playbooks complete on all iad2 hosts.
It will be of course impossible to fully test some apps before they are migrated. Thats fine, just do the best you can testing things so we have less to fix/do on move week.
a few things are not completed yet:
everyone should have the same access they do to machines in iad2 as they do to machines in phx2. Note that you need to put in your .ssh/config:
Host *.iad2.fedoraproject.org 10.3.*.*
HostName %h
ProxyCommand ssh -q bastion-iad01.fedoraproject.org /usr/bin/nc %h 22
proxy01.iad2 is up and running, you may want to use your local /etc/hosts file to use it, (Cf point #5 below). This will allow you to hit iad2 hosts for testing
openshift is up and running in iad2. You may need to adjust your playbooks and run them to deploy your application, as of this writing only fas and ipsilon are running.
ansible-playbook can/should be run from batcave01.phx2 for now, we will move to batcave01.iad2 soon.
In order to test you can use a local /etc/hosts file to point your local brownser to iad2 instead of phx2:
38.145.60.20 fedoraproject.org getfedora.org admin.fedoraproject.org apps.fedoraproject.org mirrors.fedoraproject.org os.fedoraproject.org console.app.os.fedoraproject.org app.os.fedoraproject.org id.fedoraproject.org qa.fedoraproject.org src.fedoraproject.org mdapi.fedoraproject.org greenwave.fedoraproject.org
/etc/rpkg/fedpkg.conf
and replace the hostname currently using pkgs.fedoraproject.org
by pkgs-iad.fedoraproject.org
(do remember to remove this after testing to be able to access phx2/normal hosts again)
Currently iad2 hosts are all using bastion01.iad2 as their vpn hub. This means they can talk to each other, but not any phx2 host. This is to allow for us to test things more easily. a few days before the move we are going to add all the iad2 hosts as 'hostname-iad2' with a 192.168.10.x ip into the vpn.fedoraproject.org dns, then connect them all to bastion01.phx2. This will allow us to switch services over by changing haproxy/apache/dns to point to the hostname-iad2 host instead of hostname. Once everything is moved at the end of the week, we will then repoint everything to bastion01.iad2 and switch names back.
Please note that due to limited cpu/vm/disk/other capacity, some services will just be down. They will come back after the machines move and are re-setup in iad2, most likely in late june / early july. There is no need to test or note that any of these are down/not here/not working
Below are a list of applications that we need assistance validating, so if you have knowledge on running any of the below apps, please feel free to test them. If you find they are working properly, please check it off. If you find any of the below apps dont work correctly, please open a ticket and add the tag 'iad2' so we can investigate it.
openshift apps:
baremetal machines:
virtual machines:
services:
packaging pipeline
openshift - passed all its tests fine.
mirrormanager
registry
fedmsg/fedora-messaging
pdc
value
koji
amazing service XYZ and its test plan