# Fedora Project IAD2 datacenter application/service validation
###### tags: `IAD2 Datacenter`

last updated: 2020-05-27 01:17UTC

This document serves to provide a checklist/working document of services/applications setup in the new IAD2 datacenter.
As services/applications/vm's are tested by their owners or others they can be checked off here and we know they are ready for the switch over from PHX2 datacenter on the week of 2020-06-08.

issues here should have a infra ticket opened on them and/or discussed on list or irc.
https://pagure.io/fedora-infrastructure/issues?tags=iad2

Please test if your application works, if anything is missing, if you can see anything that would break.

Please also note if you are missing any command line history or files in your home dir that you expect/need, or cannot access things you normally do (sudo should work if it works in phx2, etc)

## testing limitations

1. I have dumped/restored databases for all services as of last week (2020-05-24) and restored the last db before bringing up the iad2 version.

2. Some hosts have local data (ex: pkgs02.phx2 has git repos on local disk). We will be rsyncing that data, and at cutover shutting down phx2 service, sync and bring up.

3. With only a few exceptions (where noted) playbooks complete on all iad2 hosts.

4. It will be of course impossible to fully test some apps before they are migrated. Thats fine, just do the best you can testing things so we have less to fix/do on move week.

5. a few things are not completed yet:

- [ ] builders for koji - waiting for iscsi storage setup
- [ ] resultsdb - waiting to decide where it goes
- [ ] odcs - in progress
- [ ] mbs - in progress
- [ ] openqa hub and workers - waiting for qvmhosts / worker installs
- [ ] sign-vault / autosign - waiting for machine installs
- [ ] mailman - going to be migrated
- [ ] notifs - going to be migrated
- [ ] osbs - not yet deplyed (cverna)

## testing access

1. everyone should have the same access they do to machines in iad2 as they do to machines in phx2. Note that you need to put in your .ssh/config:
<code>
Host \*.iad2.fedoraproject.org 10.3.\*.\*
HostName %h
ProxyCommand ssh -q bastion-iad01.fedoraproject.org /usr/bin/nc %h 22
</code>
2. proxy01.iad2 is up and running, you may want to use your local /etc/hosts file to use it, (Cf point #5 below). This will allow you to hit iad2 hosts for testing

3. openshift is up and running in iad2. You may need to adjust your playbooks and run them to deploy your application, as of this writing only fas and ipsilon are running.

4. ansible-playbook can/should be run from batcave01.phx2 for now, we will move to batcave01.iad2 soon.

5. In order to test you can use a local /etc/hosts file to point your local brownser to iad2 instead of phx2:

38.145.60.20    fedoraproject.org getfedora.org admin.fedoraproject.org apps.fedoraproject.org mirrors.fedoraproject.org os.fedoraproject.org console.app.os.fedoraproject.org app.os.fedoraproject.org id.fedoraproject.org qa.fedoraproject.org src.fedoraproject.org mdapi.fedoraproject.org greenwave.fedoraproject.org

6. To commit in dist-git in iad, edit your ``/etc/rpkg/fedpkg.conf`` and replace the hostname currently using ``pkgs.fedoraproject.org`` by ``pkgs-iad.fedoraproject.org``

(do remember to remove this after testing to be able to access phx2/normal hosts again)

## a note on vpns

Currently iad2 hosts are all using bastion01.iad2 as their vpn hub. This means they can talk to each other, but not any phx2 host. This is to allow for us to test things more easily.
a few days before the move we are going to add all the iad2 hosts as 'hostname-iad2' with a 192.168.10.x ip into the vpn.fedoraproject.org dns, then connect them all to bastion01.phx2. This will allow us to switch services over by changing haproxy/apache/dns to point to the hostname-iad2 host instead of hostname. Once everything is moved at the end of the week, we will then repoint everything to bastion01.iad2 and switch names back.

## a note on things that aren't here

Please note that due to limited cpu/vm/disk/other capacity, some services will just be down. They will come back after the machines move and are re-setup in iad2, most likely in late june / early july. There is no need to test or note that any of these are down/not here/not working

* asknot
* Fedocal
* Badges
* Nuancier
* Anitya
* elections
* koschei
* message-tagging-service
* review-stats
* All staging services (*.stg.fedoraproject.org)
* simple-koji-ci
* While not technically impacted by the move, we will turn it off so that we lessen the load on the builders

Validation by host
=
Below are a list of applications that we need assistance validating, so if you have knowledge on running any of the below apps, please feel free to test them. If you find they are working properly, please check it off.
If you find any of the below apps dont work correctly, please open a ticket and add the tag 'iad2' so we can investigate it.

openshift apps:
- [ ] bodhi
- [ ] compose-tracker
- [x] coreos-cincinnati
- [x] Check for glaring anomalies in [metrics](https://status.updates.coreos.fedoraproject.org/metrics)
- [ ] coreos-koji-tagger
- [ ] distgit-bugzilla-sync
- [ ] docsbuilding
- [x] fas - interface up and running fine
- [ ] greenwave
- 2 of the 3 deployments worked, the last one is blocked by https://pagure.io/fedora-infrastructure/issue/8977
- [x] ipsilon
- [ ] mdapi
- Persistent storage issue: https://pagure.io/fedora-infrastructure/issue/8976
- [ ] messaging-bridges
- [ ] monitor-gating
- [ ] transtats
- [x] waiverdb
- [ ] websites

baremetal machines:
- [ ] autosign01
- [ ] sign-vault01
- [ ] aarch64 builders
- [ ] ppc64le builders
- [ ] access to s390x builders
- [ ] backup server

virtual machines:

- [x] bastion01.iad2.fedoraproject.org
- [ ] batcave01.iad2.fedoraproject.org
- [ ] blockerbugs01.iad2.fedoraproject.org
- [ ] bodhi-backend01.iad2.fedoraproject.org
- [ ] bugzilla2fedmsg01.iad2.fedoraproject.org
- [ ] busgateway01.iad2.fedoraproject.org
- [ ] certgetter01.iad2.fedoraproject.org
- [ ] compose-iot01.iad2.fedoraproject.org
- [ ] compose-rawhide01.iad2.fedoraproject.org
- [ ] compose-x86-01.iad2.fedoraproject.org
- [x] datagrepper01.iad2.fedoraproject.org
- [x] db01.iad2.fedoraproject.org | needs final sync when moving
- [x] db03.iad2.fedoraproject.org | needs final sync when moving
- [x] db-datanommer01.iad2.fedoraproject.org | needs final sync when moving
- [x] db-fas01.iad2.fedoraproject.org | needs final sync when moving
- [x] db-koji01.iad2.fedoraproject.org | needs final sync when moving
- [x] dl01.iad2.fedoraproject.org
- [x] dl02.iad2.fedoraproject.org
- [x] dl03.iad2.fedoraproject.org
- [ ] dl04.iad2.fedoraproject.org
- [ ] dl05.iad2.fedoraproject.org
- [ ] fedimg01.iad2.fedoraproject.org
- [ ] github2fedmsg01.iad2.fedoraproject.org
- [ ] ipa01.iad2.fedoraproject.org | not yet replicating
- [ ] ipa02.iad2.fedoraproject.org | not yet replicating
- [x] koji01.iad2.fedoraproject.org
- [x] koji02.iad2.fedoraproject.org
- [ ] kojipkgs01.iad2.fedoraproject.org
- [ ] kojipkgs02.iad2.fedoraproject.org
- [ ] log01.iad2.fedoraproject.org | rsyslog not working right
- [ ] loopabull01.iad2.fedoraproject.org
- [ ] mbs-backend01.iad2.fedoraproject.org
- [ ] mbs-frontend01.iad2.fedoraproject.org
- [ ] memcached01.iad2.fedoraproject.org
- [ ] mm-backend01.iad2.fedoraproject.org
- [ ] mm-crawler01.iad2.fedoraproject.org
- [ ] mm-frontend01.iad2.fedoraproject.org
- [ ] mm-frontend-checkin01.iad2.fedoraproject.org
- [ ] noc01.iad2.fedoraproject.org
- [x] ns01.iad2.fedoraproject.org
- [x] ns02.iad2.fedoraproject.org
- [ ] oci-candidate-registry01.iad2.fedoraproject.org
- [ ] oci-registry01.iad2.fedoraproject.org
- [x] os-control01.iad2.fedoraproject.org
- [x] os-master01.iad2.fedoraproject.org
- [x] os-master02.iad2.fedoraproject.org
- [x] os-master03.iad2.fedoraproject.org
- [x] os-node01.iad2.fedoraproject.org
- [x] os-node02.iad2.fedoraproject.org
- [x] os-node03.iad2.fedoraproject.org
- [x] os-node04.iad2.fedoraproject.org
- [x] os-node05.iad2.fedoraproject.org
- [ ] pdc-backend01.iad2.fedoraproject.org
- [ ] pdc-backend02.iad2.fedoraproject.org
- [ ] pdc-backend03.iad2.fedoraproject.org
- [ ] pdc-web01.iad2.fedoraproject.org
- [ ] pkgs01.iad2.fedoraproject.org
- [ ] proxy01.iad2.fedoraproject.org
- [ ] proxy101.iad2.fedoraproject.org
- [ ] proxy10.iad2.fedoraproject.org
- [ ] proxy110.iad2.fedoraproject.org
- [ ] rabbitmq01.iad2.fedoraproject.org | rabbitmq isn't clustering
- [ ] rabbitmq02.iad2.fedoraproject.org | rabbitmq isn't clustering
- [ ] rabbitmq03.iad2.fedoraproject.org | rabbitmq isn't clustering
- [ ] secondary01.iad2.fedoraproject.org
- [ ] sign-bridge01.iad2.fedoraproject.org
- [ ] sundries01.iad2.fedoraproject.org
- [x] tang01.iad2.fedoraproject.org
- [x] tang02.iad2.fedoraproject.org
- [ ] value01.iad2.fedoraproject.org
- [x] wiki01.iad2.fedoraproject.org | works, but cannot login

services:

- [ ] packaging pipeline
- [x] pkgs.fedoraproject.org clone works over ssh
- [x] src.fedoraproject.org clone works over http
- w/ SELinux off
- [ ] src.fedoraproject.org login works
- Strange state: failure
- [ ] src.fedoraproject.org web interface works
- [x] src.fedoraproject.org https push works
- Hooks crash (py3 related)
- [ ] src.fedoraproject.org forking works
- [x] pkgs.fedoraproject.org ssh push works
- [ ] gssapi auth to koji works
- [ ] krb5 keytab auth to koji works
- [ ] koji build works from src.fedoraproject.org
- [ ] bodhi web interface loads
- [ ] bodhi web new update works
- [ ] waiverdb works
- [ ] greenwave works
- [ ] ci runs
- [ ] bodhi command line new update works
- [ ] bodhi updates composes work
- [ ] updates sync from compose to ftp works
- [ ] rawhide compose works

- [x] openshift - passed all its tests fine.
- [ ] mirrormanager
- [ ] registry
- [ ] fedmsg/fedora-messaging
- [ ] pdc
- [ ] value
- [ ] meeting logs can be viewed
- [ ] mote works
- [ ] koji
- [x] web interface
- [ ] xmlrpc
- [ ] builders checkin
- [ ] packages and logs download (kojipkgs)

- [ ] amazing service XYZ and its test plan