# CentOS-8-Stream upstream migration
###### tags: `Design`

## Aim
* Ensure migrating to centos-8-stream is seemless to tripleo users

## Steps to success :)

* **master promotions will be turned off** to provide time and space for folks to validate centos-8-stream works in the following steps
#### Stream Content on CentOS-8 nodes ( test project ) https://review.rdoproject.org/r/#/c/32041/
* update promotion-testing-hash-master.yml to use centos-8-stream
* update container builds and overcloud-images to use centos-8-stream via tripleo-repos
* run the master integration line w/ centos-8 content
* if SUCCESS move to next step
* if FAIL debug / retry

#### Stream Content on Stream nodes ( test project ) https://review.rdoproject.org/r/#/c/32080/
* update nodepool nodes
* rerun w/ centos-8-stream + centos-8-stream nodes
* if SUCCESS move to next step
* if FAIL debug / retry

#### Ensure content-providers for centos-8-stream ( upstream DNM patch )
* Ensure centos-8-stream content provider for master works

#### CentOS-8-Stream promotion for master
* Ensure the required jobs w/ centos-8-stream have passed promotion requirements.

#### Document revert path

#### Target for Wednesday - March 3
* Turn off promoter - on all centos 8 (t, u, v, m)
* ~~Run another (conatiner and image stream node build, push) plus release node, stream content jobs ahead of Sunday~~
* testprojects and reviews with tripleo-ci-testing-hash release files (and depends-on change to tripleo-ci and tripleo-repos)
* Run all branches with stream node on all jobs

## Details
* job names will NOT change.
* registry and overcloud image locations will NOT change
* users should only see that the nodepool node and repos and content are now centos-8-stream based.

## Patches

### Patches needed for integration line (RDO)
* Updates release file (promotion-testing-hash-master) to use centos-8-stream repos
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/774911
* Updates ansible task to call tripleo-repos to with --stream in case it is a stream node
* https://review.opendev.org/c/openstack/tripleo-ci/+/775429
* Adding c8-stream nodesets for tripleo-ovb-centos-8-stream-primary-vexxhost, two-centos-8-stream-nodes-vexxhost
* ~~https://review.rdoproject.org/r/#/c/32080/~~
* vexx and psi? or just vexx?
* Updates the actual integration line to stream repo on release node - https://review.rdoproject.org/r/#/c/32146/
* TODO:
* either integration line nodesets
* add nodeset to jobs ( also tricky becuase of inheritance)
* or change nodeset label (most tricky - using centos 8 release on other releases - unless they all move to stream)
* Updates release file (promotion-testing-hash-train) to use centos-8-stream repos
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/778554
* Updates release file (promotion-testing-hash-ussuri) to use centos-8-stream repos
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555
* Updates release file (promotion-testing-hash-victoria) to use centos-8-stream repos
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557

NOTE: ci.centos - what is required there???

* Testproject jobs - Duplicate of master integration line but with centos-8-stream node/repos
* Centos-8 nodes + Centos-8-Stream repo
* https://review.rdoproject.org/r/#/c/32041/
* Centos-8-Stream nodes + Centos-8-stream repo
* https://review.rdoproject.org/r/#/c/32080/
* Other patches for testing
* https://review.rdoproject.org/r/#/c/29969/26/.zuul.yaml
* https://review.rdoproject.org/r/#/c/32063/ - testing failing jobs
* Testproject jobs - Victoria integration line - centos 8 stream repos on release node
* https://review.rdoproject.org/r/#/c/32158/
* Testproject jobs - Ussuri integration line - centos 8 stream repos on release node
* https://review.rdoproject.org/r/32159
* Testproject jobs - Train integration line - centos 8 stream repos on release node
* https://review.rdoproject.org/r/#/c/32160/
### Patches needed for component lines - needed as soon as we promote (RDO)
* No container or image build jobs
* Possibly test with node definition change

### Patches neeeded for upstream check
* DNM: Testing centos-8-stream node https://review.opendev.org/c/openstack/tripleo-ci/+/771717
* DNM: Add master specific build image jobs for stream https://review.opendev.org/c/openstack/tripleo-ci/+/778189
* DNM: Update master release to c8 stream repos/image https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655

### Order of patches to be merged:
* Updates release file (promotion-testing-hash-master) to use centos-8-stream repos
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/774911
* Updates ansible task to call tripleo-repos to with --stream in case it is a stream node
* https://review.opendev.org/c/openstack/tripleo-ci/+/775429
* Updates the actual integration line to stream repo on release node
* https://review.rdoproject.org/r/#/c/32146/
* **PROMOTION**
* DNM: Update master release to c8 stream repos/image https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655
* DNM: Testing centos-8-stream node https://review.opendev.org/c/openstack/tripleo-ci/+/771717
* DNM: Add master specific build image jobs for stream https://review.opendev.org/c/openstack/tripleo-ci/+/778189
* ** NO COMPONENT CHANGES NEEDED - no build image or containers **

------------------

* rdo patches w/ -1 wf to change centos-8 image to stream
* https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L371 (single node is laready merged)
* https://review.rdoproject.org/r/#/c/32061/ - ovb and two centos nodes
* https://review.rdoproject.org/r/#/c/32146/ - updates integration line to stream repo on release node
* upstream patches w/ -1 wf on Changing nodes to centos-8-stream
* https://review.opendev.org/c/openstack/tripleo-ci/+/771717
* https://review.opendev.org/c/openstack/tripleo-ci/+/771717/5/zuul.d/nodesets.yaml
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655 (master.yaml)

## Bugs found from test project jobs

* Some repos are not 8-stream repos
* https://logserver.rdoproject.org/69/29969/29/check/periodic-tripleo-ci-build-containers-ubi-8-no-push/1c11f28/etc/yum.repos.d/ - 775429/1
* ^ These repos are laid down by tripleo-repos here - https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-repos/tasks/main.yml#L92 . But for stream OS a flag --stream has to be passed to tripleo-repos command.
* $stream is not rendered by the node - 775429/2
* https://logserver.rdoproject.org/69/29969/36/check/periodic-tripleo-centos-8-buildimage-overcloud-full-master-no-upload/72ea9e7/overcloud-full.log - nothing provides pacemaker-libs(x86-64) = 2.0.5-7.el8 needed by pacemaker-remote-2.0.5-7.el8.x86_64 - 29969/38

### Errors in integration line testprojects
#### Master
* periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master FAILURE in 44m 40s (https://bugs.launchpad.net/tripleo/+bug/1917621)
* periodic-tripleo-ci-centos-8-scenario010-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
* periodic-tripleo-ci-centos-8-scenario004-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
* periodic-tripleo-ci-centos-8-scenario001-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master (https://bugs.launchpad.net/tripleo/+bug/1917583)
* periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master (same error as https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/8fceedf/logs/undercloud/var/log/tempest/stestr_results.html.gz - logged??)

#### Victoria
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria (tempest errors - https://logserver.rdoproject.org/58/32158/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/1374917/logs/undercloud/var/log/tempest/stestr_results.html.gz)

## Check list for dday
#### Target dday for upstream master migration
* rlandy, sshnaidman and frenzy_friday (Ananya)
- [ ] **ensure** we have all integration jobs passing on c8 integration job
- [x] https://review.rdoproject.org/r/#/c/32041/ master
- [x] https://review.rdoproject.org/r/#/c/32158/ victoria
- [ ] https://review.rdoproject.org/r/#/c/32159/ ussuri
- [ ] https://review.rdoproject.org/r/#/c/32160/ train
- [x] **merge** Updates release file (promotion-testing-hash-master) to use centos-8-stream repos - https://review.opendev.org/c/openstack/tripleo-quickstart/+/774911 wf+2
- [x] **merge** Adding a check to see if os is stream os. https://review.opendev.org/c/openstack/tripleo-ci/+/775429 (can merge - no-op, WF+2)
- [x] restore and **merge** Moving master Integration line to centos 8 stream repo content on centos 8 node. https://review.rdoproject.org/r/#/c/32146/ +2 ready to merge, after depnds
- [x] **(promotion)** revert patch to disable master line and run the master integration line
- [ ] **(promotion)** turn on master promotions for centos-8-stream http://10.0.148.74/
- [ ] check https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=ae6d10b12d253f18b12eb0a88ed200ed
- [ ] check https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=c45c895ecdae89eab0b9ab240ac7eb91
- [x] FULL PASS https://trunk.rdoproject.org/api-centos8-master-uc/api/civotes_agg_detail.html?ref_hash=edd46672cb9b7a661ecf061942d71a72 THIS IS PROMOTING
- [x] **ensure** master - centos-8-stream content on release nodes promotion completes
- [x] **upstream check change - merge** Set image and container build jobs non-voting: stream https://review.opendev.org/c/openstack/tripleo-ci/+/778189
- [x] **upstream check change - merge** Update master release to c8 stream repos/image https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655
- [x] **upstream check change - merge** Define centos-8 stream nodesets for upstream jobs https://review.opendev.org/c/openstack/tripleo-ci/+/771717 (no-op change - w+'ed)')
- [x] **upstream check change - merge** Add volume_mounts for centos 8 container builds https://review.opendev.org/c/openstack/tripleo-ci/+/778782 (https://review.opendev.org/c/openstack/tripleo-ci/+/778783 now passes)
- [x] **upstream check change - merge** Add dirs to be mounted when building containers - centos https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/779036
- [x] **ensure** all patches with this topic branch are merged: https://review.opendev.org/q/topic:%22centos-8-stream-content-master%22+(status:open%20OR%20status:merged) SAGI - If all is working well... merge everything up to here:
- [x] **Communicate** change to upstream via openstack-discuss[tripleo]
- [x] **Monitor/FIX** any fall out from master content moving to centos-8-stream
- [x] **FIX** any overcloud image or container build jobs that are now on centos-8-stream
- [x] **Sync** together re: status and next steps

#### Next Steps:
- [X] **Revert** https://review.rdoproject.org/r/#/c/32161/ to re-enable stable branch lines
- [X] **Victoria release file** https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557
- [X] (same ofr ussuri, train)
- [x] **Change node label** rdo and upstream

#### OPEN Questions:
* ci.centos (no plan for stream nodes there)
* https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/894/ passing job there so far

#### Stream on stream items ( after all branches are moved)
- [x] (nodepool) merge rdo changes to convert centos8 to centos-8-stream
- [x] (nodepool) merge upstream changes to change from centos8 to centos-8-stream nodes

## Victoria ( topic : centos-8-stream-content-victoria)

- [X] Testproject for integration line - https://review.rdoproject.org/r/#/c/32158/
- [ ] Test topic for check jobs - TO DO
- [X] Releasefile for integration line - https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557
- [X] Updating the integration line - https://review.rdoproject.org/r/#/c/32224/
- [x] Run Integration
- [x] Releasefile for upstream - https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315
- [x] Promote

## Ussuri ( topic : centos-8-stream-content-ussuri)

- [x] Testproject for integration line - https://review.rdoproject.org/r/#/c/32159/
- [ ] Failures:
- [x] periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri
- [x] tempest tempest.api.object_storage.test_container_services.ContainerTest.test_list_container_contents_with_path
- [ ] skipped w/ https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/779858

- [ ] periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri
- [ ] NV waive
- [ ] periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri
- [ ] NV waive
- [ ] periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri
- [ ] NV waive
- [x] periodic-tripleo-ci-centos-8-standalone-upgrade-ussuri
- [x] tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
- [ ] skipped w/ https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/779859
- [ ] Test topic for check jobs - TO DO
- [x] Releasefile for integration line - https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555
- [x] Updating the integration line - https://review.rdoproject.org/r/#/c/32276/
- [X] Enable integration stable2
- [x] Run Integration
- [X] Releasefile for upstream - https://review.opendev.org/c/openstack/tripleo-quickstart/+/779799
- [ ] Promote
- [ ] shooting for https://trunk.rdoproject.org/api-centos8-ussuri/api/civotes_agg_detail.html?ref_hash=15ab02dfc568e1cc043f1cbbab2168a1

## Train ( topic : centos-8-stream-content-train)

- [X] Testproject for integration line - https://review.rdoproject.org/r/#/c/32160/ WF -1
- Failures: https://review.rdoproject.org/zuul/status
- periodic-tripleo-ci-centos-8-scenario002-standalone-train - Standalone deploy Timed out at Wait for containers to start for step 3 using paunch - Passed in Testproject https://review.rdoproject.org/r/#/c/32063/
- periodic-tripleo-ci-centos-8-scenario003-standalone-train - Standalone deploy Timed out - Passed later in https://review.rdoproject.org/r/#/c/32063/
- periodic-tripleo-ci-centos-8-scenario010-standalone-train - ERROR ovsdbapp.backend.ovs_idl.idlutils - Unable to open stream to tcp:192.168.24.1:6642 to retrieve schema: Connection refused - Failed again
- periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train - Did not rerun
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train - Did not rerun
- periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train - Passed in https://review.rdoproject.org/r/#/c/32063/
- [ ] Test topic for check jobs - TO DO
- [x] Releasefile for integration line - https://review.opendev.org/c/openstack/tripleo-quickstart/+/778554 WF -1
- [x] Updating the integration line - https://review.rdoproject.org/r/#/c/32284/ WF -1
- [x] Enable integration stable3 -
- [x] Run Integration - - https://review.rdoproject.org/r/#/c/32349/
- [X] Releasefile for upstream - https://review.opendev.org/c/openstack/tripleo-quickstart/+/779803 WF -1 - has errors
- [x] Promote

## Nodeset changes
- [x] Use stream labels on all centos 8 nodesets - https://review.rdoproject.org/r/c/rdo-jobs/+/32414 - (cloud specific nodes)
- [x] Remove stream nodesets from integration line - https://review.rdoproject.org/r/c/rdo-jobs/+/32483
- [x] Remove c8 stream specific nodesets definition from rdo - https://review.rdoproject.org/r/c/rdo-jobs/+/32413
- [x] DNM: Use stream labels on all centos 8 nodesets https://review.opendev.org/c/openstack/tripleo-ci/+/780654
- [x] Remove the centos-8-stream pipeline https://review.rdoproject.org/r/c/config/+/32439
- [x] Remove centos-8-stream pipeline jobs https://review.rdoproject.org/r/c/rdo-jobs/+/32440
- [x] Move tripleo_image_source to c8 stream - https://review.opendev.org/c/openstack/tripleo-ci/+/780899
- [x] Remove tripleo-image-source var from integration lines - https://review.rdoproject.org/r/c/rdo-jobs/+/32583
- [x] TODO: Remove stream-specific definitions from upstream (last thing) - https://review.opendev.org/c/openstack/tripleo-ci/+/780950