CentOS-8-Stream upstream migration

tags: `Design`

Aim

Ensure migrating to centos-8-stream is seemless to tripleo users

Steps to success :)

master promotions will be turned off to provide time and space for folks to validate centos-8-stream works in the following steps

Stream Content on CentOS-8 nodes ( test project ) https://review.rdoproject.org/r/#/c/32041/

update promotion-testing-hash-master.yml to use centos-8-stream
update container builds and overcloud-images to use centos-8-stream via tripleo-repos
run the master integration line w/ centos-8 content
- if SUCCESS move to next step
- if FAIL debug / retry

Stream Content on Stream nodes ( test project ) https://review.rdoproject.org/r/#/c/32080/

update nodepool nodes
rerun w/ centos-8-stream + centos-8-stream nodes
- if SUCCESS move to next step
- if FAIL debug / retry

Ensure content-providers for centos-8-stream ( upstream DNM patch )

Ensure centos-8-stream content provider for master works

CentOS-8-Stream promotion for master

Ensure the required jobs w/ centos-8-stream have passed promotion requirements.

Document revert path

Target for Wednesday - March 3

Turn off promoter - on all centos 8 (t, u, v, m)
~~Run another (conatiner and image stream node build, push) plus release node, stream content jobs ahead of Sunday~~
testprojects and reviews with tripleo-ci-testing-hash release files (and depends-on change to tripleo-ci and tripleo-repos)
Run all branches with stream node on all jobs

Details

job names will NOT change.
registry and overcloud image locations will NOT change
users should only see that the nodepool node and repos and content are now centos-8-stream based.

Patches

Patches needed for integration line (RDO)

Updates release file (promotion-testing-hash-master) to use centos-8-stream repos
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/774911
Updates ansible task to call tripleo-repos to with –stream in case it is a stream node
- https://review.opendev.org/c/openstack/tripleo-ci/+/775429
Adding c8-stream nodesets for tripleo-ovb-centos-8-stream-primary-vexxhost, two-centos-8-stream-nodes-vexxhost
- ~~https://review.rdoproject.org/r/#/c/32080/~~
- vexx and psi? or just vexx?
Updates the actual integration line to stream repo on release node - https://review.rdoproject.org/r/#/c/32146/
TODO:
- either integration line nodesets
- add nodeset to jobs ( also tricky becuase of inheritance)
- or change nodeset label (most tricky - using centos 8 release on other releases - unless they all move to stream)
Updates release file (promotion-testing-hash-train) to use centos-8-stream repos
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/778554
Updates release file (promotion-testing-hash-ussuri) to use centos-8-stream repos
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/778555
Updates release file (promotion-testing-hash-victoria) to use centos-8-stream repos
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557

NOTE: ci.centos - what is required there???

Testproject jobs - Duplicate of master integration line but with centos-8-stream node/repos
- Centos-8 nodes + Centos-8-Stream repo
  - https://review.rdoproject.org/r/#/c/32041/
- Centos-8-Stream nodes + Centos-8-stream repo
  - https://review.rdoproject.org/r/#/c/32080/
- Other patches for testing
  - https://review.rdoproject.org/r/#/c/29969/26/.zuul.yaml
  - https://review.rdoproject.org/r/#/c/32063/ - testing failing jobs
Testproject jobs - Victoria integration line - centos 8 stream repos on release node
- https://review.rdoproject.org/r/#/c/32158/
Testproject jobs - Ussuri integration line - centos 8 stream repos on release node
- https://review.rdoproject.org/r/32159
Testproject jobs - Train integration line - centos 8 stream repos on release node
- https://review.rdoproject.org/r/#/c/32160/

Patches needed for component lines - needed as soon as we promote (RDO)

No container or image build jobs
Possibly test with node definition change

Patches neeeded for upstream check

DNM: Testing centos-8-stream node https://review.opendev.org/c/openstack/tripleo-ci/+/771717
DNM: Add master specific build image jobs for stream https://review.opendev.org/c/openstack/tripleo-ci/+/778189
DNM: Update master release to c8 stream repos/image https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655

Order of patches to be merged:

Updates release file (promotion-testing-hash-master) to use centos-8-stream repos
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/774911
Updates ansible task to call tripleo-repos to with –stream in case it is a stream node
- https://review.opendev.org/c/openstack/tripleo-ci/+/775429
Updates the actual integration line to stream repo on release node
- https://review.rdoproject.org/r/#/c/32146/
PROMOTION
DNM: Update master release to c8 stream repos/image https://review.opendev.org/c/openstack/tripleo-quickstart/+/777655
DNM: Testing centos-8-stream node https://review.opendev.org/c/openstack/tripleo-ci/+/771717
DNM: Add master specific build image jobs for stream https://review.opendev.org/c/openstack/tripleo-ci/+/778189
** NO COMPONENT CHANGES NEEDED - no build image or containers **

rdo patches w/ -1 wf to change centos-8 image to stream
- https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L371 (single node is laready merged)
- https://review.rdoproject.org/r/#/c/32061/ - ovb and two centos nodes
- https://review.rdoproject.org/r/#/c/32146/ - updates integration line to stream repo on release node
upstream patches w/ -1 wf on Changing nodes to centos-8-stream

Bugs found from test project jobs

Some repos are not 8-stream repos
- https://logserver.rdoproject.org/69/29969/29/check/periodic-tripleo-ci-build-containers-ubi-8-no-push/1c11f28/etc/yum.repos.d/ - 775429/1
- ^ These repos are laid down by tripleo-repos here - https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/tripleo-repos/tasks/main.yml#L92 . But for stream OS a flag –stream has to be passed to tripleo-repos command.
$stream is not rendered by the node - 775429/2
https://logserver.rdoproject.org/69/29969/36/check/periodic-tripleo-centos-8-buildimage-overcloud-full-master-no-upload/72ea9e7/overcloud-full.log - nothing provides pacemaker-libs(x86-64) = 2.0.5-7.el8 needed by pacemaker-remote-2.0.5-7.el8.x86_64 - 29969/38

Errors in integration line testprojects

Master

periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master FAILURE in 44m 40s (https://bugs.launchpad.net/tripleo/+bug/1917621)
periodic-tripleo-ci-centos-8-scenario010-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
periodic-tripleo-ci-centos-8-scenario004-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
periodic-tripleo-ci-centos-8-scenario001-standalone-master (https://bugs.launchpad.net/tripleo/+bug/1917621)
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master (https://bugs.launchpad.net/tripleo/+bug/1917583)
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master (same error as https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/8fceedf/logs/undercloud/var/log/tempest/stestr_results.html.gz - logged??)

Victoria

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria (tempest errors - https://logserver.rdoproject.org/58/32158/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/1374917/logs/undercloud/var/log/tempest/stestr_results.html.gz)

Check list for dday

Target dday for upstream master migration

Next Steps:

Revert https://review.rdoproject.org/r/#/c/32161/ to re-enable stable branch lines
Victoria release file https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557
(same ofr ussuri, train)
Change node label rdo and upstream

OPEN Questions:

ci.centos (no plan for stream nodes there)
https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-master-current-tripleo/894/ passing job there so far

Stream on stream items ( after all branches are moved)

(nodepool) merge rdo changes to convert centos8 to centos-8-stream
(nodepool) merge upstream changes to change from centos8 to centos-8-stream nodes

Victoria ( topic : centos-8-stream-content-victoria)

Testproject for integration line - https://review.rdoproject.org/r/#/c/32158/
Test topic for check jobs - TO DO
Releasefile for integration line - https://review.opendev.org/c/openstack/tripleo-quickstart/+/778557
Updating the integration line - https://review.rdoproject.org/r/#/c/32224/
Run Integration
Releasefile for upstream - https://review.opendev.org/c/openstack/tripleo-quickstart/+/779315
Promote

Ussuri ( topic : centos-8-stream-content-ussuri)

Train ( topic : centos-8-stream-content-train)

Testproject for integration line - https://review.rdoproject.org/r/#/c/32160/ WF -1
- Failures: https://review.rdoproject.org/zuul/status
  - periodic-tripleo-ci-centos-8-scenario002-standalone-train - Standalone deploy Timed out at Wait for containers to start for step 3 using paunch - Passed in Testproject https://review.rdoproject.org/r/#/c/32063/
  - periodic-tripleo-ci-centos-8-scenario003-standalone-train - Standalone deploy Timed out - Passed later in https://review.rdoproject.org/r/#/c/32063/
  - periodic-tripleo-ci-centos-8-scenario010-standalone-train - ERROR ovsdbapp.backend.ovs_idl.idlutils - Unable to open stream to tcp:192.168.24.1:6642 to retrieve schema: Connection refused - Failed again
  - periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train - Did not rerun
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train - Did not rerun
  - periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train - Passed in https://review.rdoproject.org/r/#/c/32063/
Test topic for check jobs - TO DO
Releasefile for integration line - https://review.opendev.org/c/openstack/tripleo-quickstart/+/778554 WF -1
Updating the integration line - https://review.rdoproject.org/r/#/c/32284/ WF -1
Enable integration stable3 -
Run Integration - - https://review.rdoproject.org/r/#/c/32349/
Releasefile for upstream - https://review.opendev.org/c/openstack/tripleo-quickstart/+/779803 WF -1 - has errors
Promote

Nodeset changes

Use stream labels on all centos 8 nodesets - https://review.rdoproject.org/r/c/rdo-jobs/+/32414 - (cloud specific nodes)
Remove stream nodesets from integration line - https://review.rdoproject.org/r/c/rdo-jobs/+/32483
Remove c8 stream specific nodesets definition from rdo - https://review.rdoproject.org/r/c/rdo-jobs/+/32413
DNM: Use stream labels on all centos 8 nodesets https://review.opendev.org/c/openstack/tripleo-ci/+/780654
Remove the centos-8-stream pipeline https://review.rdoproject.org/r/c/config/+/32439
Remove centos-8-stream pipeline jobs https://review.rdoproject.org/r/c/rdo-jobs/+/32440
Move tripleo_image_source to c8 stream - https://review.opendev.org/c/openstack/tripleo-ci/+/780899
Remove tripleo-image-source var from integration lines - https://review.rdoproject.org/r/c/rdo-jobs/+/32583
TODO: Remove stream-specific definitions from upstream (last thing) - https://review.opendev.org/c/openstack/tripleo-ci/+/780950

wes hayutin

2021/03/04 13:31:20

M: Add master specific build image jobs for stream

needs a topic branch for upstream, a topic branch for rdo (Edited)

2021/03/04 20:18:21

we have all integration jobs passing on c8 integration job

links to test projects please (Edited)

2021/03/04 20:22:23

revert patch to disable master line and run the master integration line

is this still need now that we are re-enabling this line today? (Edited)

2021/03/04 20:22:55

turn on master promotions for centos-8-stream

same.. still needed? (Edited)

rlandy

2021/03/04 21:02:59

I ma leaving the whole check list her e- we can mark things off on the day of they are done (Edited)

2021/03/04 21:05:05

https://review.rdoproject.org/r/#/c/32041/ master (Edited)

2021/03/04 21:05:39

https://review.rdoproject.org/r/#/c/32158/ victoria (Edited)

2021/03/04 21:05:53

https://review.rdoproject.org/r/#/c/32159/ ussuri (Edited)

2021/03/04 21:06:07

https://review.rdoproject.org/r/#/c/32160/ train (Edited)

sshnaidm

2021/03/07 12:56:24

completes

https://review.rdoproject.org/zuul/buildset/c20862854e764ec3b631a08676565bf6 (Edited)

2021/03/07 19:45:40

ictori

re-ran 035 here: https://review.rdoproject.org/r/#/c/32193/ and it passed (Edited)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

CentOS-8-Stream upstream migration

tags: Design

Aim

Steps to success :)

Stream Content on CentOS-8 nodes ( test project ) https://review.rdoproject.org/r/#/c/32041/

Stream Content on Stream nodes ( test project ) https://review.rdoproject.org/r/#/c/32080/

Ensure content-providers for centos-8-stream ( upstream DNM patch )

CentOS-8-Stream promotion for master

Document revert path

Target for Wednesday - March 3

Details

Patches

Patches needed for integration line (RDO)

Patches needed for component lines - needed as soon as we promote (RDO)

Patches neeeded for upstream check

Order of patches to be merged:

Bugs found from test project jobs

Errors in integration line testprojects

Master

Victoria

Check list for dday

Target dday for upstream master migration

Next Steps:

OPEN Questions:

Stream on stream items ( after all branches are moved)

Victoria ( topic : centos-8-stream-content-victoria)

Ussuri ( topic : centos-8-stream-content-ussuri)

Train ( topic : centos-8-stream-content-train)

Nodeset changes

tags: `Design`