Deploy TripleO on CentOS Stream-9 in CI for master & wallaby

tags: Design

Repos and DLRN API ENDPOINT

DLRN repos

• https://trunk.rdoproject.org/centos9-master/
  • Current Using consistent repos

DLRN API ENDPOINT

• https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36156
• API endpoint testing in progress

CentOS Composes

• https://composes.stream.centos.org/production/

Container-tools

• container-tools:latest

CentOS stream 9 package Koji

• https://kojihub.stream.centos.org/koji/index

Code review topic

Use topic or hashtag: "cs9"

• https://review.opendev.org/q/(topic:%2522cs9%2522+OR+hashtag:cs9)

Current Status

• https://review.opendev.org/c/openstack/tripleo-common/+/800580/comment/cdc0104d_55ca030d/ *

Build Container Pipeline

GOAL: Build containers in integration line

Reviews

• Added support for CentOS stream 9 container build - https://review.opendev.org/c/openstack/tripleo-ci/+/800577 (chandan) - READY
• Update tcib for CentOS Stream 9 - https://review.opendev.org/c/openstack/tripleo-common/+/800580 (chandan) - READY
• DNM: Trying out CentOS Stream 9 container build - https://review.rdoproject.org/r/c/testproject/+/18953
• Switch to mariadb:10.5 module - https://review.opendev.org/c/openstack/tripleo-common/+/808009
• openstack-periodic-integration-main-cs9 pipeline - https://review.rdoproject.org/r/c/config/+/35817
• Adds component criteria for master on c9 stream - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/35832
• Adding pipeline and jobs for c9 stream - https://review.rdoproject.org/r/c/rdo-jobs/+/35831

TODO:

• release[distro] skiplist design for container build ( keep in mind the promoter)

Tasks remaining here:

• Add a new pipeline to trigger cs9 jobs - Ananya (Done)
• Add a job to promote content from consistent to tripleo-ci-testing - Ananya
• Add the container-build job definition - Ananya
• Enable container-build in the new pipeline - Ananya
• Start looking into downstream container-build on RHEL9 (ysandeep)
• Add support for CentOS stream 9 in tripleo-repos (existing tooling)

Image Build Pipeline

GOAL: Build overcloud and IPA images in integration line

marios looking here until sagi is back

hackmd (Sagi's Work)

• https://hackmd.io/@rh-openstack-ci/rJeaXyVZK

Reviews

• Add configs for CentOS 9 image - https://review.opendev.org/c/openstack/tripleo-common/+/806018
• Preliminary work to support CentOS 9 Stream - https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/806809
• Correct path for CentOS 9 stream base image - https://review.opendev.org/c/openstack/diskimage-builder/+/806819
• Add support for CentOS Stream 9 - https://review.opendev.org/c/openstack/tripleo-ci/+/808177
• Overcloud Image build - https://review.rdoproject.org/r/c/testproject/+/35465
• [WIP] Add support for CentOS Stream 9 in DIB - https://review.opendev.org/c/openstack/diskimage-builder/+/811392 (alfredo)
• Bunch of patches from Steve - https://review.opendev.org/q/topic:"grubenv"+(status:open OR status:merged)

Tasks Remaining (marios)

• Finish Diskimage builder support for cs9
• Add the image build job definition
• Enable image build in the new pipeline

Run Standalone and Undercloud

GOAL: Enable standalone and undercloud job

Reviews

• [WIP] centos 9 periodic release file - https://review.opendev.org/c/openstack/tripleo-quickstart/+/806910

• DNM: Trying out CentOS Stream 9 standalone job | https://review.rdoproject.org/r/c/testproject/+/35229

• Remove fetch-zuul-cloner https://review.rdoproject.org/r/c/config/+/36113

• update endpoint and baseurl https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3615

I am currently promoting hashes till tripleo-ci-tesing

• virtualenv https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36158
• Correct location of component criteria: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36174

Tasks Remaining (sandeep)

• Consume current containers in undercloud and standalone jobs
• Make sure IPA job and fs039 job works with mariadb:10.5

Bugs

1. https://bugzilla.redhat.com/show_bug.cgi?id=1996616 - nothing provides ansible-core needed by ansible-freeipa-0.3.8-3.el9.noarch

2. https://bugzilla.redhat.com/show_bug.cgi?id=1997069 - packages requiring "python3dist(ansible)" fails to install with ansible-core

3. https://issues.redhat.com/browse/RHELBU-1226 - Ansible Execution Environment and RPM in RHEL

4. https://bugzilla.redhat.com/show_bug.cgi?id=2003556 - mariadb-server in centos9 stream is not shipping wsrep_sst_rsync_tunnel

   Informational: https://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional

5. https://bugs.launchpad.net/tripleo/+bug/1946461 - CentOS Stream 9 missing packages tracker for container build

6. https://bugs.launchpad.net/tripleo/+bug/1947181 - C9 Standalone deploy fails on build-test-package dependency install task because of missing Mock package.

7. https://bugs.launchpad.net/bugs/1946822 - Standalone deployment on Centos9 failing during python3-tripleoclient dependencies install, nothing provides genisoimage needed by python3-ironicclient-4.8.0-0.20210921104434.b5df386.el9.noarch

8. https://bugs.launchpad.net/tripleo/+bug/1947193 - Centos9 standalone deployment failing with error: "heat.common.exception.StackValidationFailed: Property error: resources.AnsibleGroupVars.properties.value: Incorrect arguments: Items to merge must be maps."

Stream Pinning Tasks

Cockpit Design for Centos 9 and RHEL 9

• pages/columns/graphs need to be added
  • Component Pipeline Wallaby c9
  • Component Pipeline Master c9
  • Component Pipeline RHOS 17 RHEL 9
  • Upstream promotions c9 for Wallaby and Master
• ruck-rover tool and any data that needs to updated
  • Ruck-rover tool already support c9
• New overarching design
  • Wallaby Component Pipeline page
    • On this page we point to Centos 8 and Centos 9 with a status (red or green) for each
      • Might also have a comparison:
        • Latest promotion
        • Components
      • c8 And c9
        • Setup failrues to shows up first in the list of jobs
        • On the component promotion, setup colors if the difference between the days are too high
          • less than 1 day = green
          • 1 day = orange
          • 2 days = Red
    • general info

Promoter Changes for c9/RHEL 9

• new config
• new RHOS-17 config

Tempest Tasks

Emit releases

Single Source of truth

• While working on CS9 we should have single source of truth. We need to keep track of the jobs which are there for CS9.

Process

​​​​- Add jobs
​​​​- Add it to criteria
​​​​- Whatever jobs are present in the criteria are the valid jobs rest we don't know.

Component pipeline

• Jobs to add:
  • promote-consistent-to-component-ci-testing - DONE
  • promote-to-promoted-components - DONE
  • standalone component jobs - DONE
  • scenario001-standalone-component - WIP - POOJA
    • https://review.rdoproject.org/r/c/rdo-jobs/+/36959/1/zuul.d/component-jobs-master-centos9.yaml
  • scenario002-standalone-component jobs - Rafael
  • scenario003-standalone-component jobs - Rafael
  • scenario004-standalone-component jobs - POOJA
  • scenario007-standalone-component jobs - POOJA
  • scenario010-standalone-component jobs - POOJA
  • scenario010-kvm-standalone-component jobs - POOJA
  • scenario012-standalone-component jobs - POOJA
  • Update and Upgrade jobs - POOJA
  • standalone-upgrade jobs - POOJA
  • standalone-full-tempest-api-component jobs -BHAGYASHRI
  • standalone-full-tempest-scenario-component jobs - BHAGYASHRI
  • standalone-on-multinode-ipa jobs - BHAGYASHRI
  • ovb component promotion jobs - fs001 - BHAGYASHRI
  • multinode component jobs - BHAGYASHRI

Note: * From above refences added jira tickets/sub tasks under sub-story: https://issues.redhat.com/browse/TRIPLEOCI-675 * First add definations like https://review.rdoproject.org/r/c/rdo-jobs/+/36959/1/zuul.d/component-jobs-master-centos9.yaml * Second test those jobs using testproject patch * Once testproject passes add jobs in the integration line likbe this https://review.rdoproject.org/r/c/rdo-jobs/+/36959/1/zuul.d/component-pipeline-main-centos9.yaml * Add all the passing jobs in the promotion criteria

OVB JOB

• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master
• periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master
• periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master
• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master
• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-rbac-master
• periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-rbac-master
• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset064-master

CS9 upstream check and gate jobs

• https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/content-provider.yaml
• https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml
• https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/multinode-jobs.yaml

CS9 wallaby

• https://review.rdoproject.org/r/q/topic:"c9-wallaby"+(status:open OR status:merged)