CentOS-8 jobs

It tracks the list of patches for getting periodic standalone and ovb jobs & even upstream jobs, ITS ALL ABOUT CENTOS-8!!!

Summary Status: March 17 2020

• Upstream migrated to CentOS-8.
• Upstream component pipeline done.
• RDO content unpinned except for neutron, mistral
• First CentOS-8 promotion w/ unpinned content on March 16th
• OVB jobs have several open blocking issues
  • w/ current unmerged patches running at 50% succcess
• Promotion server, ready for CentOS-8

Known Blocking Bugs

1. OVB is stuck on providing nodes

   • https://bugs.launchpad.net/tripleo/+bug/1866204

2. overcloud deploy failing on fs030 and fs016 while pulling mariadb container from undercloud registry

   • https://bugs.launchpad.net/tripleo/+bug/1867599

3. overcloud deploy failed due to Systemd start for pcsd failed

   • https://bugs.launchpad.net/tripleo/+bug/1867602

Closed Blocking Bugs

• promoted-components dirs get reaped and the top-level file does not contain all components

  • https://bugs.launchpad.net/tripleo/+bug/1866696

• Duplicate declaration: Package[collectd-python] is already declared at puppet-collectd and puppet-tripleo

  • https://bugs.launchpad.net/tripleo/+bug/1866965

• network issues in centos-8 ovb deployment https://bugs.launchpad.net/tripleo/+bug/1866202

• centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters' -

  • https://bugs.launchpad.net/tripleo/+bug/1866184

• tripleo-ci-centos-7-scenario010-standalone jobs are failing tesmpest tests "ServiceError: Request on service 'load-balancer' with url 'http://192.168.24.1:9876' failed with code 500"

  • https://bugs.launchpad.net/tripleo/+bug/1866107

• tripleo-ci-centos-8-scenario001-standalone tempest-conf fails
  500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument

• https://bugs.launchpad.net/tripleo/+bug/1865754

• centos-8 multinode, undercloud, ovb jobs are hanging on the undercloud install

  • https://bugs.launchpad.net/tripleo/+bug/1865574
  • waiting on https://review.rdoproject.org/r/#/c/25647/

• python2 packages appearing in octavia host-prep-tasks

  • https://bugs.launchpad.net/tripleo/+bug/1848353

CentOS-8 container build

• [testproject]- try a build push centos-8 containers-https://review.rdoproject.org/r/#/c/25064/

Depends-On:

• [DNM] needed override setting in config repo https://review.opendev.org/#/c/709034/
• turn off temp dir cleanup in container builds https://review.opendev.org/#/c/708967/ - Emilien has other patches that have merged
• Use md5sum if baseurl shows up more than once in delorean.repo file https://review.opendev.org/#/c/709193/
• [kolla] - Build collectd image for CentOS8 - https://review.opendev.org/#/c/709701/
  • Needed for scenario-01 standalone
• turn off temp dir cleanup in container builds
  https://review.opendev.org/#/c/708967/
• Use md5sum if baseurl shows up more than once in delorean.repo file https://review.opendev.org/#/c/709193/

CentOS-8 tox (lint, unit, molecule) and upstream zuul roles

• 2020-03-17: getting close to switch default nodeset on rdo to centos-8, got agreement with yatin, waiting for tristan/amorale call. likely merge to happen during next 48h.
• fix docker for centos-8 https://review.opendev.org/708704
• fix upgrade project lint https://review.opendev.org/710111
• triple-ansible molecule -> centos8 https://review.opendev.org/#/c/711535/
• <wes> Sorin please migrate tripleo-upgrade-centos-7-molecule to centos-8</wes>
• c8-ready topic marks PRs making upstream changes for enabling centos-8 support.

CentOS-8 OverCloud Images

Summary: Issues w/ image server and permissions.

https://review.rdoproject.org/r/#/c/24997/
tripleo-centos-8-buildimage-overcloud-full SUCCESS in 19m 08s
tripleo-centos-8-buildimage-ironic-python-agent SUCCESS in 15m 29s (non-voting)

Current required reviews

• Depends-On: https://review.opendev.org/711078
• Fix image server for centos-8 perm
  • https://review.rdoproject.org/r/#/c/25393/
• [tripleo-ci] - Improve build images role for CentOS8 and check job - https://review.opendev.org/#/c/708599/
• [rdo-jobs] - Added CentOS-8 build images jobs -https://review.rdoproject.org/r/#/c/24997/
• Add periodic and base centos-8 build image jobs https://review.rdoproject.org/r/25611
• [tripleo-common] - Added overcloud images yaml for CentOS-8 - https://review.opendev.org/#/c/708606/
• [diskimage-builder] - Add CentOS 8 support - https://review.opendev.org/#/c/684308/
• [diskimage-builder] - Change tgt pkg-map to target-restore CentOS/RHEL-8 - https://review.opendev.org/#/c/709159/
• [tripleo-image-elements] - Disable yum-plugin-priorities elements for CentOS8 - https://review.opendev.org/#/c/709082/
• [tripleo-puppet-elements] - Disable yum-plugin-priorities elements for CentOS8 https://review.opendev.org/#/c/709085/

Standalone

Summary: centos-8-standalone and standalone scenario[1-4] running in check/gate upstream.

all jobs here are centos-8 obviously

• https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)
  Depends-On: https://review.opendev.org/#/c/710049/
  Depends-On: https://review.opendev.org/709837
  Depends-On: https://review.opendev.org/709868
  https://review.opendev.org/709959
  https://review.opendev.org/#/c/708402/
• Testing periodic standalone scenario-1-4 and full tempest
  • [rdo-jobs]- [WIP] CentOS-8 standalone 1-4 -https://review.rdoproject.org/r/#/c/25329/
• HA: fix rabbitmq readiness check for rabbitmq-server 3.8 https://review.opendev.org/#/c/710233/
• Depends-On: https://review.opendev.org/709996
  Depends-On: https://review.opendev.org/#/c/709727/
• Testing check plain standalone job
  • [tripleo-ci] - [WIP] CentOS-8 standalone check - https://review.opendev.org/#/c/709437/
• standalone centos-8 full tempest
  • break this into API and Scenario
  • https://review.rdoproject.org/r/#/c/25325/
• https://review.opendev.org/711287 Add centos-8 scenario010 standalone jobs.

OVB Jobs

**Summary: no succes here yet. Current error is:**mandre

2020-03-04 23:07:40 |   - '[2020/03/04 11:07:10 PM] [WARNING] Error in ''ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex'', restarting br-ex:'
2020-03-04 23:07:40 |   - Unexpected error while running command.
2020-03-04 23:07:40 |   - 'Command: /sbin/ip route add 0.0.0.0/0 via 10.0.0.1 dev br-ex'
2020-03-04 23:07:40 |   - 'Exit code: 1'
2020-03-04 23:07:40 |   - 'Stdout: '''''
2020-03-04 23:07:40 |   - 'Stderr: ''Cannot find device "br-ex"\n'''

https://bugs.launchpad.net/tripleo/+bug/1866202

• BUGS

We must use ethX-like interfaces, to do that we need to add net.ifnames=0 to GRUB options in DIB and build images with it.

(sshnaidm) I talked with RDO folks and we agreed to add this parameter to DIB element bootloader. After it's added need to ask both RDO and upstream infra to include it in building centos8 image.
patch of ethX interfaces for RDO:~~

• https://softwarefactory-project.io/r/#/c/17697/

patch of ethX interfaces for upstream:

• Add option to get ethX interfaces on centos-8 https://review.opendev.org/710986
• semms like it's not gonna be merged, got pushbacks from infra about changing defaults, let's revisit the need

Testing fs01 ovb job

• [rdo-jobs] - [WIP] CentOS8 ovb fs01 - https://review.rdoproject.org/r/#/c/25332/

Patch that fixes preparing playbook is extracted to:

• https://review.opendev.org/#/c/710982/ Remove unnecessary yum calls (passes this playbook here)

Depends-On: https://review.opendev.org/#/c/709560/
Depends-On: https://review.opendev.org/709866 - needs review
Depends-On: https://review.opendev.org/709813
Depends-On: https://review.opendev.org/709837
Depends-On: https://review.opendev.org/709868
Depends-On: https://review.opendev.org/#/c/710887/ - needs review

• Fixed the mirror path for atop - https://review.opendev.org/711894
• Set overcloud_container_cli to podman for CentOS8 - https://review.opendev.org/711798

Depends-On:
* [config] - centos8 var file for ovb-manage role - https://review.rdoproject.org/r/25335
*

Multinode jobs:

Summary: Passing inconsistently - need errors posted

* add to upstream check https://review.opendev.org/#/c/710179/
it has same issue as OVB or any other job that tries to install undercloud

need multinode-scenario000-updates

tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates
https://review.opendev.org/#/c/710876/

failure: https://4a7ff0ffe3e6d0e85ad2-bfdf0aa5980ea9561e0ca2e62d8db60b.ssl.cf2.rackcdn.com/710876/2/check/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/0e9fb60/logs/undercloud/home/zuul/overcloud_update_prepare.log

Zuul project overrides

Most tripleo projects override the default zuul settings in tripleo-ci

See the following:

• https://review.opendev.org/#/q/topic:centos-8-jobs+(status:open+OR+status:merged)
• https://bugs.launchpad.net/tripleo/+bug/1865525

Promotion jobs

periodic-tripleo-centos-8-master-promote-promoted-components-to-tripleo-ci-testing
periodic-tripleo-centos-8-master-containers-build-push
periodic-tripleo-centos-8-buildimage-overcloud-full-master
periodic-tripleo-centos-8-buildimage-ironic-python-agent-master
periodic-tripleo-ci-centos-8-standalone-master
periodic-tripleo-ci-centos-8-scenario001-standalone-master
periodic-tripleo-ci-centos-8-scenario002-standalone-master
periodic-tripleo-ci-centos-8-scenario003-standalone-master
periodic-tripleo-ci-centos-8-scenario004-standalone-master
periodic-tripleo-ci-centos-8-scenario007-standalone-master
periodic-tripleo-ci-centos-8-scenario010-standalone-master
periodic-tripleo-ci-centos-8-singlenode-featureset027-master
periodic-tripleo-ci-centos-8-scenario010-standalone-master
periodic-tripleo-ci-centos-8-containers-undercloud-minion-master
periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master
# OVB jobs - not fully qualified
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master

• Fix criteria match from promote_source - https://review.rdoproject.org/r/#/c/25363/

• Add promote_source variable to component pipeline standalone job. https://review.rdoproject.org/r/25410

• Get the dlrn reporting output to check results https://review.rdoproject.org/r/25416

• Adds Standalone CentOS 8 Jobs {plain standalone +1,2,3 scenario} - https://review.rdoproject.org/r/#/c/25415/

• add centos-8 containers-multinode - https://review.rdoproject.org/r/25414

• Replace centos-7 jobs with centos-8 jobs in the master pipeline https://review.rdoproject.org/r/#/c/25429/

• https://review.rdoproject.org/r/25543 Move component pipelines to run daily

• Testing undercloud and multinode periodic https://review.rdoproject.org/r/#/c/25411/

• https://review.rdoproject.org/r/25549 Replace centos 7 with centos 8 for fs027 and fs010 - master

• https://review.rdoproject.org/r/25550 DNM: Add centos 8 fs030

• Test standalone tempest https://review.rdoproject.org/r/#/c/25325/

• https://review.rdoproject.org/r/25718 Add c8 standalone-full-tempest to the master integration pipeline

• https://review.rdoproject.org/r/#/c/25757/ Remove the last c7 jobs and move master pipeline to 3x daily

standalone 7 and 10 periodic jobs:

• Added c8 scenario 7 and scenario 10 standalone - https://review.rdoproject.org/r/25618
  Depends-On:

• Fixed package names for CentOS-8 octavia deployment - https://review.opendev.org/710913

• Use ansible_python to set ansible_python_intepreter - https://review.opendev.org/710980

• Add periodic centos8 scenario10 ovn standalone master job definition https://review.rdoproject.org/r/25745

• Add centos8 scenario10-ovn-provider-standalone-master periodic layout https://review.rdoproject.org/r/25746

• Add periodic-scenario010-ovn standalone commented master centos8 criteria https://review.rdoproject.org/r/25747

• Added tripleo-ci-centos-8-scenario007-multinode-oooq-container - https://review.opendev.org/712294

• Remove rhel_image_source workaround - https://review.opendev.org/712248

• Remove duplicate entry of collectd-python package - https://review.opendev.org/712289

• Master C8 ovb check and periodic job - https://review.rdoproject.org/r/25804

• Added c8 ovb fs01 to periodic pipeline and check - https://review.rdoproject.org/r/25822

Update jobs

• https://review.opendev.org/710876/ Add centos-8 scenario000-multinode update job
  • previous-current-tripleo pin added
  • watching review for status
• https://review.opendev.org/712124 tripleo-ci master: Make centos8 scenario000 multinode job voting in gate
• https://review.rdoproject.org/r/25800 rdo-jobs master: Add periodic centos8 featureset037 multinode updates master job
• https://review.rdoproject.org/r/25801 config master: Add periodic centos-8 featureset037-updates-master to master pipe
• rdo-infra/ https://review.rdoproject.org/r/25799 ci-config master: Add centos8 fs37 updates master to centos8 master criteria
• (test) https://review.rdoproject.org/r/25802 Run periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master

Release files and base jobs

• update dashboards https://code.engineering.redhat.com/gerrit/#/c/194131/1/feed-dashboard.sh
• Add release files for master CentOS 8 https://review.opendev.org/#/c/709560/
• Add base centos 8 jobs for parenting https://review.opendev.org/#/c/709950/
• Add base centos 8 jobs for parenting https://review.rdoproject.org/r/#/c/25390/
• update build-test-packages for centos-8 https://review.opendev.org/#/c/709837/
• Fixed the condition for gather_domains https://review.opendev.org/#/c/709959
• Add minimal promotion criteria file for centos 8 master https://review.rdoproject.org/r/#/c/25394/
• Add centos-8 nodeset for OVB jobs https://review.rdoproject.org/r/#/c/25391/
• Correct the component dlrn repo link in centos 8 master release file https://review.opendev.org/#/c/710134/
• Added missing -y with epel install command - https://review.opendev.org/710203
• Set container build hash based on dlrn md5 checksum - https://review.opendev.org/710230 NEEDS REVERT
• Update emit releases - https://review.opendev.org/710441
• https://review.rdoproject.org/r/25715 Add tests to the master centos-8 criteria file

Component Pipeline

Summary: first pass complete w/ promtion jobs and standalone only

Testing

• https://review.rdoproject.org/r/25411 Component pipeline
  * https://review.rdoproject.org/r/#/c/25411/ Promote jobs

Reviews

• https://review.rdoproject.org/r/#/c/25649/ Add component scenarios and multinode tests
• Depends-On: https://review.opendev.org/#/c/711156 Add job.repos to container updates for component jobs
• https://review.rdoproject.org/r/#/c/25702/ Build out component pipeline with additional jobs
• Move component pipeline to run 1.5 hours apart
• https://review.rdoproject.org/r/25717
  Add centos-8 component standalone tempest jobs
• https://review.rdoproject.org/r/25767 Skip promoted-components promotions if the hash is in target

Second Phase - Prepare for the shinnin ( unpinning )

In order to have a known good working build..

Add any standalone, multinode, and ovb job to the component pipeline to ensure broken components do not get promoted.

| Pipeline | Jobs |
| –––– | –––– | –––– |
| component-baremetal | ovb fs001, standalone, standalone 12
| component-cinder | standalone, 01,04
| component-clients | containers-multinode, standalone
| component-cloudops |standalone 01,02
| component-common | standalone[1-4], containers-multinode
| component-compute | standalone-fulltempest,
| component-glance | standalone[1-4], x
| component-manila | standalone04, x
| component-network | containers-multinode, standalone 7,10
| component-octavia | containers-multinode, standalone 7,10
| component-security | contaiers-multinode, standalone
| component-swift | standalone02, x
| component-tripleo | containers-multinode, standalone
| component-tempest | standalone-fulltempest-api standalone-fulltempest-scenario
| component-ui | standalone

Failing component pipeline tests

scenario001 tests - same failure as in integration pipeline (https://bugs.launchpad.net/tripleo/+bug/1866965)

• periodic-tripleo-ci-centos-8-scenario001-standalone-cinder-master
• periodic-tripleo-ci-centos-8-scenario001-standalone-common-master
• periodic-tripleo-ci-centos-8-scenario001-standalone-cloudops-master
• periodic-tripleo-ci-centos-8-scenario001-standalone-glance-master

featureset010 - msg: 'Container(s) with bad ExitCode: [''container-puppet-ironic_inspector''], check logs in /var/log/containers/stdouts/'

• periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-clients-master

standalone - only in UI (may be timing)

• periodic-tripleo-ci-centos-8-standalone-ui-master

full-tempest

• periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master
• periodic-tripleo-ci-centos-8-standalone-full-tempest-tempest-master

Unpinning Ussuri

Required coverage prior unpinning Ussuri:

• Undercloud jobs
• Standalone , full tempest, Standalone scenario 1-4, 7, 10
• Containers-Multinode
• OVB featureset01
• (tripleo weekly irc) 16:07 < marios|ruck> weshay|ruck: chandankumar: how about a full-tempest job or -api/-scenario pair

Testing the unpinned commits from OpenStack Ussuri will have two fronts.

1. Create a series of testproject like changes e.g. https://review.rdoproject.org/r/#/c/24683/ to test the unmerged unpinning review.

• Chandan is using the following patch to preemptively test the unpinning
  • https://review.rdoproject.org/r/25612

2. Use a second and temporary phase of the component pipeline. See link

Reviews:

• Added jobs for distgit testing - https://review.rdoproject.org/r/25659

• [Gaint review] [WIP] Unpin pinned projects in Ussuri - https://review.rdoproject.org/r/25612

  • Current Status: Standalone , full temepst, scenario-2,3,4,7,10 and fs027 passing
    • fs010 is passing as we have pinned mistral now https://review.rdoproject.org/r/#/c/25612/16/tags/ussuri-uc.yml@318
    • https://bugs.launchpad.net/tripleo/+bug/1866621

• glance - Remove policy.json from spec file - https://review.rdoproject.org/r/#/c/25723/

Taking ironic un pinning in a seperate patch and testing there:

• ovb distgit jobs - https://review.rdoproject.org/r/25726
• Unpinning Ironic components - https://review.rdoproject.org/r/25727
• Add rootwrap.d/ironic-lib.filters file - https://review.rdoproject.org/r/#/c/25855/

Upstream Check jobs episode

• Added CentOS-8 standalone jobs - https://review.opendev.org/710177
• Add c8 multinode containers job - https://review.opendev.org/710179
• make centos-8 build containers, voting, gating - https://review.opendev.org/#/c/710091/2
  * [DNM] centos8 multionde tests - https://review.opendev.org/710196
• Build collectd image for CentOS8 - https://review.opendev.org/709701
• Improve build images role for CentOS8 https://review.opendev.org/#/c/708599/

Undercloud jobs

Summary: bug, need to land patches

• c8 undercloud job - https://review.rdoproject.org/r/25366
• https://review.opendev.org/710856 tripleo-ci upstream check
• Added tripleo-ci-centos-8-undercloud-oooq - https://review.opendev.org/710906
• Adds tripleo-ci-centos-8-containers-undercloud-minion - https://review.opendev.org/710961

Repo personification

• DNM yet, add current w/ include_pkgs to tripleo-ci-testing tag - https://review.opendev.org/710098

Moving CentOS 8 deps repo to CBS builds (test only)

* DNM Test new deps repo using CBS builds - tripleo-repos - https://review.opendev.org/#/c/710272/
* DNM Test new deps based on final CBS builds - kolla - https://review.opendev.org/#/c/710240/
* WIP Add support for ceph in CentOS 8 jobs and test cbs deps - p-o-i - https://review.opendev.org/#/c/707110/
* [DNM] Test new splitted deps - packstack - https://review.opendev.org/#/c/709197/
* [DNM] Test new deps repo based on CBS builds - oooq - https://review.opendev.org/#/c/710232/
* [DNM] Test new deps in CBS in CentOS-8 standalone 1-4 - rdo-jobs - https://review.rdoproject.org/r/#/c/25421/

Promotion Server

Summary: container promotion is not working correctly

* https://review.rdoproject.org/r/#/c/24716/
* https://review.rdoproject.org/r/#/c/24721/
* https://review.rdoproject.org/r/#/c/24782/
* https://review.rdoproject.org/r/#/c/24818/

• BIG ONE https://review.rdoproject.org/r/#/c/24774/
• https://review.rdoproject.org/r/#/c/25782/