# Ruck Rover - 08 Apr 2022 - 14 Apr 2022
###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/M1UXHF7iTbmJFkK4bQET9g?edit
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

## HandOver Notes:
* https://bugs.launchpad.net/tripleo/+bug/1968615
* https://bugs.launchpad.net/tripleo/+bug/1964129
* https://bugs.launchpad.net/tripleo/+bug/1968184
* https://bugs.launchpad.net/tripleo/+bug/1968732
* Upstream vanilla cs9 standalone is broken at validation check
* fix https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837801
* RHOS-17 rhel-8
* ceph-tag bump to latest: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/402150
* RHOS17 RHEL9
* Fix with testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/402613
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/9c0af39/job-output.txt
`package rdo-openvswitch-1:2.15-2.el9s.noarch requires network-scripts-openvswitch2.15, but none of the providers can be installed`
* https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/402643

* Not urgent: https://bugs.launchpad.net/tripleo/+bug/1960310

## Notes
* Downstream Line
* https://code.engineering.redhat.com/gerrit/c/testproject/+/402613 is green and get https://review.opendev.org/c/zuul/zuul-jobs/+/837850
* Upstream line
* Make sure it merges https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837801

## Active Bugs
- https://bugs.launchpad.net/tripleo/+bug/1969158 master OVB jobs are failing overcloud deployment - UNREACHABLE | Manage firewall rules
- https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/view/phase-1-pipelines/job/rdo_trunk-promote-train-centos8-current-tripleo/
- [RHOS17 RHEL-9, wallaby upstream blocker] https://bugs.launchpad.net/tripleo/+bug/1968732
- test moved to skip list
- [Validations-libs] https://bugs.launchpad.net/tripleo/+bug/1968184
- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836988
- https://review.opendev.org/c/openstack/tripleo-ci/+/834631
- https://bugs.launchpad.net/tripleo/+bug/1968184/comments/3 can close based on this
- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837851
- [RHOS-16.2] https://bugzilla.redhat.com/show_bug.cgi?id=2073227
- https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/402027 merged
- Need to wait for network component
- [master, buildah] https://bugs.launchpad.net/tripleo/+bug/1964129
- [master, undercloud-upgrade] https://bugs.launchpad.net/tripleo/+bug/1943938
- https://review.opendev.org/c/openstack/tripleo-ci/+/837512
- Making undercloud upgrade job nv
- [IPA DNS] https://bugs.launchpad.net/tripleo/+bug/1968615
- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/837410
- https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=incident&sys_id=b6189b29dbba8d98c31ec7a605961956)
- downstream-trigger job has started failing: https://review.rdoproject.org/zuul/builds?job_name=create-scenario010-trigger-for-downstream
Discussion on #opendev
<opendevreview> Ian Wienand proposed opendev/system-config master: mirror-update: switch fedora to kernel.org mirror https://review.opendev.org/c/opendev/system-config/+/837625

## 14th Apr, 2022

### rlandy notes
* 16.2 network fix is in. All the network CI tests passed.
Waiting on QE Jenkins jobs to pass before rekciking 16.2 line. Rekciked twice. pls ping attila if it failed
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_component-network-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/
* **Wallaby c8 promoted** - Reverting https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41241 Jakob's second patch. After that we need to revert the first.

### Upstream Promotion
* Master CS9
* Wallaby CS9
* https://bugs.launchpad.net/tripleo/+bug/1968732
* Wallaby CS8
~~https://review.rdoproject.org/r/q/topic:temp_criteria_removal_c8_wallaby need to merge it~~
* Train CS8
Re-running train failed jobs: https://review.rdoproject.org/r/c/testproject/+/41905

### Downstream promotion
* RHOS17 RHEL9
* Fix with testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/402613
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/9c0af39/job-output.txt
`package rdo-openvswitch-1:2.15-2.el9s.noarch requires network-scripts-openvswitch2.15, but none of the providers can be installed`
* https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/402643

* RHOS17 RHEL8
* ceph-tag bump to latest: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/402150
* testproject passing: https://code.engineering.redhat.com/gerrit/c/testproject/+/401805/2#message-06d33cdae50aed15a6210be43479d59695ea0020

* RHOS16 RHEL8
- https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_component-network-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/191/ passed
- Wait for RHOS-16.2 line to kick

### Gate Failure

## 13th Apr, 2022

### Notes
* remove fs01/fs035 from cs8 wallaby criteria patch ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41870~~
https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41896 has to be merged
* Keep an eye on cs8 wallaby promotion
* After promotion, revert the patches

* ceph-tag bump to latest: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/402150
* testproject passing: https://code.engineering.redhat.com/gerrit/c/testproject/+/401805/2#message-06d33cdae50aed15a6210be43479d59695ea0020
* Please merge this one, then kick RHEL8 RHOS-17 line

* Check with Jesse to close this bug https://bugs.launchpad.net/tripleo/+bug/1943938 since undercloud upgrade jobs are passing.

### RDO Promotions

* Master CS9
* Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/41847

* Wallaby CS9
* Promoted

* Wallaby CS8
* re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/36255

* Train CS8

### Downstream Promotions

* RHOS 17 RHEL 9
* Promoted

* RHOS 17 RHEL 8
* Container build timeout, re-running here: https://code.engineering.redhat.com/gerrit/c/testproject/+/401779 passed
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/401800 to fix sc jobs

* RHOS 16 RHEL 8
* Promoted

### Gate Failures
* Undercloud upgrade job failure: https://bugs.launchpad.net/tripleo/+bug/1943938

### LP bug 1960310

* https://bugs.launchpad.net/tripleo/+bug/1960310
* wallaby tempest bump: https://review.rdoproject.org/r/c/rdoinfo/+/41809
* Depends-On https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/41824
* Testing master skip:
* https://review.rdoproject.org/r/c/testproject/+/41826
* unskipped tests: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/837544
* Testing wallaby skip
* Testing it with https://review.rdoproject.org/r/c/rdoinfo/+/41809
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/837562
~~* Yoga and Xena tempest bump: https://review.rdoproject.org/r/c/rdoinfo/+/41810~~

## 12th Apr, 2022

### Notes for rlandy
* **WALLABY c9 PROMOTED**
* **WALLABY c8 still not promoted** - running https://review.rdoproject.org/r/c/testproject/+/36255
* https://code.engineering.redhat.com/gerrit/c/testproject/+/401837 to re-run kvm job
* Need to work on promoting cs8 wallaby
* RHEL-9 rhos17 will also promote after
https://code.engineering.redhat.com/gerrit/c/testproject/+/401851
* **RHEL-9 rhos17 PROMOTED**
* **16.2 PROMOTED**
* Keep an eye on undercloud upgrade job
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/836596 onces merged **(merged)**
* if this issue seen in the job: https://bugs.launchpad.net/tripleo/+bug/1943938/comments/7
* open a new bug

### Gate Blockers:
* https://bugs.launchpad.net/tripleo/+bug/1968732
* Impacted release
* Wallaby gate
* RHOS-17 RHEL-9
* Skip list patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/837569
* test patch: https://code.engineering.redhat.com/gerrit/c/testproject/+/401851

### RDO promotions
* Master CS9

* Wallaby CS9
* [IPA job re-run] https://review.rdoproject.org/r/c/testproject/+/41834

* Wallaby CS8
* [IPA job re-run] https://code.engineering.redhat.com/gerrit/c/testproject/+/401837

* Victoria CS8

* Train CS8

### Downstream promotion

* [RHOS-16]
* https://code.engineering.redhat.com/gerrit/c/testproject/+/401776 - rerunning fs01, 35, 20

* [RHOS17 RHEL8]
* https://code.engineering.redhat.com/gerrit/c/testproject/+/401805
* depends-on: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/401800 (ceph bump)
* More info:
* need this patch https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/400430
* tripleo promotion get this in integration line
* https://bugzilla.redhat.com/show_bug.cgi?id=2072665
* https://access.redhat.com/errata/RHSA-2022:1174

* [RHOS-17 RHEL-9]
* https://code.engineering.redhat.com/gerrit/c/testproject/+/401851

## 11th Apr, 2022

Notes:
- Master promotion, need f039 - https://review.rdoproject.org/r/c/testproject/+/40169
- cs9 wallaby promotion needs 64 - https://review.rdoproject.org/r/c/testproject/+/41794
- Question to Chandan: How to decide whether to use `zed.yml` or `zed-uc.yml`? [README.rst](https://github.com/redhat-openstack/rdoinfo/blob/master/README.rst) says:

> * *release* is the name of Openstack release
> * *release-uc* all projects with in the files, pins to the source-branch of all projects
> included in Openstack upper-constraints to the versions in that file.
> In other words, `-uc` tag is for the current development release with
> all dependencies that are set to version defined in upper-constraints file.

* Downstream
* Jobs are failing with https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?result=RETRY_LIMIT&result=RETRY&skip=0
* Infra comments
* <dpawlik> chandankumar, abregman: so tl;dr we have some issue with new zuul that is running inside the containers. We will let you know when it's fixed

* RDO promotion
* OVB Jobs are failing with retry_limit
* Error: Resource CREATE failed: WaitConditionTimeout: resources.baremetal_env.resources.bmc.resources.bmc_wait_condition: 0 of 1 received
* check Vexxhost ticket 362257
* https://logserver.rdoproject.org/openstack-regular/opendev.org/openstack/tripleo-ci/master/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/3fdf300/job-output.txt
* https://logserver.rdoproject.org/69/40169/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/e329db6/job-output.txt
* https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-rbac-wallaby/805df92/job-output.txt
* https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/8776e1c/job-output.txt
* https://logserver.rdoproject.org/28/41128/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/ae0b05e/job-output.txt
* https://review.rdoproject.org/zuul/builds?result=RETRY
* Any jobs which failed < 30 mins

* RDO C9 Wallaby @jm1
* Failed jobs:
- [periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby/37e3b1c):
[`TASK [os_tempest : Executing python-tempestconf]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby/37e3b1c/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz) fails with:

            2022-04-11 07:54:53.127 224093 INFO tempest.lib.common.rest_client [-] Request (main): 500 GET http://10.0.0.5:9292/v2/images 0.014s
            2022-04-11 07:54:53.127 224093 WARNING tempest.lib.common.rest_client [-] status >= 400 response with empty body
            2022-04-11 07:54:53.128 224093 CRITICAL tempest [-] Unhandled error: tempest.lib.exceptions.ServerFault: Got server fault
            Details: b''

We will ignore this bug for now because job is not in criteria [upstream](https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-9/wallaby.yaml)
and [downstream](https://code.engineering.redhat.com/gerrit/gitweb?p=tripleo-environments.git;a=blob;f=ci-scripts/dlrnapi_promoter/config_environments/rdo/RedHat-9/rhos-17.yaml;h=cc891cba4e78d681ca31e2b142f66a2083bdb644;hb=refs/heads/master).
- [periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby/ff06ae3):
Reran successfully, see below.
- [periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/4f75863):
Reran successfully, see below.
- [periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/381dca1):
TODO
- [periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby/6ab6afd):
Reran successfully, see below.
* Re-ran jobs successfully: https://review.rdoproject.org/r/c/testproject/+/41794

* RDO C8 Wallaby @jm1
* ~~Failed jobs:~~
- [periodic-tripleo-ci-build-containers-ubi-8-push-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/0a10a5a):
failed because the system was slower than usual so it took longer than usual. a lot of io-intensive steps such as installing software
or creating files took more time. a good indicator is `TASK [configure-swap : Create swap backing file]`:
`76.9834 s, 112 MB/s` in successful job vs `280.248 s, 30.7 MB/s` in failed job. Rerunning "fixed" this issue.
- [periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby/ee17883):
timed out in `opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildimages/run-v3.yaml` when running
`TASK [oooci-build-images : Run build-images.sh]` but `logs/script_build-err.log` shows no error.
Reran without errors, so probably timeout is too low with 5400 seconds.
- [periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-wallaby/1ca9c44):
same as with `periodic-tripleo-centos-8-buildimage-overcloud-full-wallaby`
* Re-ran jobs successfully: https://review.rdoproject.org/r/c/testproject/+/41780

* Failed jobs:
- [periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-wallaby/c12b852)
- [periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-rbac-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-rbac-wallaby/727d22a)
- [periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/94dd412)
- [periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby/af7b56f)
- [periodic-tripleo-ci-centos-8-scenario003-standalone-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario003-standalone-wallaby/1a8b277)
- [periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario004-standalone-wallaby/c0f28f4)
- [periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby/5a167f5)
- [periodic-tripleo-ci-centos-8-standalone-scenario001-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario001-skipped-tempest-wallaby/54a4a6e)
- [periodic-tripleo-ci-centos-8-standalone-scenario002-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario002-skipped-tempest-wallaby/4b947d2)
- [periodic-tripleo-ci-centos-8-standalone-scenario003-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario003-skipped-tempest-wallaby/ac022d5)
- [periodic-tripleo-ci-centos-8-standalone-scenario004-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario004-skipped-tempest-wallaby/6126009)
- [periodic-tripleo-ci-centos-8-standalone-scenario007-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario007-skipped-tempest-wallaby/74cdef4)
- [periodic-tripleo-ci-centos-8-standalone-scenario010-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-scenario010-skipped-tempest-wallaby/857d7da)
- [periodic-tripleo-ci-centos-8-standalone-skipped-tempest-wallaby](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-skipped-tempest-wallaby/3c0c8c1)
* Re-running jobs: https://review.rdoproject.org/r/c/testproject/+/41808

* RDO C8 train
* Promoted today
* Re-add kvm in criteria: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41238

## 10th Apr, 2022
* downstream
* all queues stalled over weekend: https://one.redhat.com/legacy/outages/list/details/625085eb293d9f00129e2621
* trying to clear what stuck
* RDO promotions
* master and train will both miss the kvm job
* https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=4c914a20a73d27db041581e38c5e66c6 is a good hash to promote - missing the kvm job ...
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41764 Temp revmove kvm=internal job from train c8 (pls merge if downstream is not up and running to rekick this job and get a kvm run)
* Merged

## 08th Apr, 2022
* gate
* https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-containers-multinode&pipeline=gate&skip=0 few failures on tempest
* on recheck seems to be working fine
~~* Master Promotion
* Re-running failed image build: https://review.rdoproject.org/r/c/testproject/+/41678~~
* RHOS-16.2 promotion
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/ffc3ac9/logs/rdoci-hp-01.v100.rdoci.lab.eng.rdu2.redhat.com/home/zuul/_home_zuul_repo_setup.sh.log
* failed due to mirror issue
* Re-running here: https://code.engineering.redhat.com/gerrit/c/testproject/+/400848
* https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/326/testReport/junit/tempest.scenario.test_network_basic_ops/TestNetworkBasicOps/test_update_instance_port_admin_state_compute_id_f5dfcc22_45fd_409f_954c_5bd500d7890b_network_slow_/ - tempest failure

* RHOS-17 on RHEL-8
* Error initializing source docker://registry.access.redhat.com/ubi8/pause:latest during overcloud deploy
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-17/1071a9d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-17/e9f3cb7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
* Local pull working fine
* Re-run: https://code.engineering.redhat.com/gerrit/c/testproject/+/400864
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17/77c1d5d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
*
* RHOS-17 on RHEL-9
* NOTE: will trigger the whole line once https://code.engineering.redhat.com/gerrit/c/testproject/+/400881 finishes
* ERROR: This release of Red Hat Ceph Storage does not allow installing for rgw multisite configuration, please see\nthe following knowledge base article for additional details
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17/af64977/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-ul76kwrz/cephadm/cephadm_command.log
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario004-standalone-rhos-17/c199487/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-k5tsq1jy/cephadm/cephadm_command.log
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17/ef620a2/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-xlns93pz/cephadm/cephadm_command.log
* Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2072665
* Fix: https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/400430/
* Need to promote tripleo component
* https://code.engineering.redhat.com/gerrit/c/testproject/+/400881

* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17/5477689/logs/undercloud/var/log/tempest/tempestconf.log.txt.gz
* Unhandled error: tempest.lib.exceptions.ServerFault: Got server fault
Details: b''
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17/efe4757/logs/undercloud/var/log/extra/errors.txt.gz
* Node provision failure