# Ruck Rover 2022-09-09 to 2022-09-12

###### tags: `ruck_rover`
###### Next RR notes: https://hackmd.io/dKeK6zo9R66heikGyCb4NA
###### Previous RR notes: https://hackmd.io/0RguTgCAQsiZz_SnF5209A

##### ruck: Bhagyashris, rover: Jakob

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

---

## 2022-09-12

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- C9 Master
+ last_promotion: 2022-09-09
+ [hash `bb8b8889ee54e27a9b34630694628bf3`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=bb8b8889ee54e27a9b34630694628bf3)
* ~~`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`~~:
- ~~[LP#1989197](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/348bd9a/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
- ~~[LP#1989197](https://logserver.rdoproject.org/56/44656/16/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/d891202/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
* ~~`periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master`~~ (not in criteria):
- [~~LP#2105408?~~ intermittent failure?](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master/6e4d4e0/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log.txt.gz)
* ~~`periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master`~~ (not in criteria):
- [`cannot install the best update candidate for package gnutls-utils-3.7.3-10.el9.x86_64`](https://sf.hosted.upshift.rdu2.redhat.com/logs/42/44742/18/check-rdo/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/72e9a0a/job-output.txt)
* will be fixed by [chandan's revert](https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`:
- [intermittent failure `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/bf5d26f/job-output.txt)
- [RETRY_LIMIT in `[configure-mirrors : Update yum/dnf cache]`](https://logserver.rdoproject.org/56/44656/16/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/a7c2d28/job-output.txt)
- [RETRY_LIMIT in `[configure-mirrors : Update yum/dnf cache]`](https://logserver.rdoproject.org/56/44656/16/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/fd1567e/job-output.txt)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`:
- [other tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/7438cdc/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- [overcloud deploy failed with `Error: duplicate constraint already exists` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/56/44656/16/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/88570ac/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`:
- [LP#2105408](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/91/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/9f2b792/logs/undercloud/var/log/tempest/stestr_results.html)
- [LP#2105408](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/4c73b12/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log)
- [LP#2105408](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/4/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/75f9ed5/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`:
- [intermittent failure `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/b272efe/job-output.txt)
- [RETRY_LIMIT in `[configure-mirrors : Update yum/dnf cache]`](https://logserver.rdoproject.org/56/44656/16/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/aafecdd/job-output.txt)
* [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656)
* [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)

- C9 Wallaby
+ NOTE from #rhos-ops:
> rlandy: amoralej|off: dpawlik|off: hmm ... https://trunk.rdoproject.org/centos9-wallaby/component/network/component-ci-testing/commit.yaml has the wrong commit branch
> commit_branch: master
> should be stable/wallaby - should be safe to skip and promote here - see last few run with different tempest failures)
+ last_promotion: 2022-09-09
+ [hash `f6749f9cda54d017021ab56bf4ec0958`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=f6749f9cda54d017021ab56bf4ec0958)
* ~~`periodic-tripleo-ci-centos-9-skipped-scenario012-standalone-wallaby`~~ (not in criteria):
- [tempest failures](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-skipped-scenario012-standalone-wallaby/a1b2c7f/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* `periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby`:
- [overcloud deploy failed with `keystoneauth1.exceptions.discovery.DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate.` in `Clean up legacy Cinder keystone catalog entries`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/5ac716b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [node provisioning failed](https://logserver.rdoproject.org/61/44661/12/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/d624dc4/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
* `periodic-tripleo-ci-centos-9-8-multinode-mixed-os`:
- [other tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-8-multinode-mixed-os/4bcfcdc/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- [`[undercloud]: FAILED...Invalid response received from the delorean server`](https://logserver.rdoproject.org/61/44661/12/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os/9212099/job-output.txt)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`:
- [intermittent failure `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/7f85478/job-output.txt)
- [overcloud deploy failed in `FATAL | Ensure certificate requests` with no error message, wtf?](https://logserver.rdoproject.org/61/44661/12/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/5c566e1/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
* ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`~~:
- ~~[LP#1987092](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/b5abcac/logs/overcloud-controller-1/var/log/containers/stdouts/galera-bundle.log.txt.gz)~~
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`:
- [intermittent failure `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/7834334/job-output.txt)
- [overcloud deploy failed with unknown error](https://logserver.rdoproject.org/61/44661/12/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/1e2e94e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* ~~`periodic-tripleo-ci-centos-8-9-multinode-mixed-os`~~ (not in criteria):
- [other² tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-9-multinode-mixed-os/5f64adf/logs/undercloud/var/log/tempest/stestr_results.html)
* ~~`periodic-tripleo-ci-centos-9-skipped-standalone-wallaby`~~ (not in criteria):
- [other³ tempest failures](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-skipped-standalone-wallaby/e6088b4/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby`~~ (not in criteria):
- [other¼ tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/8037be8/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44661)
* [reruns 2](https://review.rdoproject.org/r/c/testproject/+/44899)
* [reruns 3](https://review.rdoproject.org/r/c/testproject/+/44900)
* [reruns 4](https://review.rdoproject.org/r/c/testproject/+/44901)
* [reruns 5](https://review.rdoproject.org/r/c/testproject/+/44902)
* [reruns 6](https://review.rdoproject.org/r/c/testproject/+/44903)
* [reruns 7](https://review.rdoproject.org/r/c/testproject/+/44904)
* [reruns 8](https://review.rdoproject.org/r/c/testproject/+/44905)
* [reruns 9](https://review.rdoproject.org/r/c/testproject/+/44906)
* [reruns 10](https://review.rdoproject.org/r/c/testproject/+/44907)
* [reruns 11](https://review.rdoproject.org/r/c/testproject/+/44908)
* [reruns 12](https://review.rdoproject.org/r/c/testproject/+/44909)
* TAKE THAT, WALLABY!!!
* NOTE: f6749f9cda54d017021ab56bf4ec0958 is missing on fs064 - can't get that to pass
trying for hash ea8639e875ec8d7ee7158bce2cd3fa2b now
https://review.rdoproject.org/r/c/testproject/+/31165 and https://review.rdoproject.org/r/c/testproject/+/36255

- C8 Wallaby
+ last_promotion: 2022-09-11
+ [hash `21d2e33704a2f14d611cf4edff39f70e`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=21d2e33704a2f14d611cf4edff39f70e)
* ~~`periodic-tripleo-ci-centos-8-skipped-standalone-wallaby`~~ (not in criteria):
- [`No container for this DLRN hash`](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-skipped-standalone-wallaby/a3ac814/logs/_No_container_for_this_DLRN_hash.log)

- C8 Train
+ last_promotion: 2022-09-09
+ [hash `489d88a4b22eb070acc39b218844ac82`](https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=489d88a4b22eb070acc39b218844ac82)
* ~~`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-train`~~ (not in criteria):
- [tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/13/44713/10/check-rdo/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-train/45f6b89/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train`:
- [intermittent (?) failure in `TASK [freeipa-setup : Deploy FreeIPA]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/1d3008b/job-output.txt)
- [overcloud deploy failed with `WSREP replication failed. Check your wsrep connection state and retry the query`](https://logserver.rdoproject.org/90/44690/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/2798945/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* ~~`periodic-tripleo-ci-centos-8-skipped-scenario012-standalone-train`~~ (not in criteria):
- [tempest rampage](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-skipped-scenario012-standalone-train/fa9b5d2/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* ~~`periodic-tripleo-ci-centos-8-skipped-standalone-train`~~ (not in criteria):
- [tempest destructiod](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-skipped-standalone-train/4e92533/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* ~~`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train`~~:
- ~~[tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/3ebdd17/logs/undercloud/var/log/tempest/stestr_results.html)~~
* [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)

- RHOS-17.1 on RHEL-9 **Refer to Bhagyashris' notes below**

- RHOS-17.1 on RHEL-8 **Refer to Bhagyashris' notes below**

- RHOS-17.0 on RHEL-9 **Refer to Bhagyashris' notes below**

- RHOS-16.2 on RHEL-8 **Refer to Bhagyashris' notes below**

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- clients
+ last_promotion: 2022-09-09
+ ~~`periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master`~~ (not in criteria):
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-api-1:***.el9.noarch requires openstack-heat-common = 1:***.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master/cd352ac/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
- common
+ last_promotion: 2022-09-11
+ ~~`periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master`~~ (not in criteria):
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-api-1:***.el9.noarch requires openstack-heat-common = 1:***.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/4ef4df0/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
- compute
+ last_promotion: 2022-09-07
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master`:
* [LP#1989197](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/2fd6041/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* [LP#1989197](https://logserver.rdoproject.org/57/44657/11/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/717f610/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- manila
+ last_promotion: 2022-09-09
+ ~~`periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master`~~:
* ~~[tempest failures](https://logserver.rdoproject.org/openstack-component-manila/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master/cd4d74b/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
- network
+ last_promotion: 2022-09-08
+ ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master`~~:
* ~~[overcloud deploy failed with `Error: duplicate constraint already exists` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/5672241/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
+ ~~`periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master`~~ (not in criteria):
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-api-1:***.el9.noarch requires openstack-heat-common = 1:***.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/486a62c/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master`:
* [LP#1989197](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/dc540aa/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* [LP#1989197](https://logserver.rdoproject.org/57/44657/11/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/b54f490/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- octavia
+ last_promotion: 2022-09-06
+ ~~`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-octavia-master`~~ (not in criteria):
* [LP#2105408](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/71/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-octavia-master/bbc6a88/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log)
- tripleo
+ last_promotion: 2022-09-08
+ ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master`~~:
* ~~[overcloud deploy failed with `Error: .../Tripleo::Pacemaker::Haproxy_with_vip.../ensure: change from 'absent' to 'present' failed` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/818fcb4/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
- common
+ last_promotion: 2022-09-10
+ ~~`periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby`~~ (not in criteria):
* [intermittent (?) failure in `TASK [install and run os-net-config]`: `ModuleNotFoundError: No module named 'oslo_concurrency'`](https://logserver.rdoproject.org/84/44684/7/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby/5fa2da2/job-output.txt)
- network
+ last_promotion: 2022-09-03
+ [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby):
* older runs:
- 2022-09-09: [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/58/44658/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/ea08fd8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- 2022-09-09: [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/31f80d8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- 2022-09-10: [overcloud deploy failed with `ERROR: Can't run container rabbitmq_bootstrap`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/0b473bf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* [overcloud deploy failed with `Error: error creating container storage: the container name "container-puppet-swift" is already in use by ***. You have to remove that container to be able to reuse that name: that name is already in use`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/7ed1819/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* [tempest failures](https://logserver.rdoproject.org/58/44658/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/8ada848/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- tripleo
+ last_promotion: 2022-09-06
+ ~~`periodic-tripleo-ci-centos-9-8-multinode-mixed-os-tripleo-wallaby`~~ (not in criteria):
* [intermittent (?) FAILURE in `TASK [install and run os-net-config]`: `nothing provides python3.9dist(pbr) >= 2 needed by os-net-config-***.el9.noarch`](https://logserver.rdoproject.org/84/44684/7/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os-tripleo-wallaby/4383f6f/job-output.txt)
+ ~~`periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-wallaby`~~:
* ~~[intermittent (?) FAILURE in `TASK [Install virtualenv]`: `Failed to download packages: python3-wheel-wheel-***.el9.noarch`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-wallaby/aba5598/job-output.txt)~~
+ ~~`periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby`~~:
* ~~[intermittent (?) FAILURE in `TASK [repo-setup : Create repo setup script]`: `Failed to connect to the host via ssh: kex_exchange_identification: read: Connection reset by peer`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby/9e2d128/job-output.txt)~~
+ ~~`periodic-tripleo-ci-centos-8-9-multinode-mixed-os-tripleo-wallaby`~~ (not in criteria):
* [intermittent (?) FAILURE in `TASK [install and run os-net-config]`: `nothing provides python3.9dist(pbr) >= 2 needed by os-net-config-***.el9.noarch`](https://logserver.rdoproject.org/84/44684/7/check/periodic-tripleo-ci-centos-8-9-multinode-mixed-os-tripleo-wallaby/cc1f140/job-output.txt)
+ ~~`periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby-validation`~~:
* ~~[overcloud deploy failed in `FATAL | Wait for puppet host configuration to finish` with `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby-validation/7018188/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
- all **GREEN** 😱
- ~~[reruns](https://review.rdoproject.org/r/c/testproject/+/44659)~~

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
- network
+ last_promotion: 2022-09-09
+ ~~`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train`~~ (not in criteria):
* [tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/5cd5938/logs/undercloud/var/log/tempest/stestr_results.html)
- ~~[reruns](https://review.rdoproject.org/r/c/testproject/+/44690)~~

* [RHOS-17.0 on RHEL-9](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
* **Refer to Bhagyashris' notes below**

* [RHOS-16.2 on RHEL-8](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
* **Refer to Bhagyashris' notes below**

### Downstream:
* rhos16.2 on rhel8:
* Promotion: 12-sept-2022
* blocked due to sc010 re-running jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/

* rhos17.1 on rhel8
* Promotion: 12-sept-2022
* rhos17 on rhel9:
* Promotion: 11-Sept-2022
* failed fs001 job so re-ran here : https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : **PASSED**
* rhos17.1 on rhel9:
* Promoted: 12-sept-2022
* blocked due full tempest api so re-ran here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : **PASSED**
* jenkins job not triggerd:
* Note : Pinged atila: - After discussion with atila we got to know that currently we don't have trigger jobs for 17.1 on rhel9 and 8 so i am adding trigger job for 17.1 on 9 and 8 here: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003
* ~~fix for trigger jekins component rhso17.1 on rhel9 jobs is here: https://code.engineering.redhat.com/gerrit/c/rhos-qe-jenkins/+/428056 ~~

### Downstream component:

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1960355
- > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
- affecting OSP17.1 as well

* https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning:
- Pacemaker performance causes intermittent galera issues in loaded CI env
- jobs fail on `Check Keystone public endpoint status` or in other places
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning:
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)
- [workaround (specifying `cpu_models` in libvirt)](
https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is
[currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/)

* https://bugs.launchpad.net/tripleo/+bug/1988514
- [Trello](https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re)
- On CIX board in lane `Waiting on external dependencies`

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

* https://bugs.launchpad.net/tripleo/+bug/1988810
- Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788

* https://bugs.launchpad.net/tripleo/+bug/1989197
- > Tempest test `neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON` fails on `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`

* https://bugs.launchpad.net/tripleo/+bug/1989341
- > Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch

* https://bugs.launchpad.net/tripleo/+bug/1989247
- > overcloud deploy fails with `sudo: unable to send audit message: Operation not permitted`
- > `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz)

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- > this has different root causes, watch out for log details!
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
- `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
- `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1988738
* Octavia ovn driver function create_vip_port wasn't updated with latest octavia changes
* [job passes since 2022-09-08](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-master)

---

## 2022-09-09

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- C9 Master:
+ last_promotion: 2022-08-31
+ [hash 3edefe9728ecab11e979d22a0d03263d](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=3edefe9728ecab11e979d22a0d03263d)
* `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`:
- [skiplist patch](https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/856764)
- [reruns with Depends-On skiplist patch](https://review.rdoproject.org/r/c/testproject/+/44656)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`:
- [patch to remove c9 master fs35 from promotion criteria](https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44863)
* [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656)
* [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)
+ [hash bb23ee0d771f83919a7fc55f41bfe6ad](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=bb23ee0d771f83919a7fc55f41bfe6ad)
* **WE DO NOT TRY TO PROMOTE THIS HASH!**
* **SEE PREVIOUS HASH ABOVE FOR TODAYS PROMOTION TARGET**
* `periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master`:
- ~~[overcloud deploy failed with `Error: 'haproxy-bundle' already exists` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/29/44829/4/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/fb38db8/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
- [overcloud deploy failed with `Create containers managed by Podman for ... Failed containers`](https://logserver.rdoproject.org/56/44656/13/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/948724b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* ~~`periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master`~~ (not in criteria):
- [failed in `TASK [repo-setup : Setup repos on live host]` with `package gnutls-utils-3.7.6-7.el9.x86_64 requires gnutls(x86-64) = 3.7.6-7.el9, but none of the providers can be installed`](https://sf.hosted.upshift.rdu2.redhat.com/logs/42/44742/12/check-rdo/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/0b350b7/job-output.txt)
- will be fixed by [chandan's revert](https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master`:
- ~~[tempest failure](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/2d35775/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
- [overcloud deploy failed with `Create containers managed by Podman for ... Failed containers`](https://logserver.rdoproject.org/56/44656/13/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/1fe781e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`:
- [LP#1989197](https://logserver.rdoproject.org/29/44829/4/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/8101527/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- [LP#1989197](https://logserver.rdoproject.org/56/44656/13/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/727e0e0/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* ~~`periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master`~~:
- ~~[other² tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/1021480/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
* ~~`periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master`~~ (not in criteria):
- ~~[other³ tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master/95f29a7/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`:
- ~~[overcloud deploy failed with `ERROR: Can't run container mysql_wait_bundle`](https://logserver.rdoproject.org/29/44829/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/ef5049f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
- [other¼ tempest failures](https://logserver.rdoproject.org/56/44656/13/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/82fb95d/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`:
- ~~[undercloud install failed in `FATAL | ensure apache is installed` with `Failed to download packages: apr-util-1.6.1-20.el9.x86_64`](https://logserver.rdoproject.org/29/44829/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/c4bca75/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)~~
- [tempest failures](https://logserver.rdoproject.org/56/44656/13/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/84ae0b6/logs/undercloud/var/log/tempest/stestr_results.html.gz)

- C9 Wallaby
+ last_promotion: 2022-09-05
+ [hash](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=a29104cf0a2a21bb5b863335b204ba1f)
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`:
* [intermittent FAILURE in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt)
* [intermittent FAILURE in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/61/44661/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/d269f17/job-output.txt)
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`:
* [overcloud deploy failed with `Error: duplicate constraint already exists` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/05be30f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
* [overcloud deploy failed with `Error: .../Tripleo::Pacemaker::Haproxy_with_vip.../ensure: change from 'absent' to 'present' failed` in `FATAL | Run init bundle puppet on the host for haproxy`](https://logserver.rdoproject.org/61/44661/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/18312d6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby`~~ (not in criteria):
* [intermittent FAILURE during overcloud deploy: `Create containers managed by Podman for ... Failed containers:`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/e510014/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ ~~`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`~~:
* ~~[intermittent FAILURE during overcloud node provisioning ](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/7c9467e/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
+ [reruns](https://review.rdoproject.org/r/c/testproject/+/44661)

- C8 Wallaby
+ last_promotion: 2022-09-07
+ [hash](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=b600fc85484a3e36e58d50642bc727b2)

- C8 Train
+ last_promotion: 2022-09-07
+ [hash](https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=4373585269a00ee0bafb01a7e4bf205b)
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`:
* [tempest failures](https://logserver.rdoproject.org/17/38017/28/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/64be90e/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)

- RHOS-17.1 on RHEL-9 **Refer to Bhagyashris' notes below**

- RHOS-17.1 on RHEL-8 **Refer to Bhagyashris' notes below**

- RHOS-17.0 on RHEL-9 **Refer to Bhagyashris' notes below**

- RHOS-16.2 on RHEL-8 **Refer to Bhagyashris' notes below**

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- clients
+ last_promotion: 2022-09-09
+ `periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master`:
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-monolith-1:16.0.1-0.20220817062359.d0e2ded.el9.noarch requires openstack-heat-common = 1:16.0.1-0.20220817062359.d0e2ded.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master/83c952c/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
- common
+ last_promotion: 2022-09-09
+ `periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master`:
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-api-1:16.0.1-0.20220817062359.d0e2ded.el9.noarch requires openstack-heat-common = 1:16.0.1-0.20220817062359.d0e2ded.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/b5cc2f0/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
- compute
+ last_promotion: 2022-09-07
+ `periodic-tripleo-ci-centos-9-scenario012-standalone-compute-master`:
* [tempest failures](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-compute-master/3c8a087/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master`:
* [other tempest failures](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/c1ad9b9/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- network
+ last_promotion: 2022-09-08
+ `periodic-tripleo-ci-centos-9-standalone-network-master`:
* [other² tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-network-master/b0bedab/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master`:
* [other³ tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/6e5bc31/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ `periodic-tripleo-ci-centos-9-containers-multinode-network-master`:
* [overcloud deploy failed in `FATAL | Check Keystone internal endpoint status`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-network-master/7c44280/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master`:
* [undercloud install failed in `TASK [undercloud-setup : Run the package installation script]` with `Problem: package openstack-heat-api-1:16.0.1-0.20220817062359.d0e2ded.el9.noarch requires openstack-heat-common = 1:16.0.1-0.20220817062359.d0e2ded.el9, but none of the providers can be installed`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/195315b/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master`:
* [other½ tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/59ef943/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- octavia
+ last_promotion: 2022-09-06
+ `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-octavia-master`:
* [BZ#2105408](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/44/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-octavia-master/09bcb5e/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log)
- security
+ last_promotion: 2022-09-05
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`:
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6146852/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- tripleo
+ last_promotion: 2022-09-08
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master`:
* [tempest failures](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/05c8d67/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ `periodic-tripleo-ci-centos-9-standalone-tripleo-master`:
* [undercloud deploy failed with `FATAL | Check Keystone public endpoint status`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/611652a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
- baremetal
+ last_promotion: 2022-09-01
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/e89614e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- common
+ last_promotion: 2022-09-08
+ `periodic-tripleo-ci-centos-9-scenario002-standalone-common-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario002-standalone-common-wallaby/8699ae8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-containers-multinode-common-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-common-wallaby/64ebdee/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-common-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-common-wallaby/2215d5f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-scenario001-standalone-common-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-common-wallaby/464029e/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-scenario004-standalone-common-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-common-wallaby/9f4230a/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby`:
* [standalone deploy failed with `Error running container image prepare: Modifying image ... failed`](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/c31cd34/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- compute
+ last_promotion: 2022-08-29
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compute-wallaby`:
* [overcloud node provisioning failed](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compute-wallaby/5ac6e30/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby`:
* [overcloud deploy failed with `Error: OCI runtime error: crun: fork: Cannot allocate memory`](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby/60e025b/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-wallaby/72454a5/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- network
+ last_promotion: 2022-09-03
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby`:
* [LP#1987092](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/ab31a2d/logs/overcloud-controller-0/var/log/containers/stdouts/galera-bundle.log.txt.gz)
- tripleo
+ last_promotion: 2022-09-06
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/0741054/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby-validation`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby-validation/1375c7b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-wallaby`:
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-wallaby/fe01aa2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-scenario004-standalone-tripleo-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-tripleo-wallaby/5594331/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby/c3b6f8f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-tripleo-wallaby`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-wallaby/422d974/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-9-standalone-tripleo-wallaby-validation`:
* [overcloud deploy failed with `Error: can only create exec sessions on running containers: container state improper`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-wallaby-validation/07aac25/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
- network
+ last_promotion: 2022-09-06
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby`:
* [overcloud node provisioning failed](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/c4e99e6/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
- tripleo
+ last_promotion: 2022-09-08
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`:
* [`Overcloud stack: FAILED. MAC conflict between hosts.`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/463af23/logs/_Overcloud_stack__FAILED__MAC_conflict_between_hosts.log)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
- compute
+ last_promotion: 2022-08-23
+ `periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-train`:
* [tempest failures](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-compute-train/c182da8/logs/undercloud/var/log/tempest/stestr_results.html)
- network
+ last_promotion: 2022-09-06
+ `periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train`:
* [similar tempest failures to compute job](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/9596aaf/logs/undercloud/var/log/tempest/stestr_results.html)
- tripleo
+ last_promotion: 2022-09-06
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train`:
* [overcloud deploy failed with `FATAL | Run NetworkConfig script | ... "async task did not complete within the requested time`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/68350a1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44690)

* [RHOS-17.0 on RHEL-9](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
* **Refer to Bhagyashris' notes below**

* [RHOS-16.2 on RHEL-8](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
* **Refer to Bhagyashris' notes below**

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1988738
* Octavia ovn driver function create_vip_port wasn't updated with latest octavia changes
- Waiting on master promotion

* https://bugs.launchpad.net/tripleo/+bug/1960355
- > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
- affecting OSP17.1 as well

* https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning:
- Pacemaker performance causes intermittent galera issues in loaded CI env
- jobs fail on `Check Keystone public endpoint status` or in other places
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning:
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)
- [workaround (specifying `cpu_models` in libvirt)](
https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is
[currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/)

* https://bugs.launchpad.net/tripleo/+bug/1988514
- [Trello](https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re)
- On CIX board in lane `Waiting on external dependencies`

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

* https://bugs.launchpad.net/tripleo/+bug/1988810
- Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788

* https://bugs.launchpad.net/tripleo/+bug/1989197
- > Tempest test `neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON` fails on `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz)

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
- `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
- `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1988347
- fails on c9 master and c9 wallaby
- fs064, fs39 and others
- intermittent failure aka when rerunning hard enough it will pass eventually!
- > `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6146852/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
- [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes
- not a legit failure: https://bugs.launchpad.net/tripleo/+bug/1988347/comments/5

### Downstream:
* rhos16.2 on rhel8:
* Promotion: 07-sept-2022
* rhos17.1 on rhel8
* Promotion: 08-sept-2022
* rhos17 on rhel9:
* Promotion: 08-Sept-2022
* rhos17.1 on rhel9:
* Promoted: 09-sept-2022

### Downstream component:

---

## 2022-09-08

Promotion Status
* Downstream Promotion Status
+ RHOS-16.2 - 07-09-2022
+ RHOS-17.0 on RHEL-9 - 04-09-2022
+ RHOS-17.1 on RHEL-9 - 07-09-2022
+ RHOS-17.1 on RHEL-8 - 07-09-2022

* Upstream Promotion Status
+ Master - 31-08-2022
+ Wallaby CS9 - 05-09-2022
+ Wallaby CS8 - 07-09-2022
+ Train - 07-09-2022

* Jobs removed to promote content

- for network master component - skipped periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master
- for wallaby cs8 - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby
- for cs9 wallaby we might remove fs064 job

Bugs:
* https://bugs.launchpad.net/tripleo/+bug/1988738
* Octavia ovn driver function create_vip_port wasn't updated with latest octavia changes
- Waiting on master promotion
* https://bugs.launchpad.net/tripleo/+bug/1988810
- Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788
* https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re

---