# Ruck Rover 2022-08-26 to 2022-08-31

###### tags: `ruck_rover`
###### Next RR notes: https://hackmd.io/94uNoMlnQgegrgy1iXV1kQ
###### Previous RR notes: https://hackmd.io/SYI2QzfeR-KJs900GVEOFg

##### ruck: Pooja, rover: Jakob

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

---

## 2022-08-31

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* all OVB jobs are failing on downstream since this afternoon - [ovb-manage : Find out UUID of instance with metadata file]. Please work with rhos-ops on this tomorrow

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- C9 Master
+ last_promotion=~~2022-08-29~~ today
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/52bfaad)~~
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/bf02ccd)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`
* ~~[intermittent FAILURE: LP#1987092](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/b873bcb/logs/overcloud-controller-2/var/log/containers/stdouts/galera-bundle.log.txt.gz)~~
* ~~[intermittent FAILURE: `FATAL | Create containers managed by Podman for ...`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
* ~~[intermittent FAILURE: `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap`](https://logserver.rdoproject.org/56/44656/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/3f51153/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`
* [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/2bb610b)
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
* [intermittent FAILURE: `FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: apr-util-openssl-1.6.1-20.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/56/44656/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/d61ea12/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/7/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/c4f8bcc/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/a1b48bb)~~
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/099130e)~~
+ ~~[TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/23/44323/56/check-rdo/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/5735af0)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`
* ~~[intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
* ~~[intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
+ [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656)

- C9 Wallaby
+ last_promotion=2022-08-29
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`
* ~~[intermittent FAILURE: `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
* ~~[intermittent POST_FAILURE: `FATAL | Clean up legacy Cinder keystone catalog entries` in overcloud deploy](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/67daa1f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
* ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/7d78527/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
+ ~~[TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/25/44325/242/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/4c5dcb1)~~
+ [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/ca39a39)
+ `periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby`
* [intermittent POST_FAILURE: `TASK [modify-image : Convert image]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby/7c2b880/job-output.txt)
+ `periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby`
* ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/a9d3647/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
* ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/982621a/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`
* ~~[intermittent FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/165bba5/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
* ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/fa9adf0/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`
* [intermittent FAILURE: `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/05eba93/job-output.txt)
* [intermittent FAILURE: `FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: mod_http2-1.15.19-2.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/45dc849/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
* [intermittent POST_FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/61/44661/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/0fa84fe/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/61/44661/7/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/a30ae4b/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-wallaby/32b1475)~~
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`
* ~~[intermittent FAILURE: `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/2edc93d/job-output.txt)~~
* ~~[intermittent FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/c19a84b/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
+ [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44661)
+ ~~[reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426493)~~

- C8 Wallaby
+ last_promotion=2022-08-30

- C8 Train
+ last_promotion=2022-08-30
+ ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/55a541f)~~
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`
* [intermittent FAILURE: `TASK [os_tempest : Create router]` failed](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/356c192/job-output.txt)
* [intermittent FAILURE: `Wait for containers to start for step 3 using paunch | overcloud-controller-0 | ... `](https://logserver.rdoproject.org/62/44662/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/7d2bf7c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`
* [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/19bf9be/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* [intermittent POST_FAILURE: `FATAL | Check Keystone role status | undercloud`](https://logserver.rdoproject.org/62/44662/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/5156671/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)

- OSP 16.2
+ last_promotion=2022-08-29
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/5b5e8bd)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/5193db4)
+ [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426819)

- OSP17
+ last_promotion=2022-08-28
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/f202039)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17/27e91e8)
+ [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426820)

- OSP17.1
+ last_promotion=2022-08-29
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-full-tempest-scenario-rhos-17.1/1a646db)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1/d461369)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17.1/66b2b3e)
+ [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426821)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- baremetal
+ last_promotion=2022-08-30
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master`
* [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/31ea938/logs/undercloud/var/log/tempest/stestr_results.html.gz)
* [intermittent FAILURE: mysqld failed to start](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/e4a7854/logs/overcloud-controller-0/var/log/containers/mysql/mysqld.log.txt.gz)
- clients
+ last_promotion=2022-08-26
+ ~~[TODO](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/2e80424)~~
- common
+ last_promotion=2022-08-31 (nevermind...)
+ [TODO](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/d834247)
- network
+ last_promotion=2022-08-29
+ [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/a5d1d89)
+ `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master`
* ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/a6fdff4/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
* ~~[intermittent FAILURE: other tempest failures](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/e8c299a/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
- tempest
+ last_promotion=2022-08-28
+ ~~[TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tempest-master/4267ca1)~~
+ ~~[TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-master/c1909e9)~~
- tripleo
+ last_promotion=2022-08-20
+ `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
* [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
+ [legit FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/f523bc1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
+ [legit FAILURE in `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master/5807435/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
+ [legit FAILURE in `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation/ca2377f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
+ [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-standalone-tripleo-master-validation`: tempest failures](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master-validation/a7574af/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ [legit FAILURE in `periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master/dece2a1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
- baremetal
+ last_promotion=2022-08-17
+ [TODO](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/50592e8)
- common
+ last_promotion=2022-08-10
+ no new content?
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
- tripleo
+ last_promotion=2022-08-28
+ [TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/9f67ab2)
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
- network
+ last_promotion=2022-08-28
+ [TODO](https://logserver.rdoproject.org/90/44690/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/5e0e876)
- [reruns 2](https://review.rdoproject.org/r/c/testproject/+/44690)

* [RHEL9 OSP17-1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline?orgId=1):
- baremetal
+ last_promotion=2022-08-19
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-baremetal-rhos-17.1/541febe)
- cinder
+ last_promotion=2022-08-19
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-cinder-rhos-17.1/28fd70a)
- cloudops
+ last_promotion=2022-08-31
+ [TODO](https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_component-cloudops-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/6/)
- common
+ last_promotion=2022-08-11
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-common-rhos-17.1/d67297a)
- compute
+ last_promotion=2022-08-30
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-compute-rhos-17.1/fd9039e)
- glance
+ last_promotion=2022-07-30
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-glance-rhos-17.1/9dde08a)
- manila
+ last_promotion=2022-08-20
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-manila-rhos-17.1/355bd24)
- network
+ last_promotion=2022-08-30
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-network-rhos-17.1/351ed42)
- octavia
+ last_promotion=2022-08-25
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-octavia-rhos-17.1/a6bcdbb)
- security
+ last_promotion=2022-08-26
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-security-rhos-17.1/4c78ed5)
- swift
+ last_promotion=2022-07-29
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-swift-rhos-17.1/467dc27)
- ui
+ last_promotion=2022-08-18
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-ui-rhos-17.1/8ec1c3a)
- validation
+ last_promotion=2022-08-30
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-validation-rhos-17.1/b84cef9)

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
- baremetal
+ last_promotion=2022-07-29
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/e45aa41)
- tripleo
+ last_promotion=2022-08-25
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17/ff84e03)
- [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426845)

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
- network
+ last_promotion=2022-08-16
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2/cc00320)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-16.2/ff3af33)
+ https://code.engineering.redhat.com/gerrit/c/testproject/+/418658 rerun on network
- octavia
+ last_promotion=2022-08-02
+ no new content?
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-octavia-rhos-17/ab09bfd)
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-cloudops-rhos-16.2/0fb6eae)
- ui
+ last_promotion=2022-05-20
+ no new content?
+ [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/58/398758/59/check/periodic-tripleo-ci-rhel-8-scenario007-standalone-network-rhos-16.2/f15fb1a)
- [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426848)

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1960355
- > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
- affecting OSP17.1 as well

* https://bugs.launchpad.net/tripleo/+bug/1988347 :warning: **BLOCKER** :warning:
* fails on master and wallaby
* fs064

* https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning:
- jobs fail on `Check Keystone public endpoint status` or in other places
- failures are probably symptoms of an underlying galera/database issue (insufficient resilience of galera/db?)
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugs.launchpad.net/tripleo/+bug/1987632
- > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
- [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
- atm only affecting check jobs
- might be an issue with host virt environments, i.e. the cloud provider

* https://bugs.launchpad.net/tripleo/+bug/1988053 :warning: **BLOCKER** :warning:
- > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
- affecting c9 master tripleo component
- [Takashi Kajinami is working on fixes](https://review.opendev.org/q/topic:bug%252F1988053)

* https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning:
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)
- [workaround (specifying `cpu_models` in libvirt)](
https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is
[currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/)

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
- [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
- `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
- `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)

### Fixed

---

## 2022-08-30

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- C9 Master
+ last_promotion=2022-08-29
+ ~~[rerun 1](https://review.rdoproject.org/r/c/testproject/+/44656)~~ passed
+ [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)
+ [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/e31bce2/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
+ [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/63a7850/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
+ [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/dfca39a/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log) due to https://bugzilla.redhat.com/2105408
- C9 Wallaby
+ last_promotion=2022-08-29
+ [rerun 1](https://review.rdoproject.org/r/c/testproject/+/44661)
+ ~~[intermittent (?) POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby` due to intermittent error in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)~~
+ [intermittent (?) POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby` due to `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)
+ [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`: tempest failures](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/19a3cfd/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby` on `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`: tempest failures](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/undercloud/var/log/tempest/stestr_results.html.gz) because [keystone `Lost connection to MySQL server during query`](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz) when [mysql container restarted or crashed](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/overcloud-controller-2/var/log/containers/mysql/mysqld.log.txt.gz)
+ ~~[intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby` in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)~~
+ [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426493)
* [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/93/426493/1/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/9e219d4/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
- C8 Wallaby
+ last_promotion=2022-08-29
- C8 Train
+ last_promotion=2022-08-30
- OSP 16.2
+ last_promotion=2022-08-29
+ [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230/74#message-902a56f8e6a5d70fd09ef84d506bb01ddf6a4a0c)
+ periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2 ~~failed with set of tempest tests~~ got green run
+ [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
+ periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2 failed with diffrent error everytime in last 3 runs.
- OSP17
+ last_promotion=2022-08-28
+ [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/209874)
+ periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17 random failures everytime failing with retry_limit
+ sc10 standalone failed with tempest failues with one diff from tempest tests
- OSP17.1
+ last_promotion=2022-08-29 (yesterday)
+ up to date
+ periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 tempest tests failure (no success till date) - added back to criteria and also failed tests got skipped (https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/855191/2 and https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/426532)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- baremetal
+ last_promotion=2022-08-26
+ ~~[overcloud deploy failed, probably intermittent](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/e6dc983/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
- clients
+ last_promotion=2022-08-26
+ ~~[tempest failed](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-clients-master/9029041/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~ and [passed in rerun](https://review.rdoproject.org/zuul/build/ea57d7cc9f2748fc866b59bac1d7f372)
+ [tempest failed on `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master` failed on rerun but on different tests](https://logserver.rdoproject.org/57/44657/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/035508b/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- common
+ last_promotion=2022-08-29
+ [probably intermittent repo mirror sync issue](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/5325a12/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
- network
+ last_promotion=2022-08-29
+ [probably intermittent](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/05f7d79/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- security
+ last_promotion=2022-08-28
+ [intermittent failure `'arecord' not found.` (see below)](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
+ [intermittent failure `'arecord' not found.` in rerun again](https://logserver.rdoproject.org/57/44657/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/3412a49/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- tempest
+ last_promotion=2022-08-28
+ [tempest failure](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master/2809c9f/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ [TIMED_OUT in rerun](https://review.rdoproject.org/zuul/build/19cc9355efb94c40ac44eaf97a31763b)
- tripleo
+ last_promotion=2022-08-20
+ LP#1988053
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44657) (again)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
- baremetal
+ last_promotion=2022-08-17
+ LP#1987092 in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`
+ [rerun of `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby` failed on another intermittent error](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- common
+ last_promotion=2022-08-10
+ no new content
- security
+ last_promotion=2022-08-25
+ [overcloud deploy failed in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`, probably intermittent](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/af73631/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
+ [rerun of `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby` failed on another intermittent error](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- tripleo
+ last_promotion=2022-08-27
+ no new content?
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44658) (again)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
- all **GREEN**😱

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
- network
+ last_promotion=2022-08-28
+ [probably intermittent](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/bea6760/job-output.txt)
- [reruns 1](https://code.engineering.redhat.com/gerrit/c/testproject/+/426497)
+ [intermittent FAILURE in `periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/97/426497/1/check/periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train/12a00c1/logs/undercloud/var/log/tempest/stestr_results.html)
- [reruns 2](https://review.rdoproject.org/r/c/testproject/+/44690)
+ [intermittent FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train`: download failed](https://logserver.rdoproject.org/90/44690/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/55f2554/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
- tripleo [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/266024)
periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17
- pipeline_component-tripleo-pcci-17_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
* up to date all components

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987092
- > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
- job is in retry state, before closing we need to have the job stable
- :warning: BLOCKER :warning:
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugs.launchpad.net/tripleo/+bug/1987632
- > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
- [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
- check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- Douglas is still investigating
- :warning: BLOCKER :warning:
- blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)

* https://bugs.launchpad.net/tripleo/+bug/1988053
- > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
- affecting c9 master tripleo component
- :warning: BLOCKER :warning:
- [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855106)

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](
https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
- [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
- `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
- `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1987641
- > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
- We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
- ~~[fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)~~
- [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814)
- c9 master promoted on 2022-08-29 with the fix above applied (`dlrn_hash is: 6faf5854d8ed01f023b9513ffbc25ef3`)

* https://bugs.launchpad.net/tripleo/+bug/1986755
- > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
- :warning: BLOCKER :warning:
- Xek proposed: https://review.opendev.org/853478
* Merged, rechicked security component
* It might need to be backported to fix wallaby too
- ipa job running in check itself - need to work with security team to continue debug/fixing.
- Wallaby component was Force promoted; We need to watch integration lines
- Once integration line has promoted we can close this bug and cix card
- > wallaby promoted - closing out

---

## 2022-08-29

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- C9 Master ~~2022-08-26~~- 2022-08-29 (today)
+ [rerun 1](https://review.rdoproject.org/r/c/testproject/+/44656)
+ periodic-tripleo-ci-centos-9-scenario001-standalone-master fails on LP#1987641
+ [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)
- C9 Wallaby ~~2022-08-22~~ - 2022-08-29 (today)
+ [rerun rlandy](https://review.rdoproject.org/r/c/testproject/+/43883)
+ [rerun jm1](https://review.rdoproject.org/r/c/testproject/+/44661) (different hash)
- C8 Wallaby 2022-08-29 (today)
- C8 Train 2022-08-26 (friday)
+ [rerun 1 (again)](https://review.rdoproject.org/r/c/testproject/+/44662)
+ periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train fails on [intermittent error](https://bugs.launchpad.net/tripleo/+bug/1938987)
+ we can skip promote this - based on these two logs (two different tempest errors) [1](https://logserver.rdoproject.org/55/36255/112/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/835fb53/logs/undercloud/var/log/tempest/stestr_results.html.gz) and [2](https://logserver.rdoproject.org/62/44662/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/fe2c998/logs/undercloud/var/log/tempest/stestr_results.html.gz)
+ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44689 Temp skip fs035 for train - **PLEASE REVERT after promo**
+ [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426435)
- OSP 16.2 ~~2022-08-25~~ - 2022-08-29 (today)
+ [buildset](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/76a8b211ea8041118b64ec7b61ec5465)
+ [reruns 1](https://code.engineering.redhat.com/gerrit/c/testproject/+/286307)
+ [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
- OSP17 2022-08-28
- OSP17.1 ~~2022-08-25~~ - 2022-08-29 (today)
+ [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
+ [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/418658)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- Red:
+ baremetal (last_promotion=2022-08-26, probably intermittent due to [different](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/32bbb4c/job-output.txt) [errors](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/8e3fd34/job-output.txt) with same dlrn_hash)
+ cinder (last_promotion=2022-08-21, LP#1987641)
+ cloudops (last_promotion=2022-08-23, LP#1987641)
+ common (last_promotion=2022-08-23, LP#1987641)
+ glance (last_promotion=2022-08-16, LP#1987641)
+ network (last_promotion=2022-08-23, probably intermittent due to [different](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/512fd2a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) [errors](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/d288117/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) with same dlrn_hash)
+ tripleo (last_promotion=2022-08-20, new bug LP#1988053)
+ [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):

* [RHEL9 OSP17.1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=1):

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987641
- > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
- We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
- ~~[fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)~~
- [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814)

* https://bugs.launchpad.net/tripleo/+bug/1987092
- > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
- job is in retry state, before closing we need to have the job stable
- :warning: BLOCKER :warning:
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- ask Jiri!

* https://bugs.launchpad.net/tripleo/+bug/1987632
- > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
- [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
- check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- Douglas is still investigating

* https://bugs.launchpad.net/tripleo/+bug/1986755
- > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
- :warning: BLOCKER :warning:
- Xek proposed: https://review.opendev.org/853478
* Merged, rechicked security component
* It might need to be backported to fix wallaby too
- ipa job running in check itself - need to work with security team to continue debug/fixing.
- Wallaby component was Force promoted; We need to watch integration lines
- Once integration line has promoted we can close this bug and cix card

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

* https://bugs.launchpad.net/tripleo/+bug/1988053
- > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
- affecting c9 master tripleo component
- no CIX card

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

### Fixed

* build containers failing for osp16.2 and osp17, Reported today but was just hitch : https://bugzilla.redhat.com/show_bug.cgi?id=2122167
Got green run on latest runs.

---

## 2022-08-26

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
- c9 master 2022-08-26 [reruns](https://review.rdoproject.org/r/c/testproject/+/44656)
- c9 wallaby 2022-08-22 [reruns](https://review.rdoproject.org/r/c/testproject/+/44661)
* https://review.rdoproject.org/r/c/testproject/+/43883 - chasing hash 4e233ad84bc9f1a0788729d4e0cc279d
passed periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby and periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby
- c8 wallaby 2022-08-22 (no reruns suggested by rr tool)
* (skip fs001 promoted - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44664 - was only missing 1 tempest test)
* missed promoting ^^ could not get patch through ... chasing 5091f496614fa524adceaf94f3d3f5b1 in https://review.rdoproject.org/r/c/testproject/+/36254
- c8 train 2022-08-26 [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)
- osp16.2 2022-08-25 (yesterday) [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
- osp17 2022-08-25 (yesterday)
- osp17.1 2022-08-25 (yesterday)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
- clients (4), cloudops (4), baremetal (4), ui (2), common (3), security (4), tripleo (4), compute (4), network (3) outdated
- Red:
+ cinder (LP#1987641)
+ cloudops (LP#1987641)
+ common (LP#1987641 + [probably intermittent error](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/55ffe3e/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz))
+ glance (LP#1987641)
+ network ([tempest failed, maybe intermittent error?](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/042c2ce/logs/undercloud/var/log/tempest/stestr_results.html.gz) + [another intermittent failure?](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/698c90f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz))
+ tripleo ([probably legit error](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/0dd495f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz) since all jobs fail on the [same](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/4158658/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) [error](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master/b74f7e5/logs/undercloud/home/zuul/undercloud_install.log.txt.gz))
+ ui (probably intermittent due to [timeout in tempest test](https://logserver.rdoproject.org/openstack-component-ui/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-ui-master/060d45b/logs/undercloud/var/log/tempest/stestr_results.html.gz))
+ validation ([maybe legit, maybe intermittent](https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/201dd4b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
- Yellow: compute
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
- octavia (4), security (5), network (3), tripleo (4) outdated
- Red:
+ clients ([probably legit haproxy error during overcloud deploy](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-wallaby/ac0612f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
+ network ([probably intermittent, timeout in tempest](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-network-wallaby/5a3e879/logs/undercloud/var/log/tempest/stestr_results.html.gz))
+ security ([`freeipa-setup : Run the tripleo-ipa preparation script`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/f28698e/job-output.txt), not related to LP#1986755?)
+ tripleo ([container keystone_bootstrap failed](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/8a6de3e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
- tripleo (4), common (4), octavia (2) outdated
- Red:
+ network ([tempest failures (timeout)](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/223574d/logs/undercloud/var/log/tempest/stestr_results.html.gz))
- [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
- compute (6), tripleo (4) outdated
- Red:
+ network ([tempest failures (timeout)](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/b927d28/logs/undercloud/var/log/tempest/stestr_results.html))

* [RHEL9 OSP17.1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=1):
* up to date : no promotion blocker
* periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 failing since

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
* rerunning failed jobs: build history is good https://code.engineering.redhat.com/gerrit/c/testproject/+/286307
* all components are up to date

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
* rerunning failed jobs as build history is good
https://code.engineering.redhat.com/gerrit/c/testproject/+/276230
* all components are up to date

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987641
- > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
- We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
- [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)

* https://bugs.launchpad.net/tripleo/+bug/1987092
- > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
- job is in retry state, before closing we need to have the job stable
- :warning: BLOCKER :warning:
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
- ask Jiri!

* https://bugs.launchpad.net/tripleo/+bug/1987632
- > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
- [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
- [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
- check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
- > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
- Douglas is still investigating

* https://bugs.launchpad.net/tripleo/+bug/1986755
- > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
- :warning: BLOCKER :warning:
- Xek proposed: https://review.opendev.org/853478
* Merged, rechicked security component
* It might need to be backported to fix wallaby too
- ipa job running in check itself - need to work with security team to continue debug/fixing.
- Wallaby component was Force promoted; We need to watch integration lines
- Once integration line has promoted we can close this bug and cix card

* https://bugs.launchpad.net/tripleo/+bug/1987323
- > FIPS + Manila + Ceph Quincy
- > ... known issue on the ceph side, ... already merged in master/quincy ...
> 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane `Waiting on external dependencies`
- Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1983817
- > periodic integration all branches RETRY Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org
- [That should be resolved based on updates in the VH support ticket, please review job runs since Aug 23 1400 UTC yesterday](https://bugs.launchpad.net/tripleo/+bug/1983817/comments/16)

---