---
# System prepended metadata

title: Ruck Rover 2022-08-26 to 2022-08-31
tags: [ruck_rover]

---

# Ruck Rover 2022-08-26 to 2022-08-31

###### tags: `ruck_rover`
###### Next RR notes: https://hackmd.io/94uNoMlnQgegrgy1iXV1kQ
###### Previous RR notes: https://hackmd.io/SYI2QzfeR-KJs900GVEOFg

##### ruck: Pooja, rover: Jakob

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

---

## 2022-08-31

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and 
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* all OVB jobs are failing on downstream since this afternoon - [ovb-manage : Find out UUID of instance with metadata file]. Please work with rhos-ops on this tomorrow

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
    - C9 Master
        + last_promotion=~~2022-08-29~~ today
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/52bfaad)~~
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/bf02ccd)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`
            * ~~[intermittent FAILURE: LP#1987092](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/b873bcb/logs/overcloud-controller-2/var/log/containers/stdouts/galera-bundle.log.txt.gz)~~
            * ~~[intermittent FAILURE: `FATAL | Create containers managed by Podman for ...`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
            * ~~[intermittent FAILURE: `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap`](https://logserver.rdoproject.org/56/44656/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/3f51153/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`
            * [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/2bb610b)
            * [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
            * [intermittent FAILURE: `FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: apr-util-openssl-1.6.1-20.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/56/44656/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/d61ea12/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
            * [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/7/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/c4f8bcc/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/a1b48bb)~~
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/099130e)~~
        + ~~[TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/23/44323/56/check-rdo/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/5735af0)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`
            * ~~[intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
            * ~~[intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
        + [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656)

    - C9 Wallaby
        + last_promotion=2022-08-29
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`
            * ~~[intermittent FAILURE: `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
            * ~~[intermittent POST_FAILURE: `FATAL | Clean up legacy Cinder keystone catalog entries` in overcloud deploy](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/67daa1f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
            * ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/7d78527/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
        + ~~[TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/25/44325/242/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/4c5dcb1)~~
        + [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/ca39a39)
        + `periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby`
            * [intermittent POST_FAILURE: `TASK [modify-image : Convert image]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-rbac-wallaby/7c2b880/job-output.txt)
        + `periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby`
            * ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/a9d3647/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
            * ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/982621a/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`
            * ~~[intermittent FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/165bba5/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
            * ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/fa9adf0/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby` 
            * [intermittent FAILURE: `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/05eba93/job-output.txt)
            * [intermittent FAILURE: `FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: mod_http2-1.15.19-2.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/45dc849/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
            * [intermittent POST_FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/61/44661/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/0fa84fe/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
            * [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/61/44661/7/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/a30ae4b/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario012-standalone-wallaby/32b1475)~~
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`
            * ~~[intermittent FAILURE: `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/2edc93d/job-output.txt)~~
            * ~~[intermittent FAILURE: node hang during overcloud node provisioning](https://logserver.rdoproject.org/61/44661/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/c19a84b/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)~~
        + [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44661)
        + ~~[reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426493)~~

    - C8 Wallaby 
        + last_promotion=2022-08-30

    - C8 Train
        + last_promotion=2022-08-30
        + ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/55a541f)~~
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`
            * [intermittent FAILURE: `TASK [os_tempest : Create router]` failed](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/356c192/job-output.txt)
            * [intermittent FAILURE: `Wait for containers to start for step 3 using paunch | overcloud-controller-0 | ... `](https://logserver.rdoproject.org/62/44662/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/7d2bf7c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`
            * [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/19bf9be/logs/undercloud/var/log/tempest/stestr_results.html.gz)
            * [intermittent POST_FAILURE: `FATAL | Check Keystone role status | undercloud`](https://logserver.rdoproject.org/62/44662/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/5156671/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
        + [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)
    
    - OSP 16.2
        + last_promotion=2022-08-29
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/5b5e8bd)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2/5193db4)
        + [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426819)

    - OSP17
        + last_promotion=2022-08-28
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/f202039)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17/27e91e8)
        + [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426820)

    - OSP17.1
        + last_promotion=2022-08-29
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-full-tempest-scenario-rhos-17.1/1a646db)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1/d461369)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17.1/66b2b3e)
        + [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426821)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
    - baremetal 
        + last_promotion=2022-08-30
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master`
            * [intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/31ea938/logs/undercloud/var/log/tempest/stestr_results.html.gz)
            * [intermittent FAILURE: mysqld failed to start](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/e4a7854/logs/overcloud-controller-0/var/log/containers/mysql/mysqld.log.txt.gz)
    - clients 
        + last_promotion=2022-08-26
        + ~~[TODO](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/2e80424)~~
    - common 
        + last_promotion=2022-08-31 (nevermind...)
        + [TODO](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/d834247)
    - network 
        + last_promotion=2022-08-29
        + [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/a5d1d89)
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master`
            * ~~[intermittent FAILURE: tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/a6fdff4/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
            * ~~[intermittent FAILURE: other tempest failures](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/e8c299a/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~
    - tempest
        + last_promotion=2022-08-28
        + ~~[TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tempest-master/4267ca1)~~
        + ~~[TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-master/c1909e9)~~
    - tripleo
        + last_promotion=2022-08-20
        + `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`
            * [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
            * [intermittent FAILURE: 'arecord' not found.](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
        + [legit FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/f523bc1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
        + [legit FAILURE in `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master/5807435/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
        + [legit FAILURE in `periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-master-validation/ca2377f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
        + [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-standalone-tripleo-master-validation`: tempest failures](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master-validation/a7574af/logs/undercloud/var/log/tempest/stestr_results.html.gz)
        + [legit FAILURE in `periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master`: LP#1988053](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master/dece2a1/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
    - baremetal
        + last_promotion=2022-08-17
        + [TODO](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/50592e8)
    - common 
        + last_promotion=2022-08-10 
        + no new content?
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
    - tripleo 
        + last_promotion=2022-08-28
        + [TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/9f67ab2)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
    - network 
        + last_promotion=2022-08-28
        + [TODO](https://logserver.rdoproject.org/90/44690/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/5e0e876)
    - [reruns 2](https://review.rdoproject.org/r/c/testproject/+/44690)

* [RHEL9 OSP17-1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline?orgId=1):
    - baremetal
        + last_promotion=2022-08-19
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-baremetal-rhos-17.1/541febe)
    - cinder
        + last_promotion=2022-08-19
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-cinder-rhos-17.1/28fd70a)
    - cloudops 
        + last_promotion=2022-08-31
        + [TODO](https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_component-cloudops-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/6/)
    - common
        + last_promotion=2022-08-11
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-common-rhos-17.1/d67297a)
    - compute
        + last_promotion=2022-08-30
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-compute-rhos-17.1/fd9039e)
    - glance 
        + last_promotion=2022-07-30
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-glance-rhos-17.1/9dde08a)
    - manila 
        + last_promotion=2022-08-20
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-manila-rhos-17.1/355bd24)
    - network 
        + last_promotion=2022-08-30
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-network-rhos-17.1/351ed42)
    - octavia
        + last_promotion=2022-08-25
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-octavia-rhos-17.1/a6bcdbb)
    - security
        + last_promotion=2022-08-26
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-security-rhos-17.1/4c78ed5)
    - swift
        + last_promotion=2022-07-29
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-swift-rhos-17.1/467dc27)
    - ui
        + last_promotion=2022-08-18
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-ui-rhos-17.1/8ec1c3a)
    - validation
        + last_promotion=2022-08-30
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/46/426546/3/check/periodic-tripleo-ci-rhel-9-standalone-validation-rhos-17.1/b84cef9)

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
    - baremetal
        + last_promotion=2022-07-29
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-baremetal-rhos-17/e45aa41)
    - tripleo 
        + last_promotion=2022-08-25
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17/ff84e03)
    - [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426845)

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
    - network
        + last_promotion=2022-08-16
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2/cc00320)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-16.2/ff3af33)
        + https://code.engineering.redhat.com/gerrit/c/testproject/+/418658 rerun on network
    - octavia
        + last_promotion=2022-08-02
        + no new content?
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-octavia/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-octavia-rhos-17/ab09bfd)
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-cloudops-rhos-16.2/0fb6eae)
    - ui
        + last_promotion=2022-05-20
        + no new content?
        + [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/58/398758/59/check/periodic-tripleo-ci-rhel-8-scenario007-standalone-network-rhos-16.2/f15fb1a)
    - [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/426848)

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1960355
    - > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
    - affecting OSP17.1 as well

* https://bugs.launchpad.net/tripleo/+bug/1988347 :warning: **BLOCKER** :warning:
    * fails on master and wallaby
    * fs064 

* https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning:
    - jobs fail on `Check Keystone public endpoint status` or in other places
    - failures are probably symptoms of an underlying galera/database issue (insufficient resilience of galera/db?)
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
    - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugs.launchpad.net/tripleo/+bug/1987632
    - > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
    - [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
    - atm only affecting check jobs
    - might be an issue with host virt environments, i.e. the cloud provider

* https://bugs.launchpad.net/tripleo/+bug/1988053 :warning: **BLOCKER** :warning:
    - > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
    - affecting c9 master tripleo component
    - [Takashi Kajinami is working on fixes](https://review.opendev.org/q/topic:bug%252F1988053)

* https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning:
    - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
    - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
    - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
    - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)
    - [workaround (specifying `cpu_models` in libvirt)](
      https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is
      [currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/)

* https://bugs.launchpad.net/tripleo/+bug/1987323 
    - > FIPS + Manila + Ceph Quincy
    - > ... known issue on the ceph side, ... already merged in master/quincy ... 
      > 17.2.4 should solve this issue, so we'll check during the next promotion.
    - On CIX board in lane `Waiting on external dependencies`
    - Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616 
    - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
    - no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
    - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
    - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
    - Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
    - [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
    - `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
    - `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)

### Fixed

---

## 2022-08-30

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and 
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
    - C9 Master
        + last_promotion=2022-08-29
        + ~~[rerun 1](https://review.rdoproject.org/r/c/testproject/+/44656)~~ passed
        + [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)
            + [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/e31bce2/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
            + [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/63a7850/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
            + [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/2/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/dfca39a/logs/undercloud/var/log/containers/libvirt/qemu/instance-00000001.log) due to https://bugzilla.redhat.com/2105408
    - C9 Wallaby
        + last_promotion=2022-08-29
        + [rerun 1](https://review.rdoproject.org/r/c/testproject/+/44661)
            + ~~[intermittent (?) POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby` due to intermittent error in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)~~
            + [intermittent (?) POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby` due to `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)
            + [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`: tempest failures](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/19a3cfd/logs/undercloud/var/log/tempest/stestr_results.html.gz)
            + [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby` on `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
            + [intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`: tempest failures](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/undercloud/var/log/tempest/stestr_results.html.gz) because [keystone `Lost connection to MySQL server during query`](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz) when [mysql container restarted or crashed](https://logserver.rdoproject.org/61/44661/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/6dca3bb/logs/overcloud-controller-2/var/log/containers/mysql/mysqld.log.txt.gz)
            + ~~[intermittent (?) FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby` in `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)~~
        + [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426493)
            * [FAILURE in `periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/93/426493/1/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/9e219d4/logs/undercloud/var/log/tempest/stestr_results.html) due to https://bugzilla.redhat.com/2105408
    - C8 Wallaby 
        + last_promotion=2022-08-29
    - C8 Train
        + last_promotion=2022-08-30
    - OSP 16.2
        + last_promotion=2022-08-29
        + [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230/74#message-902a56f8e6a5d70fd09ef84d506bb01ddf6a4a0c)
        + periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2 ~~failed with set of tempest tests~~ got green run 
        + [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
        + periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2 failed with diffrent error everytime in last 3 runs.
    - OSP17
        + last_promotion=2022-08-28
        + [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/209874)
        + periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17 random failures everytime failing with retry_limit
        + sc10 standalone failed with tempest failues with one diff from tempest tests
    - OSP17.1
        + last_promotion=2022-08-29 (yesterday)
        + up to date
        + periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 tempest tests failure (no success till date) - added back to criteria and also failed tests got skipped (https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/855191/2 and https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/426532)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
    - baremetal
        + last_promotion=2022-08-26
        + ~~[overcloud deploy failed, probably intermittent](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/e6dc983/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
    - clients
        + last_promotion=2022-08-26
        + ~~[tempest failed](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-clients-master/9029041/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~ and [passed in rerun](https://review.rdoproject.org/zuul/build/ea57d7cc9f2748fc866b59bac1d7f372)
        + [tempest failed on `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master` failed on rerun but on different tests](https://logserver.rdoproject.org/57/44657/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/035508b/logs/undercloud/var/log/tempest/stestr_results.html.gz)
    - common
        + last_promotion=2022-08-29
        + [probably intermittent repo mirror sync issue](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/5325a12/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
    - network
        + last_promotion=2022-08-29
        + [probably intermittent](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/05f7d79/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
    - security 
        + last_promotion=2022-08-28
        + [intermittent failure `'arecord' not found.` (see below)](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
        + [intermittent failure `'arecord' not found.` in rerun again](https://logserver.rdoproject.org/57/44657/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/3412a49/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - tempest
        + last_promotion=2022-08-28
        + [tempest failure](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master/2809c9f/logs/undercloud/var/log/tempest/stestr_results.html.gz)
        + [TIMED_OUT in rerun](https://review.rdoproject.org/zuul/build/19cc9355efb94c40ac44eaf97a31763b)
    - tripleo
        + last_promotion=2022-08-20
        + LP#1988053
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657) (again)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
    - baremetal
        + last_promotion=2022-08-17
        + LP#1987092 in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`
        + [rerun of `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby` failed on another intermittent error](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
    - common
        + last_promotion=2022-08-10
        + no new content
    - security
        + last_promotion=2022-08-25
        + [overcloud deploy failed in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`, probably intermittent](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/af73631/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)
        + [rerun of `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby` failed on another intermittent error](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
    - tripleo
        + last_promotion=2022-08-27
        + no new content?
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658) (again)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
    - all **GREEN**😱

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
    - network
        + last_promotion=2022-08-28
        + [probably intermittent](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/bea6760/job-output.txt)
    - [reruns 1](https://code.engineering.redhat.com/gerrit/c/testproject/+/426497)
        + [intermittent FAILURE in `periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train`: tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/97/426497/1/check/periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train/12a00c1/logs/undercloud/var/log/tempest/stestr_results.html)
    - [reruns 2](https://review.rdoproject.org/r/c/testproject/+/44690)
        + [intermittent FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train`: download failed](https://logserver.rdoproject.org/90/44690/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/55f2554/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
    - tripleo [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/266024)
        periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17
    - pipeline_component-tripleo-pcci-17_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
    * up to date all components

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987092 
    - > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
    - job is in retry state, before closing we need to have the job stable
    - :warning: BLOCKER :warning:
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
    - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugs.launchpad.net/tripleo/+bug/1987632
    - > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
    - [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
    - check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
    - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
    - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
    - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
    - Douglas is still investigating
    - :warning: BLOCKER :warning:
    - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)

* https://bugs.launchpad.net/tripleo/+bug/1988053
    - > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
    - affecting c9 master tripleo component
    - :warning: BLOCKER :warning:
    - [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/855106)

* https://bugs.launchpad.net/tripleo/+bug/1987323 
    - > FIPS + Manila + Ceph Quincy
    - > ... known issue on the ceph side, ... already merged in master/quincy ... 
      > 17.2.4 should solve this issue, so we'll check during the next promotion.
    - On CIX board in lane `Waiting on external dependencies`
    - Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616 
    - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
    - no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
    - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
    - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](
      https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
    - [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
    - `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
    - `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
    - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1987641
    - > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
    - We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
    - ~~[fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)~~
    - [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814)
    - c9 master promoted on 2022-08-29 with the fix above applied (`dlrn_hash is: 6faf5854d8ed01f023b9513ffbc25ef3`)

* https://bugs.launchpad.net/tripleo/+bug/1986755
    - > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path. 
    - :warning: BLOCKER :warning:
    - Xek proposed: https://review.opendev.org/853478
        * Merged, rechicked security component
        * It might need to be backported to fix wallaby too
    - ipa job running in check itself - need to work with security team to continue debug/fixing.
    - Wallaby component was Force promoted; We need to watch integration lines
    - Once integration line has promoted we can close this bug and cix card
    - > wallaby promoted - closing out

---

## 2022-08-29

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and 
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
    - C9 Master ~~2022-08-26~~- 2022-08-29 (today)
        + [rerun 1](https://review.rdoproject.org/r/c/testproject/+/44656)
            + periodic-tripleo-ci-centos-9-scenario001-standalone-master fails on LP#1987641
        + [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429)
    - C9 Wallaby ~~2022-08-22~~ - 2022-08-29 (today)
        + [rerun rlandy](https://review.rdoproject.org/r/c/testproject/+/43883)
        + [rerun jm1](https://review.rdoproject.org/r/c/testproject/+/44661) (different hash)
    - C8 Wallaby 2022-08-29 (today)
    - C8 Train 2022-08-26 (friday)
        + [rerun 1 (again)](https://review.rdoproject.org/r/c/testproject/+/44662)
            + periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train fails on [intermittent error](https://bugs.launchpad.net/tripleo/+bug/1938987)
            + we can skip promote this - based on these two logs (two different tempest errors) [1](https://logserver.rdoproject.org/55/36255/112/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/835fb53/logs/undercloud/var/log/tempest/stestr_results.html.gz) and [2](https://logserver.rdoproject.org/62/44662/3/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/fe2c998/logs/undercloud/var/log/tempest/stestr_results.html.gz)
            + https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44689 Temp skip fs035 for train - **PLEASE REVERT after promo**
        + [rerun 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426435)
    - OSP 16.2 ~~2022-08-25~~ - 2022-08-29 (today)
        + [buildset](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/76a8b211ea8041118b64ec7b61ec5465)
        + [reruns 1](https://code.engineering.redhat.com/gerrit/c/testproject/+/286307)
        + [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
    - OSP17 2022-08-28
    - OSP17.1 ~~2022-08-25~~ - 2022-08-29 (today)
        + [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
        + [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/418658)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
    - Red:
        + baremetal (last_promotion=2022-08-26, probably intermittent due to [different](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/32bbb4c/job-output.txt) [errors](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/8e3fd34/job-output.txt) with same dlrn_hash)
        + cinder (last_promotion=2022-08-21, LP#1987641)
        + cloudops (last_promotion=2022-08-23, LP#1987641)
        + common (last_promotion=2022-08-23, LP#1987641)
        + glance (last_promotion=2022-08-16, LP#1987641)
        + network (last_promotion=2022-08-23, probably intermittent due to [different](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/512fd2a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) [errors](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/d288117/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) with same dlrn_hash)
        + tripleo (last_promotion=2022-08-20, new bug LP#1988053)
        + [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):

* [RHEL9 OSP17.1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=1):

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987641
    - > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
    - We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
    - ~~[fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)~~
    - [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854814)

* https://bugs.launchpad.net/tripleo/+bug/1987092 
    - > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
    - job is in retry state, before closing we need to have the job stable
    - :warning: BLOCKER :warning:
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
    - ask Jiri!

* https://bugs.launchpad.net/tripleo/+bug/1987632
    - > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
    - [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
    - check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
    - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
    - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
    - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
    - Douglas is still investigating

* https://bugs.launchpad.net/tripleo/+bug/1986755
    - > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path. 
    - :warning: BLOCKER :warning:
    - Xek proposed: https://review.opendev.org/853478
        * Merged, rechicked security component
        * It might need to be backported to fix wallaby too
    - ipa job running in check itself - need to work with security team to continue debug/fixing.
    - Wallaby component was Force promoted; We need to watch integration lines
    - Once integration line has promoted we can close this bug and cix card

* https://bugs.launchpad.net/tripleo/+bug/1987323 
    - > FIPS + Manila + Ceph Quincy
    - > ... known issue on the ceph side, ... already merged in master/quincy ... 
      > 17.2.4 should solve this issue, so we'll check during the next promotion.
    - On CIX board in lane `Waiting on external dependencies`
    - Not a blocker

* https://bugs.launchpad.net/tripleo/+bug/1988053
    - > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
    - affecting c9 master tripleo component
    - no CIX card

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616 
    - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
    - no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
    - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
    - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

### Fixed

* build containers failing for osp16.2 and osp17, Reported today but was just hitch : https://bugzilla.redhat.com/show_bug.cgi?id=2122167
Got green run on latest runs.

---

## 2022-08-26

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and 
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)
    - c9 master 2022-08-26 [reruns](https://review.rdoproject.org/r/c/testproject/+/44656)
    - c9 wallaby 2022-08-22 [reruns](https://review.rdoproject.org/r/c/testproject/+/44661)
        * https://review.rdoproject.org/r/c/testproject/+/43883 - chasing hash 4e233ad84bc9f1a0788729d4e0cc279d
passed periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby and periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby
    - c8 wallaby 2022-08-22 (no reruns suggested by rr tool)
        * (skip fs001 promoted - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44664 - was only missing 1 tempest test)
        * missed promoting ^^ could not get patch through ... chasing 5091f496614fa524adceaf94f3d3f5b1 in https://review.rdoproject.org/r/c/testproject/+/36254
    - c8 train 2022-08-26 [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)
    - osp16.2 2022-08-25 (yesterday) [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230)
    - osp17 2022-08-25 (yesterday)
    - osp17.1 2022-08-25 (yesterday)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
    - clients (4), cloudops (4), baremetal (4), ui (2), common (3), security (4), tripleo (4), compute (4), network (3) outdated
    - Red:
        + cinder (LP#1987641)
        + cloudops (LP#1987641)
        + common (LP#1987641 + [probably intermittent error](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/55ffe3e/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz))
        + glance (LP#1987641)
        + network ([tempest failed, maybe intermittent error?](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/042c2ce/logs/undercloud/var/log/tempest/stestr_results.html.gz) + [another intermittent failure?](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/698c90f/logs/undercloud/home/zuul/undercloud_install.log.txt.gz))
        + tripleo ([probably legit error](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/0dd495f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz) since all jobs fail on the [same](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/4158658/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) [error](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-tripleo-master/b74f7e5/logs/undercloud/home/zuul/undercloud_install.log.txt.gz))
        + ui (probably intermittent due to [timeout in tempest test](https://logserver.rdoproject.org/openstack-component-ui/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-ui-master/060d45b/logs/undercloud/var/log/tempest/stestr_results.html.gz))
        + validation ([maybe legit, maybe intermittent](https://logserver.rdoproject.org/openstack-component-validation/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/201dd4b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
    - Yellow: compute
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
    - octavia (4), security (5), network (3), tripleo (4) outdated
    - Red: 
        + clients ([probably legit haproxy error during overcloud deploy](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-wallaby/ac0612f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
        + network ([probably intermittent, timeout in tempest](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-network-wallaby/5a3e879/logs/undercloud/var/log/tempest/stestr_results.html.gz))
        + security ([`freeipa-setup : Run the tripleo-ipa preparation script`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/f28698e/job-output.txt), not related to LP#1986755?)
        + tripleo ([container keystone_bootstrap failed](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/8a6de3e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz))
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
    - tripleo (4), common (4), octavia (2) outdated
    - Red:
        + network ([tempest failures (timeout)](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/223574d/logs/undercloud/var/log/tempest/stestr_results.html.gz))
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
    - compute (6), tripleo (4) outdated
    - Red:
        + network ([tempest failures (timeout)](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/b927d28/logs/undercloud/var/log/tempest/stestr_results.html))

* [RHEL9 OSP17.1](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/v8gltz4Mz/rhos-17-full-component-pipeline?orgId=1):
    * up to date : no promotion blocker
    * periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 failing since 

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
    * rerunning failed jobs: build history is good https://code.engineering.redhat.com/gerrit/c/testproject/+/286307
    * all components are up to date 

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
    * rerunning failed jobs as build history is good 
https://code.engineering.redhat.com/gerrit/c/testproject/+/276230
    *  all components are up to date 

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1987641
    - > cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
    - We need a fix ([revert](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542) was -1'ed)
    - [fix](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854755)

* https://bugs.launchpad.net/tripleo/+bug/1987092 
    - > centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'
    - job is in retry state, before closing we need to have the job stable
    - :warning: BLOCKER :warning:
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
    - ask Jiri!

* https://bugs.launchpad.net/tripleo/+bug/1987632
    - > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
    - [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
    - check job, is this relevant? might fail because submitted patches are broken?!

* https://bugzilla.redhat.com/2105408
    - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
    - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
    - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
    - Douglas is still investigating

* https://bugs.launchpad.net/tripleo/+bug/1986755
    - > Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path. 
    - :warning: BLOCKER :warning:
    - Xek proposed: https://review.opendev.org/853478
        * Merged, rechicked security component
        * It might need to be backported to fix wallaby too
    - ipa job running in check itself - need to work with security team to continue debug/fixing.
    - Wallaby component was Force promoted; We need to watch integration lines
    - Once integration line has promoted we can close this bug and cix card

* https://bugs.launchpad.net/tripleo/+bug/1987323 
    - > FIPS + Manila + Ceph Quincy
    - > ... known issue on the ceph side, ... already merged in master/quincy ... 
      > 17.2.4 should solve this issue, so we'll check during the next promotion.
    - On CIX board in lane `Waiting on external dependencies`
    - Not a blocker

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616 
    - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
    - no CIX card

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1983817
    - >  periodic integration all branches RETRY Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org
    - [That should be resolved based on updates in the VH support ticket, please review job runs since Aug 23 1400 UTC yesterday](https://bugs.launchpad.net/tripleo/+bug/1983817/comments/16)

---
