# Ruck Rover 2022-09-01 to 2022-09-01

###### tags: `ruck_rover`
###### Next RR notes: https://hackmd.io/0RguTgCAQsiZz_SnF5209A
###### Previous RR notes: https://hackmd.io/7qAKWCiCQA6IEdE9WXYn4Q

##### ruck: Pooja, rover: Jakob

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

---

## 2022-09-01

### Reruns and Investigations:

(crossed out := known bugs recorded above)

**NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

* [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

- C9 Master
+ last_promotion=2022-08-31
+ `periodic-tripleo-ci-build-containers-ubi-9-quay-push-master`
* [`package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-master/31a529b/job-output.txt), might be LP#1988397 or related to LP#1985981
+ `periodic-tripleo-ci-build-containers-centos-9-push-master`
* [`package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/ea8ed37/job-output.txt), might be LP#1988397 or related to LP#1985981
* ```
<ysandeep> fyi.. there was a bug related to catatonit few days back: https://bugs.launchpad.net/tripleo/+bug/1985981
<ysandeep> https://review.opendev.org/c/openstack/tripleo-quickstart/+/853142 was temp workaround

        + [reruns](https://review.rdoproject.org/r/c/testproject/+/44656)

    - C9 Wallaby
        + last_promotion=2022-08-29
        + `periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby`
            * [`package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby/98c918e/job-output.txt), might be LP#1988397 or related to LP#1985981
        + `periodic-tripleo-ci-build-containers-centos-9-push-wallaby`
            * [`package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-wallaby/70aa364/job-output.txt), might be LP#1988397 or related to LP#1985981
        + yesterdays [reruns](https://review.rdoproject.org/r/c/testproject/+/44661) still in progress

    - C8 Wallaby
        + last_promotion=2022-08-31
        + `check-rdo/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-wallaby`:
            * [overcloud inspection failed with `FATAL | ... No nodes are manageable at this time`](https://sf.hosted.upshift.rdu2.redhat.com/logs/43/44343/23/check-rdo/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-wallaby/b8b22c4/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz)

    - C8 Train
        + last_promotion=2022-08-30
        + missing fs001 to promote - rerun in https://review.rdoproject.org/r/c/testproject/+/44662
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train`
            * [overcloud deploy failed with `FATAL | Wait for containers to start for step 3 using paunch`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/5e6d88f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`
            * [overcloud deploy failed with `FATAL | Wait for containers to start for step 3 using paunch`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/029c435/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
            * [LP#1938987](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz)
        + `periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train`
            * [tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train/d779aba/logs/undercloud/var/log/tempest/stestr_results.html.gz)
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`
            * [tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/911cf14/logs/undercloud/var/log/tempest/stestr_results.html.gz)
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-train`
            * [`TASK [modify-image : Mount image]`](https://sf.hosted.upshift.rdu2.redhat.com/logs/12/44712/1/check-rdo/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-train/6f5fe9b/job-output.txt)
        + [reruns](https://review.rdoproject.org/r/c/testproject/+/44662)

    - OSP 17.1 2022-08-29 (2 days out)
        - [reruns](https://code.engineering.redhat.com/gerrit/c/testproject/+/209874)

    - OSP 17 2022-08-28 (3 days out)
        - [periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/8031467/job-output.txt) No space left on device
        - Blocked on OVBs Instance unable to reach metadata
            * [Sandeep's Ticket](https://redhat.service-now.com/help?id=rh_ticket&table=incident&sys_id=59d370e397e11550eabfbf2e6253afcf)

    - OSP 16.2 : 2022-08-29 (2 days out)
        - Bring back to prodchain blocked [CIX card](https://trello.com/c/THK34EMH/2593-cixrhos-162rhel8task-libvirt-setup-common-define-volume-pool-breaks-periodic-tripleo-ci-rhel-8-bmenvd-3ctlr1comp-featureset035-r)
            * [rerun](https://code.engineering.redhat.com/gerrit/c/testproject/+/276230/76)
        - Blocked on OVBs Instance unable to reach metadata
            * [Sandeep's Ticket](https://redhat.service-now.com/help?id=rh_ticket&table=incident&sys_id=59d370e397e11550eabfbf2e6253afcf)

* [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1):
    - clients
        + last_promotion=2022-08-31
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`
            * [`FATAL | Create containers managed by Podman for ...`](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/20f5385/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
            * [`FATAL | Create containers managed by Podman for ...`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - common
        + last_promotion=2022-08-31
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master`
            + [probably LP#1988397](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/4b1cb76/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
    - network
        + last_promotion=2022-08-31
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master`
            * [probably LP#1988397](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/beb77c7/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz)
    - security
        + last_promotion=2022-08-30
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`
            * [LP#1988347](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/a600b45/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657)

* [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1):
    - common
        + last_promotion=2022-08-10
        + no new content?
        + `periodic-tripleo-ci-centos-8-9-multinode-mixed-os-common-wallaby`
            * [TODO](https://logserver.rdoproject.org/99/44499/7/check/periodic-tripleo-ci-centos-8-9-multinode-mixed-os-common-wallaby/71ec5c0)
        + `periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby`
            * [TODO](https://logserver.rdoproject.org/99/44499/7/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby/ffd3468)
    - network 
        + last_promotion=2022-08-30
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby`
            * [tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/f137aee/logs/undercloud/var/log/tempest/stestr_results.html.gz)
        + `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby`
            * [other tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/25732e4/logs/undercloud/var/log/tempest/stestr_results.html.gz)
    - tripleo
        + last_promotion=2022-08-31
        + `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`
            * [`FATAL | Create containers managed by Podman for ...`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658)

* [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1):
    - network
        + last_promotion=2022-08-30
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby`
            * [overcloud deploy failed with `ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: ...deploy_steps_playbook.yaml ... timeout, Return Code: 254`](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/d52f0e2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44659)

* [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1):
    - tripleo
        + last_promotion=2022-08-26
        + `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train`
            * [tempest failures](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/8ccd4ba/logs/undercloud/var/log/tempest/stestr_results.html.gz)
    - [reruns](https://review.rdoproject.org/r/c/testproject/+/44690)

* [RHEL9 OSP17](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1):
    * tripleo component lagging with : 
        * [periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17/2d35d6f/job-output.txt) Find out UUID of instance with metadata URL
        * Rest all components are up to date

* [RHEL8 OSP16.2](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1):
    * network component is lagging with : 
        * [periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2](https://sf.hosted.upshift.rdu2.redhat.com/logs/58/418658/25/check/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2/9496d30/job-output.txt)
        * Rest all components are up to date

### Known Bugs

* https://bugs.launchpad.net/tripleo/+bug/1960355
    - > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
    - affecting OSP17.1 as well

* https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning:
    - Pacemaker performance causes intermittent galera issues in loaded CI env
    - jobs fail on `Check Keystone public endpoint status` or in other places
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0)
    - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18)

* https://bugs.launchpad.net/tripleo/+bug/1987632
    - > cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
    - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001)
    - [lot of passes on wednesday and earlier](https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&result=SUCCESS)
    - atm only affecting check jobs
    - might be an issue with host virt environments, i.e. the cloud provider

* https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning:
    - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
    - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
    - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t)
    - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`)
    - [workaround (specifying `cpu_models` in libvirt)](
      https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is
      [currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/)

* https://bugs.launchpad.net/tripleo/+bug/1988347 :warning: **BLOCKER** :warning:
    - fails on c9 master and c9 wallaby
    - fs064, fs39 and others
    - intermittent failure aka when rerunning hard enough it will pass eventually!
    - > `FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}`
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/6ea2096/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/0c48b0f/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/a0594a2/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master`](https://logserver.rdoproject.org/57/44657/6/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master/8226ab0/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/545f2f1/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/40d3797/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
    - Ansible pretends that commands `arecord` and `ptrrecord` are while trying to run modules in [role tripleo_ipa_dns](https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles/tripleo_ipa_dns/tasks/dns.yaml)
    - [history of job `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master&skip=0) shows e.g. a failure on [2022-08-22](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/f8018e3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) but subsequent passes

* https://bugs.launchpad.net/tripleo/+bug/1987323 
    - > FIPS + Manila + Ceph Quincy
    - > ... known issue on the ceph side, ... already merged in master/quincy ... 
      > 17.2.4 should solve this issue, so we'll check during the next promotion.
    - On CIX board in lane `Waiting on external dependencies`
    - Not a blocker

* https://bugs.launchpad.net/tripleo/+bug/1988397
    - tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages 
    - ```
      <ysandeep> we still have mirror issues:
      <ysandeep> job have .40 version installed
      <ysandeep> https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/var/log/extra/package-list-installed.txt
      <ysandeep> ~~~
      <ysandeep> NetworkManager.x86_64                            1:1.40.0-1.el9                        @baseos                       
      <ysandeep> NetworkManager-libnm.x86_64                      1:1.40.0-1.el9                        @baseos 
      <ysandeep> ~~~
      <ysandeep> I don't see .40 rpms for NetworkManager on rackspace mirror: http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> facebook mirror looks good
      <ykarel> rackspace one not updated since 10 hours
      <ykarel> http://dfw.mirror.rackspace.com/centos-stream/timestamp.txt
      <jm1> ysandeep, ykarel: and neither on http://mirror.mtl01.iweb.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/
      <ykarel> yes ^ syncs from mirror which is rax currently so should be same 
      <ykarel> http://mirror.mtl01.iweb.opendev.org/centos-stream/timestamp.txt
      <ykarel> 2022-09-01T06:08:45,769043757+00:00

- no CIX card

### Intermittent Failures

* https://bugs.launchpad.net/tripleo/+bug/1987616
- > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card

* https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz)

* `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz)

* `Connection failure: The read operation timed out` in `TASK [Get current autoholds]`
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt)

* `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

* `FATAL | Wait for puppet host configuration to finish | undercloud | error`...
- `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt)
- `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)
- [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)

* `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy
- [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz)

### Fixed

* https://bugs.launchpad.net/tripleo/+bug/1988053 :warning: **BLOCKER** :warning:
- > Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
- affecting c9 master tripleo component
- [Takashi Kajinami is working on fixes](https://review.opendev.org/q/topic:bug%252F1988053)
- c9 master tripleo component promoted today, bug is fixed

---