Ruck Rover 2022-09-13 to 2022-09-15

# Ruck Rover 2022-09-13 to 2022-09-15 ###### tags: `ruck_rover` ###### Next RR notes: https://hackmd.io/a92u74hDQhe5afE_LUtbRw ###### Previous RR notes: https://hackmd.io/s4TgnCY-QQGKv2ONxTjOZA ##### ruck: Bhagyashris, rover: Jakob [RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com) [RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/) [OpenStack Program Meeting 2022]( https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit) Zuul Status: * [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/) * [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status) * [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status) --- ## 2022-09-15 ### TLDR * Upstream - **TODO** * Downstream - **TODO** ### Reruns and Investigations: (crossed out := known bugs recorded above) **NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status. #### Integration Lines * [centos-9 master]( http://dashboard-ci.tripleo.org/d/mde16Npnz/centos-9-and-centos-8-comparison-master) - last_promotion=2022-09-13 - [hash `8c71d10c1dacd8b2b5fc59dbdd753186`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=8c71d10c1dacd8b2b5fc59dbdd753186) + [`periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/8ffe44e) + [`periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/69063f9) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master): * [Overcloud stack: FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/adf4075) + [`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master): * [Tempest tests FAILED. Reason: unknown ](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/118/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/7fba01d) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master): * [Overcloud stack: FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/8a99ecc) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master): * [N/A](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/588602f) * [centos-9 wallaby]( http://dashboard-ci.tripleo.org/d/NRJeqYBnz/centos-9-and-centos-8-comparison-wallaby) - last_promotion=2022-09-13 - [hash `164135bf6866a2568241e186f56b903b`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=164135bf6866a2568241e186f56b903b) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby): * [Overcloud stack: FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/8679de7) + [`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/117/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby/bbd68f1) + [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby/a630f17) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/ed97c89) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/f4cd59b) + [`periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby): * [N/A](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby/e700966) + [`periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/6df3180) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/4d47b44) * [centos-8 wallaby]( http://dashboard-ci.tripleo.org/d/NRJeqYBnz/centos-9-and-centos-8-comparison-wallaby) - last_promotion=2022-09-14 - [hash `5b62e82eebd71259ed086d27d7a2abae`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=5b62e82eebd71259ed086d27d7a2abae) * [centos-8 train]( http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train) - last_promotion=2022-09-12 - [hash `54e3db5e5134970e993eeb7d124da84c`](https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=54e3db5e5134970e993eeb7d124da84c) + [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train): * [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/65/31165/92/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/f55e2c7) * [rhel-9 osp17-1]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline) - last_promotion=2022-09-14 - [hash `fb7fef6ff353625dadab13dfa1c2c80e`](https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel9-osp17-1/api/civotes_agg_detail.html?ref_hash=fb7fef6ff353625dadab13dfa1c2c80e) * [rhel-9 osp17]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline) - last_promotion=2022-09-11 - [hash `0990788a06ae1c3505956399b88fd630`](https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel9-osp17/api/civotes_agg_detail.html?ref_hash=0990788a06ae1c3505956399b88fd630) * [rhel-8 osp16-2]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline) - last_promotion=2022-09-13 - [hash `546f5261237fd7e18a39ce31ae980932`](https://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=546f5261237fd7e18a39ce31ae980932) + [`periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2`]( https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2): * [N/A](https://sf.hosted.upshift.rdu2.redhat.com/logs/00/422400/14/check/periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2/f09c1b4) #### Component Lines * [centos-9 master]( http://dashboard-ci.tripleo.org/d/M1DYSidGz/component-pipeline) - baremetal + last_promotion=2022-09-13 + [hash `a99bf274e4baec8e585bc9979e492bb8d85d17b5`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=a99bf274e4baec8e585bc9979e492bb8d85d17b5&distro_hash=d7d161658d469b9211bb2b8cfabc9aa4fca9ed68) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master): - [Overcloud stack: FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/f5ac18b) - cinder + last_promotion=2022-09-14 + [hash `87205a242bbf1c0c39490a1d518d6918d08addca`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=87205a242bbf1c0c39490a1d518d6918d08addca&distro_hash=f8434e1ff5730482ab08ad641743a491c74646b4) * [`periodic-tripleo-ci-centos-9-scenario004-standalone-cinder-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario004-standalone-cinder-master): - [N/A](https://logserver.rdoproject.org/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-cinder-master/8a11593) - common + last_promotion=2022-09-14 + [hash `ee5d8644c328e850c561c99b2098aeba2bf1b471`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=ee5d8644c328e850c561c99b2098aeba2bf1b471&distro_hash=e078d66a731739522e3da5dbb8786aa4496740db) - compute + last_promotion=2022-09-07 + [hash `299c898cf4af1d96a4c683305d3c262285eccdaf`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=299c898cf4af1d96a4c683305d3c262285eccdaf&distro_hash=2af3f376e5c5ad8caa7e307511b550bcdfb93d1d) * [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master): - [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/57/44657/18/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/d4674de) - security + last_promotion=2022-09-13 + [hash `44cdbc68658d8fe3013d1dd253a268bec343c087`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=44cdbc68658d8fe3013d1dd253a268bec343c087&distro_hash=8cbe449b641de048de115303a8a4e9226068befa) - tempest + last_promotion=2022-09-05 + [hash `b39bbcc0bc5be39cc4d9e3cf2addf1924c8df49a`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=b39bbcc0bc5be39cc4d9e3cf2addf1924c8df49a&distro_hash=c770ffa7eab8d0e5ad9c2a476b166f3e3175fb89) - tripleo + last_promotion=2022-09-14 + [hash `948a5f23d866e2d30a5d4c2e723a63d1aae637e6`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_detail.html?commit_hash=948a5f23d866e2d30a5d4c2e723a63d1aae637e6&distro_hash=d7b1b549640cd2d447542d79bbefdadf4d271d90) * [centos-9 wallaby]( http://dashboard-ci.tripleo.org/d/b0e2jBLnk/component-pipeline-wallaby-centos9) - tempest + last_promotion=2022-09-07 + [hash `d5fc8ad86bb895ebf59ff73bdcf36a9a6b7eff4f`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_detail.html?commit_hash=d5fc8ad86bb895ebf59ff73bdcf36a9a6b7eff4f&distro_hash=048d9d649c588abdc0b659ced95fdc6ea2d007f0) * [`periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-wallaby): - [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-wallaby/3fcb900) - tripleo + last_promotion=2022-09-12 + [hash `39be6cfe5cc0e18bbea85705f593b7608cadd0a6`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_detail.html?commit_hash=39be6cfe5cc0e18bbea85705f593b7608cadd0a6&distro_hash=bfb6fe68901ad22fe9ff2addedeaead32889acc4) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby): - [Tempest tests FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/81c8259) * [centos-8 wallaby]( http://dashboard-ci.tripleo.org/d/TWhcr9_Mk/component-pipeline-wallaby) - network + last_promotion=2022-09-09 + [hash `b125402309516d25e40113f3963d1bdf0a25421b`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_detail.html?commit_hash=b125402309516d25e40113f3963d1bdf0a25421b&distro_hash=d1361717722f81c13cf3874411ee1cbbe775d169) * [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby): - [MAC conflict between hosts. Overcloud stack: FAILED. Reason: infra ](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/e403b0f) - tripleo + last_promotion=2022-09-13 + [hash `39be6cfe5cc0e18bbea85705f593b7608cadd0a6`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_detail.html?commit_hash=39be6cfe5cc0e18bbea85705f593b7608cadd0a6&distro_hash=bfb6fe68901ad22fe9ff2addedeaead32889acc4) * [`periodic-tripleo-ci-centos-8-standalone-tripleo-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-tripleo-wallaby): - [N/A](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tripleo-wallaby/46c6dc5) * [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby): - [Overcloud stack: FAILED. Reason: unknown ](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/579a59f) * [centos-8 train]( http://dashboard-ci.tripleo.org/d/mOvYIiOMk/component-pipeline-train) - tripleo + last_promotion=2022-09-10 + [hash `f7fa1d52a7f5d7ae868bc6a57da5e7cae1c84748`](https://trunk.rdoproject.org/api-centos8-train/api/civotes_detail.html?commit_hash=f7fa1d52a7f5d7ae868bc6a57da5e7cae1c84748&distro_hash=90903973080e5443bd999c959ea13f406c63d5a2) * [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train`]( https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train): - [Timeout. Reason: unknown ](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/b804f15) * [rhel-9 osp17-1]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline) * [rhel-9 osp17]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline) * [rhel-8 osp16-2]( http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline) ### Known Bugs * https://bugs.launchpad.net/tripleo/+bug/1960355 - > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) - affecting OSP17.1 as well * https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning: - Pacemaker performance causes intermittent galera issues in loaded CI env - jobs fail on `Check Keystone public endpoint status` or in other places - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0) - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18) * https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning: - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup) - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6. - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t) - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`) - [workaround (specifying `cpu_models` in libvirt)]( https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is [currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/) * https://bugs.launchpad.net/tripleo/+bug/1988514 - [Trello](https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re) - On CIX board in lane `Waiting on external dependencies` * https://bugs.launchpad.net/tripleo/+bug/1987323 - > FIPS + Manila + Ceph Quincy - > ... known issue on the ceph side, ... already merged in master/quincy ... > 17.2.4 should solve this issue, so we'll check during the next promotion. - On CIX board in lane `Waiting on external dependencies` - Not a blocker * https://bugs.launchpad.net/tripleo/+bug/1988810 - Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788 * https://bugs.launchpad.net/tripleo/+bug/1989197 - > Tempest test `neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON` fails on `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master` * https://bugs.launchpad.net/tripleo/+bug/1989341 - > Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch * https://bugs.launchpad.net/tripleo/+bug/1989247 - > overcloud deploy fails with `sudo: unable to send audit message: Operation not permitted` - > `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * https://bugs.launchpad.net/tripleo/+bug/1989452 - > multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host * https://bugs.launchpad.net/tripleo/+bug/1989606 - > container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use" ### Intermittent Failures * https://bugs.launchpad.net/tripleo/+bug/1987616 - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply' - no CIX card * https://bugs.launchpad.net/tripleo/+bug/1938987 - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use') - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz) * `TASK [freeipa-setup : Run the tripleo-ipa preparation script]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt) * `Connection failure: The read operation timed out` in `TASK [Get current autoholds]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt) * `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - > this has different root causes, watch out for log details! - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * `FATAL | Wait for puppet host configuration to finish | undercloud | error`... - `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt) - `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) * `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz) ### Fixed ### Downstream: * rhos16.2 on rhel8: * Promotion: 13-sept-2022 * In todays run conatiner build job failed so re-running the job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 * container build job failing due to this change https://review.opendev.org/c/openstack/tripleo-ci/+/850736 * job passed on revert change https://review.opendev.org/c/openstack/tripleo-ci/+/857776 job result: periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2_1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/39d84ef71b86408aa433b5edbd452acc : SUCCESS in 1h 08m 23s * rhos17.1 on rhel8 * Promotion: 14-sept-2022 * rhos17 on rhel9: * Promotion: 11-Sept-2022 * Note: we don't have new content to promote. * rhos17.1 on rhel9: * Promoted: 15-sept-2022 * Waiting for jekins jobs result, integration line is GREEN ### Downstream component: --- ## 2022-09-14 ### Notes * doug: hi o/ - we merged a small change on sshd_config that affects centos-9 images (https://softwarefactory-project.io/r/c/config/+/26113) - note that this image isn't build on daily basis anymore, so our jobs won't get this update unless we update the base image and request rhos-infra team to unpause the build. Just in case you need a new image, keep an eye on related ssh issues with missing kex alghoritm curve25519. ### Reminder: Please update https://docs.engineering.redhat.com/display/PRODCHAIN/2022-09-12+Meeting+Notes for the program call. wallaby c8, c9, train, 16.2, 17 , 17.1 on rhel-9 and rhel-8 ### Reruns and Investigations: (crossed out := known bugs recorded above) **NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status. * [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) - C9 Master + last_promotion: 2022-09-13 + [hash `573ed7f477dfe3ed338408f28dac7035`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=573ed7f477dfe3ed338408f28dac7035) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master)~~: - ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/222e953)~~ * [`periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master): - [Overcloud deploy failed with `ERROR: Can't run container glance_api_db_sync`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/6706077/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * [Overcloud stack: FAILED. Reason: `ERROR: Can't run container rabbitmq_bootstrap`](https://logserver.rdoproject.org/56/44656/21/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/81b0212/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master): - [undercloud install failed with `ensure apache is installed...Failed to download packages: centos-logos-httpd-***el9.noarch: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/6b6ada3/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) * [Overcloud stack: FAILED. Reason: `ConnectionRefusedError: [Errno 111] Connection refused` ](https://logserver.rdoproject.org/56/44656/21/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/c3fb2a5/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master): - [intermittent failure `TASK [freeipa-setup : Run the tripleo-ipa preparation script]`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/6265a3a/job-output.txt) * [Overcloud stack: FAILED. Reason: `ERROR: Can't run container nova_db_sync`](https://logserver.rdoproject.org/56/44656/21/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/7ed641e/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master): - [Overcloud deploy failed with `ERROR: Can't run container mysql_wait_bundle`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/4d0da94/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * [Tempest tests FAILED](https://logserver.rdoproject.org/56/44656/21/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/29df173/logs/undercloud/var/log/tempest/stestr_results.html.gz) * ~~[`periodic-tripleo-ci-centos-9-scenario001-standalone-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario001-standalone-master)~~: - ~~[TODO](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario001-standalone-master/a2c9332)~~ * [`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master): - [LP#2105408?](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/109/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/0e4c843/logs/undercloud/var/log/tempest/stestr_results.html) * [LP#2105408?](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/109/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/0e4c843/logs/undercloud/var/log/tempest/stestr_results.html) * [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656) * [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429) (internal) - C9 Wallaby + last_promotion: 2022-09-13 + [hash `84acb1bac5b6c640738a43e63c532c2c`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=84acb1bac5b6c640738a43e63c532c2c) * [`periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby/6329674) * [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby/9c8a8eb) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/11873f7) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/a9756ff) * ~~[`periodic-tripleo-ci-centos-8-9-multinode-mixed-os`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-9-multinode-mixed-os)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-9-multinode-mixed-os/a88b2c9) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/3fda6a3) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby)~~ (not in criteria): - [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/37/44737/23/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby/79d5853) * ~~[`periodic-tripleo-ci-centos-9-8-multinode-mixed-os`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-8-multinode-mixed-os)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-8-multinode-mixed-os/daeab46) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby)~~ (not in criteria): - [TODO](https://sf.hosted.upshift.rdu2.redhat.com/logs/38/44738/23/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby/4a8d7b4) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/7ac5426) * [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44661) - C8 Wallaby + last_promotion: 2022-09-11 + [hash `5b62e82eebd71259ed086d27d7a2abae`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=5b62e82eebd71259ed086d27d7a2abae) * [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby): - [`TASK [run-test : run toci_gate_test.sh]` failed with `HTTPConnectionPool(host='mirror.regionone.ibm-bm2-nodepool.rdoproject.org', port=80): Read timed out.`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/5affdf9/job-output.txt) - [overcloud deploy failed with `Error: 'rabbitmq-bundle' already exists. Too many tries`](https://logserver.rdoproject.org/83/44983/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/8aad894/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [overcloud deploy failed with `tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time`](https://logserver.rdoproject.org/83/44983/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby/6700c1a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44983) - C8 Train + last_promotion: 2022-09-12 + [hash `54e3db5e5134970e993eeb7d124da84c`](https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=54e3db5e5134970e993eeb7d124da84c) * [`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train): - [tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/1844f24/logs/undercloud/var/log/tempest/stestr_results.html) - [other (?) tempest failures](https://logserver.rdoproject.org/62/44662/13/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/40a4f2d/logs/undercloud/var/log/tempest/stestr_results.html.gz) * [reruns](https://review.rdoproject.org/r/c/testproject/+/44662) * hash: 4e3db5e5134970e993eeb7d124da84c missing on fs035 - rerun https://review.rdoproject.org/r/c/testproject/+/31165 - RHOS-17.1 on RHEL-9 **Refer to Bhagyashris' notes below** - RHOS-17.1 on RHEL-8 **Refer to Bhagyashris' notes below** - RHOS-17.0 on RHEL-9 **Refer to Bhagyashris' notes below** - RHOS-16.2 on RHEL-8 **Refer to Bhagyashris' notes below** #### Component Lines * [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1): - baremetal + last_promotion: 2022-09-13 + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master): * [TODO](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/c0ceb4b) - clients + last_promotion: 2022-09-09 + ~~[`periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master/2f6706b) - common + last_promotion: 2022-09-11 + ~~[`periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master/03aa0d4) - compute + last_promotion: 2022-09-07 + [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master): * [TODO](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/5c6b3ab) - manila + last_promotion: 2022-09-12 + [`periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master): * [TODO](https://logserver.rdoproject.org/openstack-component-manila/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master/74eace2) - tempest + last_promotion: 2022-09-05 + ~~[`periodic-tripleo-ci-centos-9-standalone-tempest-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-tempest-master)~~: * ~~[TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tempest-master/4540c6d)~~ + [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master): * [TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master/8785c6a) - tripleo + last_promotion: 2022-09-12 + ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master)~~: * ~~[TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/00f4c19)~~ - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657) * [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1): - common + last_promotion: 2022-09-10 + ~~[`periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/84/44684/7/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby/5fa2da2) - network + last_promotion: 2022-09-13 + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby): * [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/62f7c50) - tempest + last_promotion: 2022-09-07 + [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby): * [TODO](https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby/b7d82af) - tripleo + last_promotion: 2022-09-12 + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby): * [TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/0747d5f) - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658) * [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1): - network + last_promotion: 2022-09-09 + [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby): * [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby/b4d2ca6) - tripleo + last_promotion: 2022-09-13 + [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby): * [TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/fad474c) - [reruns](https://review.rdoproject.org/r/c/testproject/+/44659) * [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1): - network + last_promotion: 2022-09-09 + ~~[`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/e7f86e0) - tripleo + last_promotion: 2022-09-10 + [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train): * [TODO](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/9f349bb) - [reruns](https://review.rdoproject.org/r/c/testproject/+/44690) * [RHOS-17.0 on RHEL-9](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1): * **Refer to Bhagyashris' notes below** * [RHOS-16.2 on RHEL-8](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1): * **Refer to Bhagyashris' notes below** ### Downstream: * rhos16.2 on rhel8: * Promotion: 13-sept-2022 * blocked due to fule-tempest api , scenario and fs020 re-running jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 * rhos17.1 on rhel8 * Promotion: 14-sept-2022 * buildimage overcloud full and tempest api job failed re-running here: https://code.engineering.redhat.com/gerrit/c/testproject/+/424992 * rhos17 on rhel9: * Promotion: 14-Sept-2022 * Note: we don't have new content to promote. * rhos17.1 on rhel9: * Promoted: 12-sept-2022 * ### Downstream component: ### Known Bugs * https://bugs.launchpad.net/tripleo/+bug/1960355 - > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) - affecting OSP17.1 as well * https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning: - Pacemaker performance causes intermittent galera issues in loaded CI env - jobs fail on `Check Keystone public endpoint status` or in other places - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0) - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18) * https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning: - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup) - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6. - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t) - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`) - [workaround (specifying `cpu_models` in libvirt)]( https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is [currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/) * https://bugs.launchpad.net/tripleo/+bug/1988514 - [Trello](https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re) - On CIX board in lane `Waiting on external dependencies` * https://bugs.launchpad.net/tripleo/+bug/1987323 - > FIPS + Manila + Ceph Quincy - > ... known issue on the ceph side, ... already merged in master/quincy ... > 17.2.4 should solve this issue, so we'll check during the next promotion. - On CIX board in lane `Waiting on external dependencies` - Not a blocker * https://bugs.launchpad.net/tripleo/+bug/1988810 - Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788 * https://bugs.launchpad.net/tripleo/+bug/1989197 - > Tempest test `neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON` fails on `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master` * https://bugs.launchpad.net/tripleo/+bug/1989341 - > Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch * https://bugs.launchpad.net/tripleo/+bug/1989247 - > overcloud deploy fails with `sudo: unable to send audit message: Operation not permitted` - > `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * https://bugs.launchpad.net/tripleo/+bug/1989452 - > multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host * https://bugs.launchpad.net/tripleo/+bug/1989606 - > container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use" ### Intermittent Failures * https://bugs.launchpad.net/tripleo/+bug/1987616 - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply' - no CIX card * https://bugs.launchpad.net/tripleo/+bug/1938987 - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use') - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz) * `TASK [freeipa-setup : Run the tripleo-ipa preparation script]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt) * `Connection failure: The read operation timed out` in `TASK [Get current autoholds]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt) * `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - > this has different root causes, watch out for log details! - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * `FATAL | Wait for puppet host configuration to finish | undercloud | error`... - `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt) - `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) * `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz) ### Fixed --- ## 2022-09-13 ### Reruns and Investigations: (crossed out := known bugs recorded above) **NOTE:** Watch for running `testproject` jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status. * [Promotions](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) - C9 Master + last_promotion: 2022-09-09 + [hash `71abe3aa5c8ad3da6d894dc93d9c0273`](https://trunk.rdoproject.org/api-centos9-master-uc/api/civotes_agg_detail.html?ref_hash=71abe3aa5c8ad3da6d894dc93d9c0273) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master)~~: - ~~[overcloud deploy failed with `Clean up legacy Cinder keystone catalog entries`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/33e26a7/logs/undercloud/home/zuul/overcloud1_deploy.log.txt.gz)~~ - ~~[intermittent failure `TASK [freeipa-setup : Deploy FreeIPA]`](https://logserver.rdoproject.org/56/44656/19/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/e25c26b/job-output.txt)~~ * [`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master): - [tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/11/44711/100/check-rdo/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/73e8876/logs/undercloud/var/log/tempest/stestr_results.html) - [other tempest failures](https://sf.hosted.upshift.rdu2.redhat.com/logs/29/426429/5/check/periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master/57896e7/logs/undercloud/var/log/tempest/stestr_results.html) * ~~[`periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master)~~: - ~~[other tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master/ec9dfc7/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~ * ~~[`periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master)~~: - ~~[overcloud deploy failed with `tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/a2ace6f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~ - ~~[undercloud install failed in `Manage package | undercloud` with `Failed to download packages: double-conversion-3.1.5-6.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/56/44656/19/check/periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master/825bf9b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz)~~ * ~~[`periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master)~~ (not in criteria): - [failed in `TASK [standalone : Write containers-prepare-parameters.yaml]` with `Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host`](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master/47f8263) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-5`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-5)~~ (not in criteria): - [overcloud deploy failed with `ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ConnectionRefusedError: [Errno 111] Connection refused`](https://logserver.rdoproject.org/88/44388/10/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-5/50185ff/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * ~~[`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master)~~: - ~~[other² tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/4022186/logs/undercloud/var/log/tempest/stestr_results.html.gz)~~ - ~~[standalone deploy failed in `ensure apache is installed` with `Failed to download packages: apr-1.7.0-11.el9.x86_64: Cannot download, all mirrors were already tried without success`](https://logserver.rdoproject.org/56/44656/19/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master/d70465f/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)~~ * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master): - [other³ tempest failures](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/d0f8795/logs/undercloud/var/log/tempest/stestr_results.html.gz) - [LP#1987092](https://logserver.rdoproject.org/56/44656/19/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/68cd2e4/logs/overcloud-controller-2/var/log/containers/stdouts/galera-bundle.log.txt.gz) * ~~[`periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master)~~ (not in criteria): - [`cannot install the best update candidate for package gnutls-utils-3.7.3-10.el9.x86_64`](https://sf.hosted.upshift.rdu2.redhat.com/logs/42/44742/20/check-rdo/periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master/3df2a8f/job-output.txt) * will be fixed by [chandan's revert](https://review.opendev.org/c/openstack/tripleo-quickstart/+/856582) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-2`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-2)~~ (not in criteria): - [node provisioning failed](https://logserver.rdoproject.org/88/44388/10/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-2/cd0c3bb/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz) * [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44656) * [reruns 2](https://code.engineering.redhat.com/gerrit/c/testproject/+/426429) (internal) * [reruns 3](https://review.rdoproject.org/r/c/testproject/+/44946) * [reruns 4](https://review.rdoproject.org/r/c/testproject/+/44947) * [reruns 5](https://review.rdoproject.org/r/c/testproject/+/44948) * [reruns 6](https://review.rdoproject.org/r/c/testproject/+/44949) - C9 Wallaby + last_promotion: ~~2022-09-09~~ 2022-09-13 + [hash `f6749f9cda54d017021ab56bf4ec0958`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=f6749f9cda54d017021ab56bf4ec0958): * yesterdays hash has passed today + [hash `ea8639e875ec8d7ee7158bce2cd3fa2b`](https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=ea8639e875ec8d7ee7158bce2cd3fa2b) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby): - [TODO](https://logserver.rdoproject.org/55/36255/127/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/af08331) * ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby/ed83944) * ~~[`periodic-tripleo-ci-centos-9-8-multinode-mixed-os`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-8-multinode-mixed-os)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-8-multinode-mixed-os/5d8e78e) * [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby): - [TODO](https://logserver.rdoproject.org/65/31165/89/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/3d50196) * [reruns 1](https://review.rdoproject.org/r/c/testproject/+/44661) - C8 Wallaby + last_promotion: 2022-09-11 + [hash `21d2e33704a2f14d611cf4edff39f70e`](https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=21d2e33704a2f14d611cf4edff39f70e) * ~~[`periodic-tripleo-ci-centos-8-skipped-standalone-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-skipped-standalone-wallaby)~~ (not in criteria): - [TODO](https://logserver.rdoproject.org/openstack-periodic-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-skipped-standalone-wallaby/a3ac814) - C8 Train + last_promotion: 2022-09-12 + [hash `54e3db5e5134970e993eeb7d124da84c`](https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=54e3db5e5134970e993eeb7d124da84c) * [`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/65c3551) * [`periodic-tripleo-centos-8-buildimage-overcloud-full-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-8-buildimage-overcloud-full-train): - [TODO](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-overcloud-full-train/98d96fd) * [reruns](https://review.rdoproject.org/r/c/testproject/+/44662) - RHOS-17.1 on RHEL-9 **Refer to Bhagyashris' notes below** - RHOS-17.1 on RHEL-8 **Refer to Bhagyashris' notes below** - RHOS-17.0 on RHEL-9 **Refer to Bhagyashris' notes below** - RHOS-16.2 on RHEL-8 **Refer to Bhagyashris' notes below** * [C9 Master](http://dashboard-ci.tripleo.org/d/2tivP9BWz/component-pipeline?orgId=1): - baremetal + last_promotion: 2022-09-12 + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master): * [tempest failures](https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/913f5ca/logs/undercloud/var/log/tempest/stestr_results.html.gz) * [other tempest failures](https://logserver.rdoproject.org/57/44657/13/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/9a843f4/logs/undercloud/var/log/tempest/stestr_results.html.gz) - network + last_promotion: 2022-09-08 + ~~[`periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master/9ac8b78) + [`periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master): * older runs: - 2022-09-12: [LP#1989197](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/dc540aa/logs/undercloud/var/log/tempest/stestr_results.html.gz) - 2022-09-12: [LP#1989197](https://logserver.rdoproject.org/57/44657/11/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/b54f490/logs/undercloud/var/log/tempest/stestr_results.html.gz) * [LP#1989197](https://logserver.rdoproject.org/57/44657/12/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/7587e6a/logs/undercloud/var/log/tempest/stestr_results.html.gz) * [LP#1989197](https://logserver.rdoproject.org/57/44657/13/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/33472cf/logs/undercloud/var/log/tempest/stestr_results.html.gz) + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master): * [tempest failures](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/9b5506c) * [overcloud deploy failed with `ERROR: Can't run container rabbitmq_bootstrap`](https://logserver.rdoproject.org/57/44657/13/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/36e896f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - security + last_promotion: 2022-09-11 + ~~[`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master)~~: * ~~[TODO](https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master/5cb9cdd)~~ - [reruns](https://review.rdoproject.org/r/c/testproject/+/44657) * [C9 Wallaby](http://dashboard-ci.tripleo.org/d/UcMR8py7z/component-pipeline-wallaby-centos9?orgId=1): - common + last_promotion: 2022-09-10 + ~~[`periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/84/44684/7/check/periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby/5fa2da2) - tripleo + last_promotion: 2022-09-12 + [`periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby): * [overcloud deploy failed with `ERROR: Can't run container container-puppet-ovn_controller`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/150abbf/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [reruns](https://review.rdoproject.org/r/c/testproject/+/44658) * [C8 Wallaby](http://dashboard-ci.tripleo.org/d/ZLvFbT9Mz/component-pipeline-wallaby?orgId=1): - all **GREEN** 😱 - ~~[reruns](https://review.rdoproject.org/r/c/testproject/+/44659)~~ * [C8 Train](http://dashboard-ci.tripleo.org/d/aJeQpzVGz/component-pipeline-train?orgId=1): - network + last_promotion: 2022-09-09 + ~~[`periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train)~~ (not in criteria): * [TODO](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train/591113e) - tripleo + last_promotion: 2022-09-10 + [`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train`](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train): * [overcloud introspection failed](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/41aaac6/logs/undercloud/home/zuul/overcloud_introspect.log.txt.gz) - [reruns](https://review.rdoproject.org/r/c/testproject/+/44690) * [RHOS-17.0 on RHEL-9](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline?orgId=1): * **Refer to Bhagyashris' notes below** * [RHOS-16.2 on RHEL-8](http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline?orgId=1): * **Refer to Bhagyashris' notes below** ### Downstream: * rhos16.2 on rhel8: * Promotion: 13-sept-2022 * blocked due to sc010 re-running jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/ **PASSED** * rhos17.1 on rhel8 * Promotion: 12-sept-2022 * rhos17 on rhel9: * Promotion: 11-Sept-2022 * failed fs001 job so re-ran here : https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : **PASSED** * rhos17.1 on rhel9: * Promoted: 12-sept-2022 * blocked due full tempest api so re-ran here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : **PASSED** * jenkins job not triggerd: * Note : Pinged atila: - After discussion with atila we got to know that currently we don't have trigger jobs for 17.1 on rhel9 and 8 so i am adding trigger job for 17.1 on 9 and 8 here: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003 * ~~fix for trigger jekins component rhso17.1 on rhel9 jobs is here: https://code.engineering.redhat.com/gerrit/c/rhos-qe-jenkins/+/428056 ~~ ### Downstream component: ### Known Bugs * https://bugs.launchpad.net/tripleo/+bug/1960355 - > Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) - affecting OSP17.1 as well * https://bugs.launchpad.net/tripleo/+bug/1987092 :warning: **BLOCKER** :warning: - Pacemaker performance causes intermittent galera issues in loaded CI env - jobs fail on `Check Keystone public endpoint status` or in other places - [zuul/builds](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0) - [affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby](https://bugs.launchpad.net/tripleo/+bug/1987092/comments/18) * https://bugzilla.redhat.com/2105408 :warning: **BLOCKER** :warning: - > RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup) - > reproducible only on RHEL 8.2 and RHEL 8.4 ... RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6. - [latest status is tracked in cix card](https://trello.com/c/msIodUbD/2616-cixbz2105408osp162rhel8octavia-tempest-test-loadbalancerscenariotesttestloadbalanceripv4crud-failing-on-load-balancer-creation-t) - blocking integration lines of c9 master (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master`) and c9 wallaby (`periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby`) - [workaround (specifying `cpu_models` in libvirt)]( https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/425806/) is [currently being tested](https://code.engineering.redhat.com/gerrit/c/testproject/+/425432/) * https://bugs.launchpad.net/tripleo/+bug/1988514 - [Trello](https://trello.com/c/TAL1jQcB/2692-cixlp1988514tripleociproa-source-image-rejected-none-of-the-signatures-were-accepted-reasons-open-etc-pki-rpm-gpg-rpm-gpg-key-re) - On CIX board in lane `Waiting on external dependencies` * https://bugs.launchpad.net/tripleo/+bug/1987323 - > FIPS + Manila + Ceph Quincy - > ... known issue on the ceph side, ... already merged in master/quincy ... > 17.2.4 should solve this issue, so we'll check during the next promotion. - On CIX board in lane `Waiting on external dependencies` - Not a blocker * https://bugs.launchpad.net/tripleo/+bug/1988810 - Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788 * https://bugs.launchpad.net/tripleo/+bug/1989197 - > Tempest test `neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON` fails on `periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master` * https://bugs.launchpad.net/tripleo/+bug/1989341 - > Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch * https://bugs.launchpad.net/tripleo/+bug/1989247 - > overcloud deploy fails with `sudo: unable to send audit message: Operation not permitted` - > `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master`](https://logserver.rdoproject.org/56/44656/5/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/007c1da/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master`](https://logserver.rdoproject.org/57/44657/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master/3fb6e42/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * https://bugs.launchpad.net/tripleo/+bug/1989452 - > multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host ### Intermittent Failures * https://bugs.launchpad.net/tripleo/+bug/1987616 - > cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply' - no CIX card * https://bugs.launchpad.net/tripleo/+bug/1938987 - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train - > ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use') - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train`](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/bd0a94d/logs/overcloud-controller-2/var/log/containers/keystone/keystone.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train`](https://logserver.rdoproject.org/62/44662/6/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/5753657/logs/overcloud-controller-0/var/log/containers/keystone/keystone.log.txt.gz) * `TASK [freeipa-setup : Run the tripleo-ipa preparation script]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/205c81b/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/f940a55/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby/bfa6f03/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz) - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby`](https://logserver.rdoproject.org/40/44840/4/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby/57331f2/job-output.txt) * `Connection failure: The read operation timed out` in `TASK [Get current autoholds]` - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/c3a98c9/job-output.txt) * `Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap` (or others) - > this has different root causes, watch out for log details! - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/61/44661/3/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/4926640/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - [FAILURE in `tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby`](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby/df357c2/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) * `FATAL | Wait for puppet host configuration to finish | undercloud | error`... - `TASK [tripleo.operator.tripleo_undercloud_install : undercloud install]` in [job-output.txt](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/job-output.txt) - `puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages:` in [logs/undercloud/home/zuul/undercloud_install.log.txt.gz](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) - [POST_FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby`](https://logserver.rdoproject.org/58/44658/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/54bb44c/logs/undercloud/home/zuul/undercloud_install.log.txt.gz) * `Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds"` during overcloud deploy - [FAILURE in `periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby`](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/0a8588f/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz) ### Fixed ---