# Ruck Rover - 2022-06-17 - 2022-06-23

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/gw8f6Y3hQcatR_l2PhhY9g
###### Next RR notes: https://hackmd.io/9hv3vTNlST2rw014LSDqcg
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

## June 25

* Ceph patches merged in both master and wallaby, sc001 and sc004 should be green now.
* ~~Missing Ceph container in quay.rdoproject.or~~g:
* Master content provider is failing due to: "Failed to pull following containers: ['quay.rdoproject.org/tripleomastercentos9/daemon:current-ceph']
* https://quay.rdoproject.org/repository/tripleomastercentos9/daemon?tab=history
* Why _"current-ceph"_ tag was **deleted**?
* Need to check with Daniel, he is the responsible for the prune tags in quay.rdoproject.org. For now @arxcruz restore the tag.
* [content provider failure log](https://b452c7de58df0e34dc61-55c346d8a0a53f06bfa3e84321f61ba5.ssl.cf5.rackcdn.com/847142/4/check/tripleo-ci-centos-9-content-provider/5530d12/logs/quickstart_install.log)
* Is the fix supposed to be a manual restore of the missing tag?
* manually restored (twice Thursday and Saturday)
* should be fixed by https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/25329 - Merged Monday
* 27 Jun: we expend this to be fixed but will ping dpawlik if it is missing again

## June 24

### Patches requiring attention

* ~~https://bugs.launchpad.net/tripleo/+bug/1979665~~
* Patch below affected by this bug
* Saw it on tripleo-ci-centos-9-standalone
* recheck'd (known intermittent issue)

* ~~tripleo_cephadm_ceph_cli is undefined~~
* https://bugs.launchpad.net/tripleo/+bug/1979651
* fixed by https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2
* failed tripleo-ci-centos-9-standalone because of https://bugs.launchpad.net/tripleo/+bug/1979665
* tested in master TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847357
* sc001: Success
* sc004: Success
* tested in wallaby TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847329
* sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
* sc004: Success
* tested in wallaby periodic https://review.rdoproject.org/r/c/testproject/+/37973
* sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
* sc004: Success
* tested in master periodic https://review.rdoproject.org/r/c/testproject/+/36256
* sc001: Success
* sc004: Success~~
* __Updates__:
* ~~Master patch (847323) in **gate**~~
* ~~Wallaby patch (847257) -> **gate**~~

* ~~Periodic Job Failure (**gate**)~~
* ~~https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84623~~1
* ~~intermittently failed for LP 1979093 (above)~~
* ~~Tested by https://review.rdoproject.org/r/c/testproject/+/36256~~

## June 23

### Patches requiring attention

* https://bugs.launchpad.net/tripleo/+bug/1979665
* Patch below affected by this bug
* Saw it on tripleo-ci-centos-9-standalone
* recheck'd (known intermittent issue)

* tripleo_cephadm_ceph_cli is undefined
* https://bugs.launchpad.net/tripleo/+bug/1979651
* fixed by https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2
* failed tripleo-ci-centos-9-standalone because of https://bugs.launchpad.net/tripleo/+bug/1979665
* tested in master TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847357
* sc001: Success
* sc004: Success
* tested in wallaby TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847329
* sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
* sc004: Success
* tested in wallaby periodic https://review.rdoproject.org/r/c/testproject/+/37973
* sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
* sc004: Success
* tested in master periodic https://review.rdoproject.org/r/c/testproject/+/36256
* sc001: Success
* sc004: Success
* __Updates__:
* Master patch (847323) in **gate**

* https://bugs.launchpad.net/tripleo/+bug/1979093 Intermittent failure adding user 'ceph-admin'
* https://review.opendev.org/c/openstack/tripleo-ansible/+/846530
* needs to be tested in wallaby before merging in master

* ~~Periodic Job Failure (**gate**)~~
* ~~https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/84623~~1
* ~~intermittently failed for LP 1979093 (above)~~
* ~~Tested by https://review.rdoproject.org/r/c/testproject/+/36256~~

* ~~001/002 standalone failing on gnocchi bug~~ (fixed)
* https://bugs.launchpad.net/tripleo/+bug/1978997
* See the job run without pacemaker 2.1.3-2.el9
* 2.1.4 exists so we were downgrading back to the bad version
* ~~https://review.opendev.org/c/openstack/tripleo-common/+/846287 - revert https://review.opendev.org/c/openstack/tripleo-common/+/847166 fails~~
* ~~https://review.opendev.org/c/openstack/tripleo-common/+/847222 - Merged~~

### Active Bugs
* https://bugs.launchpad.net/tripleo/+bug/1979646 fs01 network wallaby component failing on neutron/dhcp-agent - reported today, need neutron/hardprov team involvement to resolve this
* https://bugs.launchpad.net/tripleo/+bug/1979665 - standalone network wallaby failing on network tempest tests
* https://bugs.launchpad.net/tripleo/+bug/1979546 - scenario010 kvm internal failing on octavia tests with error status
* https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing
* This is also affecting other jobs
* https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

* [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
* Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
- Track the health of fs01 and fs035

* https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9
- https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

* https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004
* merged

### Promotions downsteam

* osp16.2 rhel8 - today (23 June 2022)
* osp17 rhel8 - 20th June 2022
* osp17 rhel9 - 21st June 2022

## June 22

### Patches requiring attention

* 001/002 standalone failing on gnocchi bug
* https://bugs.launchpad.net/tripleo/+bug/1978997
* Innocent victim: https://review.opendev.org/c/openstack/puppet-tripleo/+/845854 2022-22-06 18:34:38 UTC
* Before we disable gnocchi https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474
* See the job run without pacemaker 2.1.3-2.el9
* conjecture: 2.1.4 exists now so we're downgrading back to the bad version
* https://review.opendev.org/c/openstack/tripleo-common/+/846287 - revert https://review.opendev.org/c/openstack/tripleo-common/+/847166 fails
* https://review.opendev.org/c/openstack/tripleo-common/+/847222 - passes - pls merge

* Wallaby: Ceph Failing on sc001 and sc004 (not master):
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/846999~~ merged
* https://review.opendev.org/c/openstack/tripleo-ansible/+/846940 backport (merged)
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/846950~~ merged
* https://review.opendev.org/c/openstack/tripleo-ansible/+/847163 backport (after 846940 merges) (merged)

* https://bugs.launchpad.net/tripleo/+bug/1979093 Intermittent failure adding user 'ceph-admin'
* https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

* Periodic Job Failure
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 (in check now)
* intermittently failed for LP 1979093 (above)

### Infra and missing ceph contianer

* mirror issue prevented merging of patches - see #opendev
* not here either: https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/
* fixed

* 'podman pull quay.rdoproject.org/tripleowallabycentos9/daemon:current-ceph' failed so arxcruz manually went in the repository to restore the tag

### Active Bugs
* https://bugs.launchpad.net/tripleo/+bug/1979546 - scenario010 kvm internal failing on octavia tests with error status
* https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing
* This is also affecting other jobs
* https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

* [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
* Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
- Track the health of fs01 and fs035

* https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9
- https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

* https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004
* merged

## June 21

* mirror issue preverting merging of patches - pls see #opendev
* not here either: https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/

### Patches requiring attention
* Need to merge https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004
* +w'd

* Wallaby is failing on sc001 and sc004 as well:
* https://review.opendev.org/c/openstack/tripleo-ansible/+/846950
* we need to backport this change ^ once it is merged

* Is https://review.rdoproject.org/r/c/testproject/+/36256 not using redis-bundle container with RPMs from https://review.opendev.org/c/openstack/tripleo-common/+/846287 ?

* This will confirm the fix for https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* So we don't need https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474 (LP 1978997 - WA disable gnocchi)
* If we add in the testptoject under vars add "build_container_images: true", will it stop periodic from pulling the usptream ceph?

### Active Bugs
* https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing
* This is also affecting other jobs
* https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

* [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
* Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
- Track the health of fs01 and fs035

* https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9
- https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

### Backlog (bugs fixed with workaround)
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

## June 20

### Active Bugs

* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

* https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

* [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
* Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
- Track the health of fs01 and fs035

### Backlog (bugs fixed with workaround)

* https://bugs.launchpad.net/tripleo/+bug/1978969 - tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 - merged

* https://bugs.launchpad.net/tripleo/+bug/1978929 -
resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different
- CI is unblocked for today, might come tomorrow also.

* https://bugzilla.redhat.com/show_bug.cgi?id=2097443 - [RHOS-17][RHEL-9]Overcloud image builds are failing with "libselinux-x-3.4-2" install/update errors
- Fix is merged now, waiting on tripleo wallaby component promotion and rhos-17 rhel-9 tripleo component promotion

## June 17

### Active Bugs
Phase1
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

Ceph
* https://bugs.launchpad.net/tripleo/+bug/1978956 - branch-override jobs are failing Generate Ceph Spec step
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159

* https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

MaybeCeph
* [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
* [fultonj] Asked mrunge for input

NotCeph
* https://bugs.launchpad.net/tripleo/+bug/1978996 - multinode-ipa job is failing with "No package openssl-3.0.1-5.el9 available"
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125/ - merged
* https://bugs.launchpad.net/tripleo/+bug/1978969 - tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 - merged

* https://bugs.launchpad.net/tripleo/+bug/1978929 -
resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different
- CI is unblocked for today, might come tomorrow also.

* https://bugs.launchpad.net/tripleo/+bug/1978458 -
cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'
- Need a proper bug report based on fs039 failures
- Hit a testproject just to confirm the root cause in case in future you see these jobs failing

* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
- Track the health of fs01 and fs035

* https://bugzilla.redhat.com/show_bug.cgi?id=2097443 - [RHOS-17][RHEL-9]Overcloud image builds are failing with "libselinux-x-3.4-2" install/update errors
- Fix is merged now, waiting on tripleo wallaby component promotion and rhos-17 rhel-9 tripleo component promotion

### CIXes to close or revisit
* https://trello.com/c/CQRTV134/2558-cixlp1977873tripleociproa-master-wallaby-deployments-are-failing-with-selinux-boolean-osenablevtpm-does-not-exist
* https://trello.com/c/9PnDFQ0i/2559-cixlp1977888tripleociproa-cics8train-diskimage-builder-requires-a-different-python-368-not-in-38
* https://trello.com/c/mLDYyWxC/2555-cixbz2094257osp17rhel9additional-partition-dev-sda4-in-rhel-guest-image-90-202204200x8664qcow2-guest-image-is-breaking-the-conve

### Jobs to actively monitor
- kvm internal, fs035, fs01, fs039

### Promotion Status

- Master :- 15 june
- wallaby c9:- 13 june
- wallaby c8:- 13 june
- train c8:- 13 june

Downstream
- RHOS-17 RHEL-9 - 13th
- RHOS-17 RHEL-8 - 15th
- RHOS-16 RHEL-8 - 14th - needs https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/ to promote - Pooja pls watch this

## Active Bugs

- Check handover bugs list

## 16 June

- Promote tripleo wallaby component to get https://review.opendev.org/c/openstack/diskimage-builder/+/846052
- https://review.rdoproject.org/r/c/testproject/+/43621

- cs9 wallaby promotion re-run fs039 mirror issue: https://review.rdoproject.org/r/c/testproject/+/43622
- train promotion: https://review.rdoproject.org/r/c/testproject/+/43623

### component pipelines
- Rekicking c9 components:- https://review.rdoproject.org/r/c/testproject/+/40171
- Rekicking c9 integration line failures:- https://review.rdoproject.org/r/c/testproject/+/40102

## Ignore everything below this line
## 15 June

## Active Bugs

* ~~https://bugs.launchpad.net/tripleo/+bug/1978456~~ - cs9 image build issue in ovb jobs
* https://bugs.launchpad.net/tripleo/+bug/1978458 -
cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'
* ~~https://bugs.launchpad.net/tripleo/+bug/1978482~~ -
Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[keystone_wsgi]
* ~~https://bugs.launchpad.net/tripleo/+bug/1972163~~ - cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
* ~~https://bugs.launchpad.net/tripleo/+bug/1978319~~ -
cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2096125~~ [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
* https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

cs9 wallaby tempest component
* ~~https://bugs.launchpad.net/tripleo/+bug/1978553~~ - ExternalGatewayForFloatingIPNotFound: External network is not reachable from subnet Therefore, cannot associate Port with a Floating IP
* ~~https://bugs.launchpad.net/tripleo/+bug/1978556~~ -
Multiple tempest tests failure with ValueError: Unable to setup floating IP for validation: port not found on tenant network
RHOS-17 RHEL-8 promoted today!

### component pipelines
* master(compute, security)
- compute line is running
- validation needs a rerun - rekcicked line
- security - fs39 is failing - rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40083
* c9 wallaby(compute)- got promoted yesterday
* c8 wallaby(network, tripleo)
- https://review.rdoproject.org/r/c/testproject/+/40932
- https://review.rdoproject.org/r/c/testproject/+/42595

### CIX cards we can clear today
- https://trello.com/c/5nFneypz/2566-cixlp1978456tripleociproa-cs9-fs02-ovb-job-is-failing-at-image-build-dib-run-parts-ignoring-non-executable-files-05-selinux-9-st
- https://trello.com/c/sZFnUP6h/2569-cixlp1978131tripleociproa-tripleo-puppet-fails-to-set-up-keystone-container-during-standalone-deployment
- https://trello.com/c/EuVXuVVO/2570-cixlp1978553tripleociproa-externalgatewayforfloatingipnotfound-external-network-is-not-reachable-from-subnet-therefore-cannot-as
- https://trello.com/c/lQNtKdRh/2571-cixlp1978556tripleociproa-multiple-tempest-tests-failure-with-valueerror-unable-to-setup-floating-ip-for-validation-port-not-fou
- https://trello.com/c/oT1GERap/2568-cixlp1978482tripleociproa-evaluation-error-error-while-evaluating-a-resource-statement-apachevhostkeystonewsgi
- https://trello.com/c/PGuOikZG/2565-cixbz2096125rhos-17rhel-8overcloud-image-build-jobs-are-failing-openstack-ironic-python-agent-builder-1280-020220327061952e0b51e
- https://trello.com/c/wiOeCsdg/2563-cixlp1978319tripleociproa-cs8-octavia-cloudops-tempest-common-network-tripleo-validation-wallaby-fails-with-error-running-contai
- https://trello.com/c/2VHETeKz/2554-cixlp1977716tripleociproa-the-conditional-check-negativeresults-in-namevalue-failed-during-setfact

## 14 June

### components pipeline
* master (validation, tripleo, security, baremetal, network)
- Testproject:- https://review.rdoproject.org/r/c/testproject/+/40083
- tripleo..https://bugs.launchpad.net/tripleo/+bug/1978482
* wallaby c8(tripleo, network)
- [network]Testproject:- https://review.rdoproject.org/r/c/testproject/+/40932
- [tripleo]Testproject:- https://review.rdoproject.org/r/c/testproject/+/42595
* All components promoted

* Chasing master and train: https://review.rdoproject.org/r/c/testproject/+/39960
* Chasing master and rhos17 on rhel8 - https://code.engineering.redhat.com/gerrit/c/testproject/+/398466
* Issue with https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone&skip=0 - fixed now

## 13 June

#### Promotions
* wallaby c9 promoted
* wallaby c8 should promote with current testproject run
* rhos-17 on rhel 9 should promote with current testproject run
* working on promoting wallaby c9 components (promoted so far: security, validation, baremetal, network) tempest and tripleo are in rerun

#### patches to clear fs02//fs01 / image build
* 845573: Make sure 05-selinux-9-stream is executable | https://review.opendev.org/c/openstack/tripleo-ci/+/845573
* testproject: https://review.rdoproject.org/r/c/testproject/+/43586

#### Tempest patches
* 43404: Backport wait_until_sshable_pingable patches | https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/43404
* https://code.engineering.redhat.com/gerrit/q/topic:wait_until_sshable_pingable
* testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/414808
* Ping mkopec or tosky to get it merged.

#### CS9 Master:

* https://bugs.launchpad.net/tripleo/+bug/1978456 -
cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded... iPXE boot failed, retrying... .'
* https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

* https://bugs.launchpad.net/tripleo/+bug/1978482 -
cs9-multinode/multinode-ipa/standalone/multinode-tripleo-master-validation/sc004-standalone/sc000-multinode-oooq-container-updates tripleo-master fails with RuntimeError: Ansible execution failed

## Active Bugs
### CS9 Wallaby / Master
* https://bugs.launchpad.net/tripleo/+bug/1977873
* master/wallaby deployments are failing with "SELinux boolean os_enable_vtpm does not exist."
* ~~https://review.opendev.org/c/openstack/diskimage-builder/+/845189/~~
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137/
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/845

1 - revert of workaround
* issue seen in [compute component- 8th june log](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/a39f0a8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
* [compute component - 9th june](https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/a5ee5d5/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
* waiting for fresh run

### CS9 Master
* https://bugs.launchpad.net/tripleo/+bug/1972163
* cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
* https://bugs.launchpad.net/tripleo/+bug/1978298
* Rekicked master component jobs here:- https://review.rdoproject.org/r/c/testproject/+/42693

### CS9 Wallaby
* https://bugs.launchpad.net/tripleo/+bug/1977716
* The conditional check ''negative_results' in name.value ' failed during set_fact
* ~~fix: https://review.opendev.org/c/openstack/validations-common/+/844823~~
* test project: https://review.rdoproject.org/r/c/testproject/+/42319
* Follow up on this

* https://bugs.launchpad.net/tripleo/+bug/1964940
* Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* verify this bug

### CS8 Wallaby
* https://bugs.launchpad.net/tripleo/+bug/1978319
* cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
* testproject: https://review.rdoproject.org/r/c/testproject/+/40101
* Assigned to @frenzyfriday
* fire a testproj and confirm

### CS8 Train
* https://bugs.launchpad.net/tripleo/+bug/1977888
* fix: https://review.opendev.org/c/openstack/tripleo-ci/+/845081
* test project: https://review.rdoproject.org/r/c/testproject/+/42319
* chasing fs035: https://review.rdoproject.org/r/c/testproject/+/36255

### RHEL 9 RHOS17
* https://bugzilla.redhat.com/show_bug.cgi?id=2094257
* ~~fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952~~
* test project: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027
* Rekicked 17 rhel9 jobs here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398340
* cross-verify

### RHEL 8 RHOS17
* https://bugzilla.redhat.com/show_bug.cgi?id=2096125 [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
* https://review.opendev.org/c/openstack/tripleo-ci/+/845544
* https://code.engineering.redhat.com/gerrit/c/testproject/+/414668

### Other
* https://bugs.launchpad.net/bugs/1971465
* fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

## 13 June

#### CS9 Master:

* https://bugs.launchpad.net/tripleo/+bug/1978456 -
cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded... iPXE boot failed, retrying... .'
* https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

## Fri, Jun 10
### Promotions
#### CS9 Wallaby / Master
* https://review.rdoproject.org/r/c/testproject/+/36254
* skip fs002: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43579
* components: https://review.rdoproject.org/r/c/testproject/+/4239
* security component is failing due to libselinux 3.4-1 issues.

#### CS8 Wallaby
* https://review.rdoproject.org/r/c/testproject/+/40101

#### RHEL-8 RHOS-16.2
* Chasing two hashes: https://code.engineering.redhat.com/gerrit/c/testproject/+/398466 (last promo 06/06)

## Thu Jun 09

## Promotions
### CS9 Wallaby / Master
* https://review.rdoproject.org/r/c/testproject/+/36254/
* https://review.rdoproject.org/r/c/testproject/+/42374
* waiting to see what tempest on fs001 looks like before promoting. Tons of failures there atm
* master also failng fs039 and fs064 and internal kvm
* wallaby only fs001
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43570 - to promote wallaby c9

### 13 June

https://bugs.launchpad.net/tripleo/+bug/1978298
- reckick the job

-------------

## Transient Bugs
### fs039 - series of unknown issues, difficult to triage [1]
They all seem to be unrelated, but they're causing havoc to the line.

#### "Can't run container"
[2]
2022-06-09 22:58:35 | 2022-06-09 22:58:35.006643 | | WARNING | ERROR: Can't run container nova_api_ensure_default_cells
[...]
2022-06-09 22:58:35 | 2022-06-09 22:58:35.010171 | | WARNING | ERROR: Can't run container placement_api_db_sync
[...]

#### Internal Server Error Keystone

[3]
2022-06-10 15:08:13 | 2022-06-10 15:08:13.341227 | fa163e2d-7640-c8bc-5072-00000000a20c | TIMING | tripleo_keystone_resources : Create identity service | undercloud | 0:29:54.730355 | 1.67s
2022-06-10 15:08:13 | 2022-06-10 15:08:13.352842 | fa163e2d-7640-c8bc-5072-00000000a20d | TASK | Create identity public endpoint
2022-06-10 15:08:17 | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)

[4]
[Fri Jun 10 15:08:17.218323 2022] [wsgi:error] [pid 17:tid 38] [remote 172.17.0.184:33542] mod_wsgi (pid=17): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/keystone'.

#### SSH Permission denied

[5]
2022-06-10 18:05:50 | 2022-06-10 18:05:50.412897 | fa163e10-0939-a651-5d12-000000001759 | FATAL | Run tripleo_os_net_config_module with network_config | overcloud-controller-2 | error={"msg": "Data could not be sent to remote host \"192.168.24.30\". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.30' (ED25519) to the list of known hosts.\r\nheat-admin@192.168.24.30: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive).\r\n"}

#### Cannot download ansible-macros
[6]
2022-06-09 15:03:03 | Error: Error downloading packages:
2022-06-09 15:03:03 | ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success

#### Failed to download packages: mod_lua-2.4.51-8
[7]
2022-06-06 22:01:03.474740 | fa163ec0-bb0e-7746-1f99-000000000cb2 | FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: mod_lua-2.4.51-8.el9.x86_64: Cannot download, all mirrors were already tried without success", "results": []}

#### overcloud-2 didn't start
[8]
One of 3 overcloud nodes didn't start.

[1]: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master

[2]: https://logserver.rdoproject.org/60/39960/51/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/a89c55a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[3]: https://logserver.rdoproject.org/d4/d49b5b1345bb354904804ec1a52f6fcbd5415e0b/openstack-periodic-integration-main/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/f56d509/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[4]: https://logserver.rdoproject.org/d4/d49b5b1345bb354904804ec1a52f6fcbd5415e0b/openstack-periodic-integration-main/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/f56d509/logs/overcloud-controller-2/var/log/containers/httpd/keystone/keystone_wsgi_error_ssl.log.txt.gz
[5]: https://logserver.rdoproject.org/60/39960/53/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/d3b7fcc/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
[6]: https://logserver.rdoproject.org/67/41367/9/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/92e2dde/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz
[7]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/592d56e/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
[8]: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/b8e5b63/logs/baremetal_2-console.log