[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
* [Operators - Check/Gate status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*)
* [pre-commit jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check&state=failure)
* [pre-commit jobs status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check)
* [unit tests jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-main-unit&state=failure)
* [kuttl tests Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-kuttl&state=failure)
* [Tempest Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-tempest&state=failure)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Uakv4hhKQxO9D4JKlbMUEw

##### ruck & rover: Cedric
##### ruck & rover alias: @ruckroverciframework

https://review.rdoproject.org/r/c/testproject/+/48811
https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/884993

:::info
**Fixes and Improvements needed**

* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
* https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
* periodic build-image jobs were failing because of that

:::

:::danger

**OPEN BUGS**

:::

:::info
### Promotion status

#### Upstream promotion

* Podified Master: 2023-06-28
* Podified Antelope: 2023-06-28
* CentOS 9 Wallaby 2023-06-26
* needs investigations
* CentOS 8 Wallaby 2023-06-28
* CentOS 8 Train 2023-06-29

#### Downstream Promotion
* Podified RHEL 9 RHOS 18.0: 2023-06-28
* RHEL 9 RHOS 17.1: 2023-06-28
* RHEL 9 RHOS 17.0: 2023-06-22
* NNH after 3876eba14585e0a14a0597a43c5e5e6c
* RHEL 8 RHOS 17.1: 2023-06-28
* RHEL 8 RHOS 16.2: 2023-06-28
:::

## Handover notes

## Jun 29, 2023
* Wallaby/CS9 has 7 failures, need to investigate to see if linked to current [vexx fire](https://trello.com/c/KYqChvHq)
* periodic-tripleo-ci-centos-9-scenario002-standalone-wallaby:
* timeout during UC deploy
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945450
* duration seems to be 45 minutes (deploy time) - wrong timeout setting ?!
* periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby:
* timeout during UC deploy
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945440
* duration seems to be 45 minutes (deploy time) - wrong timeout setting ?!
* periodic-tripleo-ci-centos-9-8-multinode-mixed-os:
* overcloud deploy failed
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003946101
* timeout during external step_4, related to keystone resource checks
* periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-wallaby:
* failed at "tripleo-upgrade : collect logs on the overcloud for the current stage" task
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945371

TASK [not all podman images updated] *******************************************
fatal: [np0003945389]: FAILED! => {"changed": false, "msg": "Not all images updated"}
fatal: [undercloud]: FAILED! => {"changed": false, "msg": "Not all images updated"}

*
* periodic-tripleo-ci-centos-9-scenario012-standalone-wallaby:
* Timeout during UC deploy
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945532
* deploy crashed after ~2h (checking keystone resources)
* periodic-tripleo-ci-centos-9-scenario004-standalone-wallaby:
* Timeout during UC deploy
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945496
* deploy crashed after ~1h (running ceph deploy)
* periodic-tripleo-ci-centos-9-standalone-wallaby:
* Timeout during UC deploy
* ibm-bm3-nodepool-cloud-centos-9-stream-tripleo - np0003945356
* deploy crashed after ~1:30 (puppet for cinder_volume)
* wallaby rerun: https://review.rdoproject.org/r/c/testproject/+/47875

## Jun 28, 2023
* Wallaby on CS9 is showing several failures (9 jobs), preventing promotion. Needs investigations.
* **re-kicked the whole line at 11:01am CET (line usually runs between 7-9pm UTC).**
* CS9 image CIX: https://bugs.launchpad.net/tripleo/+bug/2025224
* fixed
* periodic-tripleo-ci-centos-9-scenario004-standalone-wallaby ended on a timeout (step_4 reached)
* periodic-tripleo-ci-centos-9-scenario002-standalone-wallaby ended on a timeout (step_2 reached)
* periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby ended with chrony unable to sync time (running on RH internal infra)
* periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-wallaby failed on "repo-setup : Setup repos on live host" - package conflict. Reopening existing LP: https://bugs.launchpad.net/tripleo/+bug/2024882

## Jun 27, 2023
* train line
* no new hash, but 3 jobs failed on tempest. Taking note for future reference "if there's a trend"
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-train
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-internal-train
* wallaby lines
* new hash, promotion failed
* periodic-tripleo-ci-centos-9-8-multinode-mixed-os hit a timeout
* rekick: https://review.rdoproject.org/r/c/testproject/+/49040
* rekick is green, we should get wallaby on cs8 promoted shortly

## Jun 26, 2023
A Monday, 2 issues rasied as CIX, solved over the day.

## Jun 23, 2023
* new bug 2023-06-22 17:22:47.684950 | primary | 2023-06-22 17:22:47 | Problem: package qemu-img-17:8.0.0-4.el9.x86_64 from quickstart-centos-appstreams requires liburing.so.1()(64bit), but none of the providers can be installed
* https://bugs.launchpad.net/tripleo/+bug/2024882
* chasing train with remote: https://review.rdoproject.org/r/c/testproject/+/48999 fs39 f425c8e341de024e10d1105bda17b266