# Ruck Rover 2023-04-21 - 2023-04-27
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2023](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/IsG12J3ASCGKBbITjPDU8Q ([index there](https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ))

##### ruck & rover: Bhagyashri
##### ruck & rover alias: @ruckroverciframework

:::warning
### Ongoing bugs
* https://bugs.launchpad.net/tripleo/+bug/2016592 - scenario003 failing on designate tests - master, just this job is failing in order to promote, so it might worth take a look

### Closed bugs
* https://bugs.launchpad.net/tripleo/+bug/2017631
* fix: https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/48371
* Note: even after merging ^ fix the issue is still coming
* cix: https://trello.com/c/ZEGI16LT/2962-cixlp2017631tripleociproa-centos9-stable-wallaby-error-could-not-find-specified-file-in-role-tasks-runyml
* Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/881184
* Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/881692
* testing here: https://review.opendev.org/c/openstack/tripleo-ansible/+/881560

### Infos for upcoming rucks
* https://issues.redhat.com/browse/RELDEL-2128 - fdp-pending repo for rhos-17.1 will be removed in future, release-delivery team will let us know
about replacement
* https://issues.redhat.com/browse/RELDEL-2132 - Updating OSP 17.1 and 18.0 moving from Ceph 5 to Ceph 6
* https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/442071
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/442065

#### Upstream promotion
CentoS 9 Master: 2023-04-07 (removing https://lists.openstack.org/pipermail/openstack-discuss/2023-March/033080.html)
CentOS 9 Zed: 2023-04-27
CentOS 9 Wallaby: 2023-04-24
CentOS 8 Wallaby: 2023-04-27
CentOS 8 Train: 2023-04-26 (no new content)

#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-04-23 (no new content)
RHEL 9 RHOS 17.1: 2023-04-26
RHEL 9 RHOS 17.0: 2023-04-25 (no new content)
RHEL 8 RHOS 17.1: 2023-04-26 (no new content)
RHEL 8 RHOS 16.2: 2023-04-27

#### Podifed Container Build Lines
openstack-periodic-container-master-centos9
openstack-periodic-container-antelope-centos9
openstack-periodic-container-zed-centos9
:::

## Handover notes

### Recent failures:

* Upstream:
* periodic-centos-9-podified-edpm-integration - zed, master and antilope jobs are failing due to tempest test failure plz check: https://logserver.rdoproject.org/openstack-periodic-container-master-centos9/review.rdoproject.org/rdo-jobs/master/periodic-centos-9-podified-edpm-integration-master/4a4ba19/job-output.txt

* Downstream:
* rhos18 on rhel9:
* container build push is failing: https://sf.hosted.upshift.rdu2.redhat.com/logs/20/20fc32bf4236ec60ca3d70d48563ca0e0ed54e5a/openstack-periodic-integration-rhos-18-rhel9/periodic-container-tcib-build-push-upload-rhel-9-rhos-18/ac91869/da5c2491-9b09-4cb3-b10f-88f979af87fc/base/ovn-bgp-agent/ovn-bgp-agent-build.log
* rhos17.1 on rhel9: mixed rhel job is failing: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs
* log: https://sf.hosted.upshift.rdu2.redhat.com/logs/03/429803/23/check/periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1/62b9ba6/logs/undercloud/home/zuul/overcloud_deploy.log overcloud deploy is failing

## Thur 26 Apr 2023

### Upstream:

* test failing zed job: https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 wallaby and zed job
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-zed https://review.rdoproject.org/zuul/build/46ecd99da3a64c369c340aa79d6d7df9 : SUCCESS in 2h 39m 57s

* **Component**:
* re-testing failing component jobs: https://review.rdoproject.org/r/c/testproject/+/47232 [DNM] Test failing component jobs

### Downstream:
* rhos17.1 on rhel9 : re-running failing jobs https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs

periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/62b9ba64a93a4b6e91c4e3db2ff3e1ed : FAILURE in 1h 03m 40s

* rhos18 on rhel9 container builds is failing: https://sf.hosted.upshift.rdu2.redhat.com/logs/20/20fc32bf4236ec60ca3d70d48563ca0e0ed54e5a/openstack-periodic-integration-rhos-18-rhel9/periodic-container-tcib-build-push-upload-rhel-9-rhos-18/ac91869/da5c2491-9b09-4cb3-b10f-88f979af87fc/base/ovn-bgp-agent/ovn-bgp-agent-build.log

* **COMPONENTS**:

## Wed 26 Apr 2023

### Upstream:

* cs9 wallaby: https://bugs.launchpad.net/tripleo/+bug/2017742
* looks like this change https://review.opendev.org/c/openstack/puppet-ironic/+/879087 is causing this issue.
* revert the patch ^ https://review.opendev.org/c/openstack/puppet-ironic/+/881439 for testing https://review.rdoproject.org/r/c/testproject/+/44874
* Fix is here : https://review.opendev.org/c/openstack/tripleo-common/+/881438 tp: https://review.rdoproject.org/r/c/testproject/+/44874

* **Component**

### Downstream:
* 17.1 on rhel9 mixed rhel job is failing retesting here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs
* periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/0d71152ef26d40538ef937e08aa0b048 : SUCCESS in 1h 44m 51s

* **COMPONENTS**:

## Tuesday 25 Apr 2023

### Upstream:

* stable wallaby cs9 blocker: https://bugs.launchpad.net/tripleo/+bug/2017631
* fix: https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/48371
* cix: https://trello.com/c/ZEGI16LT/2962-cixlp2017631tripleociproa-centos9-stable-wallaby-error-could-not-find-specified-file-in-role-tasks-runyml
* tp: https://review.rdoproject.org/r/c/testproject/+/37916 DNM - Rechecking jobs - cs9 wallaby
* https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job - **PASSED**
*

* **Component**:
* re-testing failing component jobs : https://review.rdoproject.org/r/c/testproject/+/47232 [DNM] Test failing component jobs

### Downstream:
*

* **COMPONENTS**:

## Monday 24 Apr 2023

### Upstream:
* https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job **Result: All jobs passed**
* Looks like we have a wallaby issue with ceph - just noticed it on the latest run - pls check in your morning

### Downstream:
*

* **COMPONENTS**:
* https://code.engineering.redhat.com/gerrit/c/testproject/+/431159 [DNM] Test failing component jobs
* periodic-tripleo-ci-rhel-9-scenario001-standalone-common-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/6a18d055ebf54ef89386fad2fcc3b86e : SUCCESS in 1h 20m 40s

## Thursday 21 Apr 2023

### Upstarem:
* cs9 zed:
* testing failing jobs here https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job
* **RESULT:** periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-zed https://review.rdoproject.org/zuul/build/003c537d06834ca6aca6498d4c5ec223 : SUCCESS in 2h 09m 11s
periodic-tripleo-ci-centos-9-scenario003-standalone-zed https://review.rdoproject.org/zuul/build/451364fefa3844f49c16eb4b967f2a32 : SUCCESS in 54m 03s

### Downstream:
* rhos17.1 on rhel9:
* fs001 and sc010 failing re-eunning job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/429803
* Result: periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/073cc1a0acac4d909a5a287ab3995ded : SUCCESS in 1h 22m 28s
periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/8bcebf11d36b4db39d73239d91971fe6 : SUCCESS in 2h 38m 31s

* **COMPONENTS**:
* rhos17.1 on rhel9 network compnent (tempest) job is failing : https://code.engineering.redhat.com/gerrit/c/testproject/+/431159
* **Result:** periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-network-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/21927c3b6e214f0aa9dc1f3f9edde341 : SUCCESS in 2h 02m 12s