---
# System prepended metadata

title: Ruck Rover 2023-04-21 - 2023-04-27
tags: [ruck_rover]

---

# Ruck Rover 2023-04-21 - 2023-04-27
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2023](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
    * For now, we just look (and investigate) if a test has multiple failures across different PRs


[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/IsG12J3ASCGKBbITjPDU8Q ([index there](https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ))

##### ruck & rover: Bhagyashri
##### ruck & rover alias: @ruckroverciframework

:::warning
### Ongoing bugs
* https://bugs.launchpad.net/tripleo/+bug/2016592 - scenario003 failing on designate tests - master, just this job is failing in order to promote, so it might worth take a look

### Closed bugs
* https://bugs.launchpad.net/tripleo/+bug/2017631
    * fix: https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/48371
      * Note:  even after merging ^ fix the issue is still coming
    * cix: https://trello.com/c/ZEGI16LT/2962-cixlp2017631tripleociproa-centos9-stable-wallaby-error-could-not-find-specified-file-in-role-tasks-runyml 
    * Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/881184
    * Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/881692
      * testing here: https://review.opendev.org/c/openstack/tripleo-ansible/+/881560 

### Infos for upcoming rucks
* https://issues.redhat.com/browse/RELDEL-2128 - fdp-pending repo for rhos-17.1 will be removed in future, release-delivery team will let us know
  about replacement
* https://issues.redhat.com/browse/RELDEL-2132 - Updating OSP 17.1 and 18.0 moving from Ceph 5 to Ceph 6
    * https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/442071
    * https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/442065
 
#### Upstream promotion
    CentoS 9 Master: 2023-04-07 (removing https://lists.openstack.org/pipermail/openstack-discuss/2023-March/033080.html)
    CentOS 9 Zed: 2023-04-27
    CentOS 9 Wallaby: 2023-04-24
    CentOS 8 Wallaby: 2023-04-27
    CentOS 8 Train: 2023-04-26 (no new content)

#### Downstream Promotion
    RHEL 9 RHOS 18.0: 2023-04-23 (no new content)
    RHEL 9 RHOS 17.1: 2023-04-26   
    RHEL 9 RHOS 17.0: 2023-04-25 (no new content)
    RHEL 8 RHOS 17.1: 2023-04-26 (no new content)
    RHEL 8 RHOS 16.2: 2023-04-27 
    
#### Podifed Container Build Lines
    openstack-periodic-container-master-centos9
    openstack-periodic-container-antelope-centos9
    openstack-periodic-container-zed-centos9
:::

## Handover notes



###  Recent failures:

  * Upstream:
    * periodic-centos-9-podified-edpm-integration - zed, master and antilope jobs are failing due to tempest test failure plz check: https://logserver.rdoproject.org/openstack-periodic-container-master-centos9/review.rdoproject.org/rdo-jobs/master/periodic-centos-9-podified-edpm-integration-master/4a4ba19/job-output.txt

  * Downstream:
    * rhos18 on rhel9:
      * container build push is failing: https://sf.hosted.upshift.rdu2.redhat.com/logs/20/20fc32bf4236ec60ca3d70d48563ca0e0ed54e5a/openstack-periodic-integration-rhos-18-rhel9/periodic-container-tcib-build-push-upload-rhel-9-rhos-18/ac91869/da5c2491-9b09-4cb3-b10f-88f979af87fc/base/ovn-bgp-agent/ovn-bgp-agent-build.log
    * rhos17.1 on rhel9: mixed rhel job is failing: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs 
      * log: https://sf.hosted.upshift.rdu2.redhat.com/logs/03/429803/23/check/periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1/62b9ba6/logs/undercloud/home/zuul/overcloud_deploy.log overcloud deploy is failing 


## Thur 26 Apr 2023




### Upstream:

  * test failing zed job: https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 wallaby and zed job
    * periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-zed https://review.rdoproject.org/zuul/build/46ecd99da3a64c369c340aa79d6d7df9 : SUCCESS in 2h 39m 57s 

  * **Component**: 
    * re-testing failing component jobs:  https://review.rdoproject.org/r/c/testproject/+/47232 [DNM] Test failing component jobs

### Downstream:
  * rhos17.1 on rhel9 : re-running failing jobs https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs


    periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/62b9ba64a93a4b6e91c4e3db2ff3e1ed : FAILURE in 1h 03m 40s


  * rhos18 on rhel9 container builds is failing: https://sf.hosted.upshift.rdu2.redhat.com/logs/20/20fc32bf4236ec60ca3d70d48563ca0e0ed54e5a/openstack-periodic-integration-rhos-18-rhel9/periodic-container-tcib-build-push-upload-rhel-9-rhos-18/ac91869/da5c2491-9b09-4cb3-b10f-88f979af87fc/base/ovn-bgp-agent/ovn-bgp-agent-build.log



  * **COMPONENTS**:


## Wed 26 Apr 2023


### Upstream:

  * cs9 wallaby: https://bugs.launchpad.net/tripleo/+bug/2017742
    *  looks like this change https://review.opendev.org/c/openstack/puppet-ironic/+/879087 is causing this issue.
    *  revert the patch ^ https://review.opendev.org/c/openstack/puppet-ironic/+/881439 for testing https://review.rdoproject.org/r/c/testproject/+/44874
    *  Fix is here : https://review.opendev.org/c/openstack/tripleo-common/+/881438 tp: https://review.rdoproject.org/r/c/testproject/+/44874 

  * **Component**
    

### Downstream:
  * 17.1 on rhel9 mixed rhel job is failing retesting here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs 
    * periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/0d71152ef26d40538ef937e08aa0b048 : SUCCESS in 1h 44m 51s 


  * **COMPONENTS**:


## Tuesday 25 Apr 2023


### Upstream:

  * stable wallaby cs9 blocker: https://bugs.launchpad.net/tripleo/+bug/2017631
    * fix: https://review.rdoproject.org/r/c/openstack/tripleo-ansible-distgit/+/48371
    * cix: https://trello.com/c/ZEGI16LT/2962-cixlp2017631tripleociproa-centos9-stable-wallaby-error-could-not-find-specified-file-in-role-tasks-runyml 
    * tp: https://review.rdoproject.org/r/c/testproject/+/37916 DNM - Rechecking jobs - cs9 wallaby
  *  https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job  - **PASSED**
  *  

  * **Component**: 
    * re-testing failing component jobs :  https://review.rdoproject.org/r/c/testproject/+/47232 [DNM] Test failing component jobs 

### Downstream:
  * 


  * **COMPONENTS**:

## Monday 24 Apr 2023


### Upstream:
  *  https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job **Result: All jobs passed**
  *  Looks like we have a wallaby issue with ceph  - just noticed it on the latest run - pls check in your morning


### Downstream:
  * 


  * **COMPONENTS**:
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/431159 [DNM] Test failing component jobs
      * periodic-tripleo-ci-rhel-9-scenario001-standalone-common-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/6a18d055ebf54ef89386fad2fcc3b86e : SUCCESS in 1h 20m 40s 



## Thursday 21 Apr 2023


### Upstarem:
  * cs9 zed:
     * testing failing jobs here https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs9 zed job
       * **RESULT:** periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-zed https://review.rdoproject.org/zuul/build/003c537d06834ca6aca6498d4c5ec223 : SUCCESS in 2h 09m 11s
periodic-tripleo-ci-centos-9-scenario003-standalone-zed https://review.rdoproject.org/zuul/build/451364fefa3844f49c16eb4b967f2a32 : SUCCESS in 54m 03s 


### Downstream:
  * rhos17.1 on rhel9:
    * fs001 and sc010 failing re-eunning job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 
      * Result: periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/073cc1a0acac4d909a5a287ab3995ded : SUCCESS in 1h 22m 28s
periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/8bcebf11d36b4db39d73239d91971fe6 : SUCCESS in 2h 38m 31s 


  * **COMPONENTS**:
    * rhos17.1 on rhel9 network compnent (tempest) job is failing : https://code.engineering.redhat.com/gerrit/c/testproject/+/431159 
      * **Result:** periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-network-rhos-17.1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/21927c3b6e214f0aa9dc1f3f9edde341 : SUCCESS in 2h 02m 12s