---
# System prepended metadata

title: Ruck Rover 2023-05-26 - 2023-06-02
tags: [ruck_rover]

---

# Ruck Rover 2023-05-26 - 2023-06-02

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
    * For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/0iGbL43uTPa9YtjWtdWfjQ

##### ruck & rover: soniya29 (svyas)
##### ruck & rover alias: @ruckroverciframework

:::info
**Fixes and Improvements needed**


* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
    * https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
    * periodic build-image jobs were failing because of that

:::

:::danger

**BUGS**

* https://bugzilla.redhat.com/show_bug.cgi?id=2209633 Tripleo Standalone jobs with ceph 6.0 failing with "msg": "JSONDecodeError: Expecting value: line 1 column 1 (char 0)"} and PermissionError: [Errno 13] Permission denied: '/var/lib/kolla/config_files/src-ceph/ceph.conf'
    * Upstream Fixes
        * ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/883825~~
        * https://review.opendev.org/c/openstack/tripleo-ansible/+/884332
    * Downstream 
        * ~~https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/443461~~
        * https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/443477
    * Tripleo-ansible fixes are failing , tht fixes got merged

* https://bugs.launchpad.net/tripleo/+bug/2019192 train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0
        
    * rekicked here 

* https://bugzilla.redhat.com/show_bug.cgi?id=2210225  17.1 Mixed rhel job fails with error 'Can't run container container-puppet-neutron

* Downstream is hitting RETRY_LIMIT issues - please check again in morning
:::
## Thursday 1/06/2023
#### Upstream promotion

* Master: Green 2023-06-01
* Antelope: Green 2023-05-
    * periodic-centos-9-podified-edpm-integration-antelope is failing 
    * https://logserver.rdoproject.org/a0/a03f8684d8e855ba4c92c18bf827604e7596094b/openstack-periodic-container-antelope-centos9/periodic-centos-9-podified-edpm-integration-antelope/4326458/job-output.txt
* CentOS 9 Wallaby 2023-06-01
* CentOS 8 Wallaby 2023-06-01
* CentOS 8 Train 2023-05-31
    * https://review.rdoproject.org/r/c/testproject/+/46082

### Upstream components:
    
#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
    * no new hash 
* RHEL 9 RHOS 17.1: RED 2023-05-22
    * fs20 internal and full_tempest_Scenario - https://bugzilla.redhat.com/show_bug.cgi?id=2211604
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443590

* RHEL 9 RHOS 17.0: 2023-05-26
    * no new hash after 26/05
* RHEL 8 RHOS 17.1: 2023-06-01
    
* RHEL 8 RHOS 16.2: 2023-05-25
    * no new hash after 25
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/430284

#### Downstream components
17.1 rhel9 :- network
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
    * sc10 is still blocked on ceph issue
    

     



































## Wednesday 31/05/2023
#### Upstream promotion

* Master: Green 2023-05-31
* Antelope: Green 2023-05-31
* CentOS 9 Wallaby 2023-05-29
    * https://review.rdoproject.org/r/c/testproject/+/40248
* CentOS 8 Wallaby 2023-05-31
* CentOS 8 Train 2023-05-26
    *  no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 

### Upstream components:
cs9 netowrk:
#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
    * no new hash on 25th
* RHEL 9 RHOS 17.1: RED 2023-05-22
    *  blocked on ceph issues
    *  lots of node_failures - pinged on #rhos-ops
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443606
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443590
    
* RHEL 9 RHOS 17.0: 2023-05-26
    * no new hash after 26/05
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 8 RHOS 17.1: 2023-05-26
    * blocked on mixed_rhel which is facing node_Failure
    * chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
    * no new hash after 25

#### Downstream components
17.1 rhel9 :- tripleo/network/common
    - https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
    - https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
## Tuesday 30-05-2023
#### Upstream promotion

* Master: Green 2023-05-30
    
* Antelope: 2023-05-
    * https://review.rdoproject.org/r/c/testproject/+/42692
* CentOS 9 Wallaby 2023-05-

* CentOS 8 Wallaby 2023-05-
* CentOS 8 Train 2023-05-26
    * https://review.rdoproject.org/r/c/testproject/+/40248
    * https://review.rdoproject.org/r/c/testproject/+/40102
    *  no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 
    
#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
    * no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
    *  blocked on ceph issues
    *  lots of node_failures - pinged on #rhos-ops
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443606
* RHEL 9 RHOS 17.0: 2023-05-26
    * no new hash after 26/05
    * blocked with node_Failures
* RHEL 8 RHOS 17.1: 2023-05-26
    * blocked on mixed_rhel which is facing node_Failure
    * chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
    * no new hash after 25

## Monday 29-05-2023
#### Upstream promotion

* Master: Green 2023-05-29
    
* Antelope: 2023-05-
    * tried enqueing line but line disappears immediately.
    * 500 internal server error
    * https://logserver.rdoproject.org/openstack-periodic-container-antelope-centos9/review.rdoproject.org/rdo-jobs/master/periodic-centos-9-antelope-promote-consistent-to-podified-ci-testing/9a2ec13/job-output.txt
    * https://review.rdoproject.org/r/c/testproject/+/46082
* CentOS 9 Wallaby 2023-05-29
* CentOS 8 Wallaby 2023-05-29(green)
* CentOS 8 Train 2023-05-26
    *  no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 
    
#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
    * no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
    *  Newer hash available on 26th
    *  blocked on ceph issues
    *  lots of node_failures - pinged on #rhos-ops
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 9 RHOS 17.0: 2023-05-26
    * no new hash after 26/05
* RHEL 8 RHOS 17.1: 2023-05-26
    * new hash available
    * blocked on node_failures
    * chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
    * no new hash after 25
    
##### Downstream components
* RH9 17.1: tripleo/network/promote
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/443590
* RH8 16.2: 
* RH9 17.1: 
## Friday 26-05-2023


#### Upstream promotion

* Master: Green 2023-05-26(green)
* Antelope: 2023-05-26(green)
* CentOS 9 Wallaby 2023-05-26(green)
* CentOS 8 Wallaby 2023-05-26(green)
* CentOS 8 Train 2023-05-21
    *  no new hash (25th)

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
    * no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
    *  Newer hash available on 26th
    *  blocked on ceph issues
    *  https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 9 RHOS 17.0: 2023-05-26
* RHEL 8 RHOS 17.1: 2023-05-26
    * new hash available
    * blocked on mixed rhel
* RHEL 8 RHOS 16.2: 2023-05-25
    * no new hash
    
##### Downstream components
* RH9 17: green
* RH8 16.2: green
* RH9 17.1: 
    * compute blocked on mixed rhel 
    * network sc10
    * tripleo sc04



* rr tool has some issues while showing results 
    * http://pastebin.test.redhat.com/1101315
    * screenshot of pipeline - ![](https://hackmd.io/_uploads/SJeElXm82.png)
