---
# System prepended metadata

title: Ruck Rover 2023-03-24 - 2023-03-30
tags: [ruck_rover]

---

# Ruck Rover 2023-03-24 - 2023-03-30
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)


Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

:::warning
**Prow Status**
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
    * For now, we just look (and investigate) if a test has multiple failures across different PRs
:::
[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/2kNmMGOzQE62mTJEohi3qQ

##### ruck & rover: Soniya, Ananya
##### ruck & rover alias: @ruckroverciframework


:::warning
### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/2013330 ovb jobs on master
### Ongoing bugs

#### Upstream check
* https://bugs.launchpad.net/tripleo/+bug/2008822 tripleo-ansible "tripleo_compute_node" molecule job failing on "tripleo_iscsid" task

#### Upstream blockers

* https://bugs.launchpad.net/tripleo/+bug/2012240 mixed-os job failing on all TestNetworkBasicOps tempest tests.

#### Downstream blockers
*  https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
    - need to monitor
* https://bugzilla.redhat.com/show_bug.cgi?id=2179806 17.1/8 Container build is failing with deps issue: nothing provides network-scripts-openvswitch3.1 needed by rhosp-network-scripts-openvswitch-3.1-1.el8osttrunk.noarch 

### Solved bugs.
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2180319 RHOSP17.1 Mixed os job is failing with TypeError: relationship() got an unexpected keyword argument 'sync_backref'~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012363 Featureset39 train fails with libibverbs-44.0-2.el8.1.x86_64.rpm: signature hdr data: BAD, no. of bytes(15176) out of range~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012489 Diskimage-builder failing to install in train job~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2009488 tripleo modify-image role failed with format: raw error~~

:::
## Thrusday 30 March
### Promotion Status 
::: info
#### Upstream promotion
    CentoS 9 Master: 2023-03-28
    CentOS 9 Zed: 2023-03-29
    CentOS 9 Wallaby: 2023-03-30
    CentOS 8 Wallaby: 2023-03-30
    CentOS 8 Train: 2023-03-28

#### Downstream Promotion
    RHEL 9 RHOS 18.0: 2023-03-29
    RHEL 9 RHOS 17.1: 2023-03-29 
    RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
    RHEL 8 RHOS 17.1: 2023-03-30
    RHEL 8 RHOS 16.2: 2023-03-30
        - https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
##### node_failures tp:
- https://code.engineering.redhat.com/gerrit/c/testproject/+/435331
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
:::

:::
## Wednesday 29 March
### Promotion Status 
::: info
#### Upstream promotion
    CentoS 9 Master: 2023-03-
    CentOS 9 Zed: 2023-03-26
    CentOS 9 Wallaby: 2023-03- (Red mixed os bug ongoing)
    CentOS 8 Wallaby: 2023-03- (Red mixed os bug ongoing)
    CentOS 8 Train: 2023-03-

#### Downstream Promotion
    RHEL 9 RHOS 18.0: 2023-03-25
    RHEL 9 RHOS 17.1: 2023-03-28 
    RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
    RHEL 8 RHOS 17.1: 2023-03-28
    RHEL 8 RHOS 16.2: 2023-03-28
        - https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
##### Downstream components:
tripleo: https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
octavia: https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
:::
## Tuesday 28 March
### Promotion Status 
::: info
#### Upstream promotion
    CentoS 9 Master: 2023-03-27
    CentOS 9 Zed: 2023-03-26
    CentOS 9 Wallaby: 2023-03-28 (Red mixed os bug ongoing)
    CentOS 8 Wallaby: 2023-03-28 (Red mixed os bug ongoing)
    CentOS 8 Train: 2023-03-

#### Downstream Promotion
    RHEL 9 RHOS 18.0: 2023-03-25
    RHEL 9 RHOS 17.1: 2023-03-28 
    RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
    RHEL 8 RHOS 17.1: 2023-03-28
    RHEL 8 RHOS 16.2: 2023-03-28
##### Downstream components:

:::        


## Tuesday 27 March
### Promotion Status
::: info
#### Upstream promotion
    CentoS 9 Master: 2023-03-24 (line still running)
    CentOS 9 Zed: 2023-03-26
    CentOS 9 Wallaby: 2023-03-25 (Red mixed os bug ongoing)
        - https://review.rdoproject.org/r/c/testproject/+/40248
    CentOS 8 Wallaby: 2023-03-24 (Red mixed os bug ongoing)
    CentOS 8 Train: 2023-03-26

#### Downstream Promotion
    RHEL 9 RHOS 18.0: 2023-03-25
    RHEL 9 RHOS 17.1: 2023-03-22
        - https://code.engineering.redhat.com/gerrit/c/testproject/+/435331(container build issue)
    RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
        - https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/current-tripleo/delorean.repo.md5
        - https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/tripleo-ci-testing/delorean.repo.md5
    RHEL 8 RHOS 17.1: 2023-03-27
        
    RHEL 8 RHOS 16.2: 2023-03-27 
##### Downstream components:
clients: https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
:::        

# Thursday 30

* Promotion chase
    * https://review.rdoproject.org/r/c/testproject/+/47975
    * https://review.rdoproject.org/r/c/testproject/+/47974

# Wednesday 29 March

* promotion chase:
    * master: https://review.rdoproject.org/r/c/testproject/+/47974
    * zed: https://review.rdoproject.org/r/c/testproject/+/47975

# Tuesday, 28 march 2023

* Promotion chase
    * master: https://review.rdoproject.org/r/c/testproject/+/47974
    * 

# Monday, 27 March 2023
* master fs35 has tempest failures (not the same sets of tests) - Ananya is checking

* post failures on 26th (looks better today)

* promotion chase:
    * master: https://review.rdoproject.org/r/c/testproject/+/47974
    * zed: should promote today (no failing jobs)
    * wallabys: https://review.rdoproject.org/r/c/testproject/+/47975 (mixed rhel)
    * train: should promote today (no failing jobs)

    
# Friday, 24 March 2023

## gate failure, following under recheck
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/873474~~

## Upstream Integration line
* master: 
    * fs35: https://review.rdoproject.org/r/c/testproject/+/47974
* zed:
    * ~~https://review.rdoproject.org/r/c/testproject/+/47975~~
* train:
    * ~~https://review.rdoproject.org/r/c/testproject/+/47976~~

* Components
    * master: https://review.rdoproject.org/r/c/testproject/+/47980
    * zed: Green
    * wallaby c9: https://review.rdoproject.org/r/c/testproject/+/47981
    * wallaby c8 : green
    * train : green

**TODO** : Need to find out what is wrong with wallaby c8 and zed tripleo - the jobs are green but the last promotion is long back. Maybe the changes are recent and the line did not run. Ananya will check on monday.

## Downstream Integration line

  * 16.2 on rhel8: 
    * re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431140    **SUCCESS**

  * 17.1 on rhel8: 
    * re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803

  * 17.1 on rhel9: 
    * re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803
      * Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2181513
        * Fix: https://review.opendev.org/c/openstack/tripleo-common/+/878524  