---
# System prepended metadata

title: Ruck Rover 2023-02-10 - 2023-02-16
tags: [ruck_rover]

---

# Ruck Rover 2023-02-10 - 2023-02-16
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)


Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/skx8dkSxTmCJbhzRcZLgrQ

##### ruck & rover: chandankumar, anbanerj
##### ruck & rover alias: @ruckroverciframework

:::warning
### Ongoing bugs

#### Upstream check

#### Upstream blockers
* https://bugs.launchpad.net/tripleo/+bug/2006649 : Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy
    * pin libvirt: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213 - merged

* 

#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2167724 RHOS18 on RHEL9.2 - Error running container image prepare
    * https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/870966 - Merged
    * https://code.engineering.redhat.com/gerrit/c/ansible-role-tripleo-modify-image/+/439594 - merged
        * https://code.engineering.redhat.com/gerrit/c/testproject/+/436232 
* https://bugzilla.redhat.com/show_bug.cgi?id=2167302 17.1/9 - Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy: 
    * https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/439570
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
*  https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
    *  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/440031

### QE Bug
* https://bugs.launchpad.net/tripleo/+bug/2000226
    * unpin systemd*-252-*: tripleo-ci/CentOS-9/master.yaml
        * revert: https://review.opendev.org/c/openstack/tripleo-quickstart/+/872672
        * testproject: https://review.rdoproject.org/r/c/testproject/+/45851
    * Jenkins component jobs on RHEL 9.2 are showing growvols errors: see example logs:  http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_component-manila-pcci-17.1_dlrn-rhel-9.2-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/4/undercloud-0/home/stack/ansible.log.gz (we need to alert g the rr from QE -  commenting one CIX card)
:::


::: success

### Closed Bugs:

* https://bugs.launchpad.net/tripleo/+bug/2006442 Master Undercloud upgrade job is failing with Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[neutron]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer
    * https://review.opendev.org/c/openstack/puppet-tripleo/+/872859 - merged
        * rebased on top of https://review.opendev.org/c/openstack/puppet-tripleo/+/872860/ - merged
    * https://review.opendev.org/c/openstack/releases/+/872858 - merged
    
    
:::

## Hand off

Due to rdo.trunk.registry deletion, all pipeline got affected.
Most of the jobs are fixed by removing rdo-registry secret.
It affected all master promotion.


octavia is out 5 days in 17.1 - there is a related cix

### Promotion Status
:::info
#### Upstream Promotion
    CentOS 8 Train: 2023-02-12
    CentOS 8 wallaby: 2023-02-13
    CentOS 9 wallaby: 2023-02-13
    CentOS 9 Zed: 2023-02-13
    CentoS 9 master: 2023-02-13
    
#### Downstream Promotion

18 - 14th
17.1 - 15th
17 - 15th
16.2 - 15th 

:::

Due to rdo.trunk.registry deletion, all pipeline got affected.
Most of the jobs are fixed by removing rdo-registry secret.
It affected all master promotion.


octavia is out 5 days in 17.1 - there is a related cix


## Mon, Feb 16
### Promotion Status
:::info
#### Upstream Promotion
     container-build job is failing due to rdo registry decomissioned
     fixed by https://review.rdoproject.org/r/c/config/+/47069
#### Downstream Promotion
    
    RHEL-9 RHOS-18 : 2023-02-14
    RHEL-9 RHOS-17.1 : 2023-02-15
    RHEL-9 RHOS-17 : 2023-02-15
    RHEL-8 RHOS-16.2 : 2023-02-15
    
:::
    
#### 17.1 octavia:


## Mon, Feb 15
### Promotion Status
:::info
#### Upstream Promotion
    CentOS 8 Train: 2023-02-12
    CentOS 8 wallaby: 2023-02-13
    CentOS 9 wallaby: 2023-02-13
    CentOS 9 Zed: 2023-02-13
    CentoS 9 master: 2023-02-13
#### Downstream Promotion
    
    RHEL-9 RHOS-18 : 2023-02-14
    RHEL-9 RHOS-17.1 : 2023-02-15
    RHEL-9 RHOS-17 : 2023-02-15
    RHEL-8 RHOS-16.2 : 2023-02-15
    
:::

#### 17.1: 
periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 overcloud deploy failing



## Mon, Feb 14
### Promotion Status
:::info
#### Upstream Promotion
    CentOS 8 Train: 2023-02-12
    CentOS 8 wallaby: 2023-02-13
    CentOS 9 wallaby: 2023-02-13
    CentOS 9 Zed: 2023-02-13
    CentoS 9 master: 2023-02-13
    validation component:
        - https://review.rdoproject.org/r/c/rdo-jobs/+/46994
#### Downstream Promotion
    
    RHEL-9 RHOS-18 : 2023-02-13
    RHEL-9 RHOS-17.1 : 2023-02-13
    RHEL-9 RHOS-17 : 2023-02-13
    RHEL-8 RHOS-16.2 : 2023-02-11
    
    compoenents:
    16.2 - Green
    17 - Green
    17.1 - Green
    18 - Green
    
:::

TP for 16.2: https://code.engineering.redhat.com/gerrit/c/testproject/+/439806

## Mon, Feb 13
### Promotion Status
:::info
#### Upstream Promotion
    CentOS 8 Train: 2023-02-12 15:24:26
    CentOS 8 wallaby: 2023-02-12 05:09:51
    CentOS 9 wallaby: 2023-02-13 03:59:18
    CentOS 9 Zed: 2023-02-13 07:04:45
#### Downstream Promotion : Resolved
    ~~All jobs are failing with post failure during pushing logs to logserver.
    https://app.slack.com/client/T027F3GAJ/C042U5CT6TH/thread/C042U5CT6TH-1676264730.797809~~
    
    RHEL-9 RHOS-18 : 2023-02-13
    RHEL-9 RHOS-17.1 : 2023-02-13
    RHEL-9 RHOS-17 : 2023-02-09
    RHEL-8 RHOS-16.2 : 2023-02-11
    
:::

TP for jobs that hit post failure:
- 18 and 17.1: https://code.engineering.redhat.com/gerrit/c/testproject/+/439799/
- 17: https://code.engineering.redhat.com/gerrit/c/testproject/+/439806
- 16.2: Failed due to the post failures this morning. I'll TP if it doesnt promote by tomorrow

periodic-tripleo-ci-rhel-9-standalone-full-tempest-api on 17 and 17.1 may have a tempest issue. I'll bug it tomorrow if it is consistent.

Jenkins jobs: Rekicked few of them to get newer logs, pinged Stevebaker on the cix card

## Fri, Feb 10

### Promotion Status
:::info
#### DownStream Promotion
    RHEL-9 RHOS-18 - 2023-02-09
    RHEL-9 RHOS-17.1 2023-02-07 Running now
    RHEL-9 RHOS-17 2023-02-09 No new content
    RHEL-8 RHOS-16.2 : 
    - tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/439741
    
    Components:
    RHEL-9 RHOS-17.1: jenkins jobs are failing
        - cinder: 
        - clients:
        - cloudops
        - compute: 
        - manilla:
        
    RHEL-9 RHOS-17 : Green
    RHEL-8 RHOS-16.2: Green
    
:::