# Ruck Rover 2023-02-10 - 2023-02-16
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/skx8dkSxTmCJbhzRcZLgrQ

##### ruck & rover: chandankumar, anbanerj
##### ruck & rover alias: @ruckroverciframework

:::warning
### Ongoing bugs

#### Upstream check

#### Upstream blockers
* https://bugs.launchpad.net/tripleo/+bug/2006649 : Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy
* pin libvirt: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213 - merged

*

#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2167724 RHOS18 on RHEL9.2 - Error running container image prepare
* https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/870966 - Merged
* https://code.engineering.redhat.com/gerrit/c/ansible-role-tripleo-modify-image/+/439594 - merged
* https://code.engineering.redhat.com/gerrit/c/testproject/+/436232
* https://bugzilla.redhat.com/show_bug.cgi?id=2167302 17.1/9 - Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy:
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/439570
* https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/440031

### QE Bug
* https://bugs.launchpad.net/tripleo/+bug/2000226
* unpin systemd*-252-*: tripleo-ci/CentOS-9/master.yaml
* revert: https://review.opendev.org/c/openstack/tripleo-quickstart/+/872672
* testproject: https://review.rdoproject.org/r/c/testproject/+/45851
* Jenkins component jobs on RHEL 9.2 are showing growvols errors: see example logs: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_component-manila-pcci-17.1_dlrn-rhel-9.2-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/4/undercloud-0/home/stack/ansible.log.gz (we need to alert g the rr from QE - commenting one CIX card)
:::

::: success

### Closed Bugs:

* https://bugs.launchpad.net/tripleo/+bug/2006442 Master Undercloud upgrade job is failing with Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[neutron]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer
* https://review.opendev.org/c/openstack/puppet-tripleo/+/872859 - merged
* rebased on top of https://review.opendev.org/c/openstack/puppet-tripleo/+/872860/ - merged
* https://review.opendev.org/c/openstack/releases/+/872858 - merged

:::

## Hand off

Due to rdo.trunk.registry deletion, all pipeline got affected.
Most of the jobs are fixed by removing rdo-registry secret.
It affected all master promotion.

octavia is out 5 days in 17.1 - there is a related cix

### Promotion Status
:::info
#### Upstream Promotion
CentOS 8 Train: 2023-02-12
CentOS 8 wallaby: 2023-02-13
CentOS 9 wallaby: 2023-02-13
CentOS 9 Zed: 2023-02-13
CentoS 9 master: 2023-02-13

#### Downstream Promotion

18 - 14th
17.1 - 15th
17 - 15th
16.2 - 15th

:::

Due to rdo.trunk.registry deletion, all pipeline got affected.
Most of the jobs are fixed by removing rdo-registry secret.
It affected all master promotion.

octavia is out 5 days in 17.1 - there is a related cix

## Mon, Feb 16
### Promotion Status
:::info
#### Upstream Promotion
container-build job is failing due to rdo registry decomissioned
fixed by https://review.rdoproject.org/r/c/config/+/47069
#### Downstream Promotion

RHEL-9 RHOS-18 : 2023-02-14
RHEL-9 RHOS-17.1 : 2023-02-15
RHEL-9 RHOS-17 : 2023-02-15
RHEL-8 RHOS-16.2 : 2023-02-15

:::

#### 17.1 octavia:

## Mon, Feb 15
### Promotion Status
:::info
#### Upstream Promotion
CentOS 8 Train: 2023-02-12
CentOS 8 wallaby: 2023-02-13
CentOS 9 wallaby: 2023-02-13
CentOS 9 Zed: 2023-02-13
CentoS 9 master: 2023-02-13
#### Downstream Promotion

RHEL-9 RHOS-18 : 2023-02-14
RHEL-9 RHOS-17.1 : 2023-02-15
RHEL-9 RHOS-17 : 2023-02-15
RHEL-8 RHOS-16.2 : 2023-02-15

:::

#### 17.1:
periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1 overcloud deploy failing

## Mon, Feb 14
### Promotion Status
:::info
#### Upstream Promotion
CentOS 8 Train: 2023-02-12
CentOS 8 wallaby: 2023-02-13
CentOS 9 wallaby: 2023-02-13
CentOS 9 Zed: 2023-02-13
CentoS 9 master: 2023-02-13
validation component:
- https://review.rdoproject.org/r/c/rdo-jobs/+/46994
#### Downstream Promotion

RHEL-9 RHOS-18 : 2023-02-13
RHEL-9 RHOS-17.1 : 2023-02-13
RHEL-9 RHOS-17 : 2023-02-13
RHEL-8 RHOS-16.2 : 2023-02-11

compoenents:
16.2 - Green
17 - Green
17.1 - Green
18 - Green

:::

TP for 16.2: https://code.engineering.redhat.com/gerrit/c/testproject/+/439806

## Mon, Feb 13
### Promotion Status
:::info
#### Upstream Promotion
CentOS 8 Train: 2023-02-12 15:24:26
CentOS 8 wallaby: 2023-02-12 05:09:51
CentOS 9 wallaby: 2023-02-13 03:59:18
CentOS 9 Zed: 2023-02-13 07:04:45
#### Downstream Promotion : Resolved
~~All jobs are failing with post failure during pushing logs to logserver.
https://app.slack.com/client/T027F3GAJ/C042U5CT6TH/thread/C042U5CT6TH-1676264730.797809~~

RHEL-9 RHOS-18 : 2023-02-13
RHEL-9 RHOS-17.1 : 2023-02-13
RHEL-9 RHOS-17 : 2023-02-09
RHEL-8 RHOS-16.2 : 2023-02-11

:::

TP for jobs that hit post failure:
- 18 and 17.1: https://code.engineering.redhat.com/gerrit/c/testproject/+/439799/
- 17: https://code.engineering.redhat.com/gerrit/c/testproject/+/439806
- 16.2: Failed due to the post failures this morning. I'll TP if it doesnt promote by tomorrow

periodic-tripleo-ci-rhel-9-standalone-full-tempest-api on 17 and 17.1 may have a tempest issue. I'll bug it tomorrow if it is consistent.

Jenkins jobs: Rekicked few of them to get newer logs, pinged Stevebaker on the cix card

## Fri, Feb 10

### Promotion Status
:::info
#### DownStream Promotion
RHEL-9 RHOS-18 - 2023-02-09
RHEL-9 RHOS-17.1 2023-02-07 Running now
RHEL-9 RHOS-17 2023-02-09 No new content
RHEL-8 RHOS-16.2 :
- tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/439741

Components:
RHEL-9 RHOS-17.1: jenkins jobs are failing
- cinder:
- clients:
- cloudops
- compute:
- manilla:

RHEL-9 RHOS-17 : Green
RHEL-8 RHOS-16.2: Green

:::