# Ruck Rover 2023-03-10 - 2023-03-16
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

:::warning
**Prow Status**
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
:::
[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Gjde-nceQZaXZSb-1F3cGA

##### ruck & rover: Pooja (pojadhav), Cedric (Cédric Jeanneret)
##### ruck & rover alias: @ruckroverciframework

:::warning
### New bugs
*
:::

:::warning
### Ongoing bugs

#### Upstream check
* https://bugs.launchpad.net/tripleo/+bug/2008822 tripleo-ansible "tripleo_compute_node" molecule job failing on "tripleo_iscsid" task
* the fix got merged. Waiting for another test run to verify it.
#### Upstream blockers
* https://bugs.launchpad.net/tripleo/+bug/2009488 tripleo modify-image role failed with format: raw error
#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
- need to monitor
:::

# Thursday, 16 March 2023
### Promotion Status
::: info
#### Upstream promotion
https://review.rdoproject.org/r/c/testproject/+/36719

CentoS 9 Master: 2023-03-14
CentOS 9 Zed: 2023-03-13
CentOS 9 Wallaby: 2023-03-13
CentOS 8 Wallaby: 2023-03-14
CentOS 8 Train: 2023-03-12 (no new content)
#### Downstream Promotion

RHEL 9 RHOS 18.0: 2023-03-15 20:30:47
RHEL 9 RHOS 17.1: 2023-03-13 19:23:21
RHEL 9 RHOS 17.0: 2023-03-07 18:04:12 (no new content)
RHEL 8 RHOS 17.1: 2023-03-16 10:32:23
RHEL 8 RHOS 16.2: 2023-03-15 15:30:04 (see note bellow)
:::

### Upstream/RDO Issues

* https://bugs.launchpad.net/tripleo/+bug/2011805 - build-containers-ubi-8-quay-push (wallaby/train) failing consistently with error util-linux-2.32.1-41.el8.x86_64.rpm: signature hdr data: BAD, no. of bytes(127884) out of range
* results currently https://review.rdoproject.org/r/c/testproject/+/36478/19#message-134489461cd0388ef52de3fd6e1e6b7a54a53bf6

* https://bugs.launchpad.net/tripleo/+bug/2011824 - fs20 on zed/master is failing consistently on tempest test "test_server_connectivity_live_migration"
Ronelle pointed to https://access.redhat.com/solutions/6049471

* periodic-tripleo-ci-centos-9-8-multinode-mixed-os -
failing consistently but with different containers may be fallout becasue of not building container for c8, once we clear out this https://bugs.launchpad.net/tripleo/+bug/2011805 then we need to check whether it is green or any other real issue.
https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-8-multinode-mixed-os&skip=0
* failure 1 - container tripleowallabycentos8/openstack-neutron-server failed to prepare
* failure 2 - tripleowallabycentos8/openstack-neutron-metadata-agent failed to prepare
* failure 3 - tempest tests

### Downstream Issues

* 17.1 on el9 (re-kick green, all should be good now)
* Pacemaker issue - opened CIX (seems hardware/resources related):
* https://trello.com/c/Y8Wu079T
* https://bugzilla.redhat.com/show_bug.cgi?id=2178923
* Network issue crashed another job
* 17.1 on el8 (re-kick green, should be all good now)
* [OVN build images failed to pass check](https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17.1/270ca80/job-output.txt):
* ovn-controller
* ovn-nb-db-server
* ovn-northd
* ovn-sb-db-server
* Rekick for all osp-17.1: https://code.engineering.redhat.com/gerrit/c/testproject/+/441064
* 16.2 on el8
* [envD IPMI issue](https://trello.com/c/Kwyxv4Gb)
* BM job removed from criteria via [this patch](https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/441041)

# Wednesday, 15 March 2023
### Promotion Status
::: info
#### Upstream promotion
https://review.rdoproject.org/r/c/testproject/+/36719

CentoS 9 Master: 2023-03-14
CentOS 9 Zed: 2023-03-13 (no new content)
CentOS 9 Wallaby: 2023-03-13
CentOS 8 Wallaby: 2023-03-14 (no new content)
CentOS 8 Train: 2023-03-12 (no new content)

#### Downstream Promotion

RHEL 9 RHOS 18.0: 2023-03-14 20:27:22
RHEL 9 RHOS 17.1: 2023-03-13 19:23:21 (flagged as red in RR tool)
RHEL 9 RHOS 17.0: 2023-03-07 18:04:12
RHEL 8 RHOS 17.1: 2023-03-14 14:26:19
RHEL 8 RHOS 16.2: 2023-03-10 07:11:30 (IPMI, known CIX)
:::
* 16.2
* [CIX created](https://trello.com/c/Kwyxv4Gb) related to the IPMI issue on envD
* 17.1 el9
* investigating, 3 jobs failed
* re-kicked 3 jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/441034
* re-kicked glance component downstream

# Tuesday, 14 March 2023
### Promotion Status
::: info
#### Upstream promotion
https://code.engineering.redhat.com/gerrit/c/testproject/+/437790
CentoS 9 Master: 2023-03-13
CentOS 9 Zed: 2023-03-13 (no new content)
CentOS 9 Wallaby: 2023-03-13 (no new content)
CentOS 8 Wallaby: 2023-03-14 (no new content)
CentOS 8 Train: 2023-03-12 (no new content)

#### Downstream Promotion

RHEL 9 RHOS 18.0: 2023-03-13 20:23:33
RHEL 9 RHOS 17.1: 2023-03-13 19:23:21
RHEL 9 RHOS 17.0: 2023-03-07 18:04:12 (no new content)
RHEL 8 RHOS 17.1: 2023-03-13 13:22:21
RHEL 8 RHOS 16.2: 2023-03-10 07:11:30 (RED)
:::

* 16.2 promotion
* Re-kicked job: https://code.engineering.redhat.com/gerrit/c/testproject/+/440967
* Failed. Seems to be related to envD and IPMI not working there. Testing on envA with the same testproject, we'll see if that env is still working and, if so, it we get a better result.

# Monday, 13 March 2023
### Promotion Status
::: info
#### Upstream promotion

CentoS 9 Master: 2023-03-13
CentOS 9 Zed: 2023-03-12
CentOS 9 Wallaby: 2023-03-13
CentOS 8 Wallaby: 2023-03-12
CentOS 8 Train: 2023-03-12

#### Downstream Promotion

RHEL 9 RHOS 18.0: 2023-03-11 20:16:27 (yellow)
RHEL 9 RHOS 17.1: 2023-03-10 16:12:37 (red)
RHEL 9 RHOS 17.0: 2023-03-07 18:04:12 (green - no new content)
RHEL 8 RHOS 17.1: 2023-03-11 16:15:50 (green)
RHEL 8 RHOS 16.2: 2023-03-10 07:11:30 (red)
:::

* osp-16.2: re-kicked the whole line
* job failed - [re-kicked](https://code.engineering.redhat.com/gerrit/c/testproject/+/440941)
* osp-17.1 on el9:
* [re-kicked](https://code.engineering.redhat.com/gerrit/c/testproject/+/440941) a job
*

## Friday, 10 March 2023
### Promotion Status
::: info
#### Upstream promotion
https://review.rdoproject.org/r/c/testproject/+/36719

CentoS 9 Master: 2023-03-09
CentOS 9 Zed: 2023-03-09 (No new content)
CentOS 9 Wallaby: 2023-03-09 (No new content)
CentOS 8 Wallaby: 2023-03-10
CentOS 8 Train: 2023-03-08 (No new content)

#### Downstream Promotion

RHEL 9 RHOS 18.0: 2023-03-09 20:10:17
RHEL 9 RHOS 17.1: 2023-03-09 16:09:47
RHEL 9 RHOS 17.0: 2023-03-07 18:04:12 (no new content)
RHEL 8 RHOS 17.1: 2023-03-09 14:09:28
RHEL 8 RHOS 16.2: 2023-03-10 07:11:30
:::

#### Vexxhost ticket node_failures
- https://support.vexxhost.com/hc/en-us/requests/363602