# Ruck Rover 2023-03-24 - 2023-03-30
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

:::warning
**Prow Status**
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
:::
[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/2kNmMGOzQE62mTJEohi3qQ

##### ruck & rover: Soniya, Ananya
##### ruck & rover alias: @ruckroverciframework

:::warning
### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/2013330 ovb jobs on master
### Ongoing bugs

#### Upstream check
* https://bugs.launchpad.net/tripleo/+bug/2008822 tripleo-ansible "tripleo_compute_node" molecule job failing on "tripleo_iscsid" task

#### Upstream blockers

* https://bugs.launchpad.net/tripleo/+bug/2012240 mixed-os job failing on all TestNetworkBasicOps tempest tests.

#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
- need to monitor
* https://bugzilla.redhat.com/show_bug.cgi?id=2179806 17.1/8 Container build is failing with deps issue: nothing provides network-scripts-openvswitch3.1 needed by rhosp-network-scripts-openvswitch-3.1-1.el8osttrunk.noarch

### Solved bugs.
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2180319 RHOSP17.1 Mixed os job is failing with TypeError: relationship() got an unexpected keyword argument 'sync_backref'~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012363 Featureset39 train fails with libibverbs-44.0-2.el8.1.x86_64.rpm: signature hdr data: BAD, no. of bytes(15176) out of range~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012489 Diskimage-builder failing to install in train job~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2009488 tripleo modify-image role failed with format: raw error~~

:::
## Thrusday 30 March
### Promotion Status
::: info
#### Upstream promotion
CentoS 9 Master: 2023-03-28
CentOS 9 Zed: 2023-03-29
CentOS 9 Wallaby: 2023-03-30
CentOS 8 Wallaby: 2023-03-30
CentOS 8 Train: 2023-03-28

#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-03-29
RHEL 9 RHOS 17.1: 2023-03-29
RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
RHEL 8 RHOS 17.1: 2023-03-30
RHEL 8 RHOS 16.2: 2023-03-30
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
##### node_failures tp:
- https://code.engineering.redhat.com/gerrit/c/testproject/+/435331
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
:::

:::
## Wednesday 29 March
### Promotion Status
::: info
#### Upstream promotion
CentoS 9 Master: 2023-03-
CentOS 9 Zed: 2023-03-26
CentOS 9 Wallaby: 2023-03- (Red mixed os bug ongoing)
CentOS 8 Wallaby: 2023-03- (Red mixed os bug ongoing)
CentOS 8 Train: 2023-03-

#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-03-25
RHEL 9 RHOS 17.1: 2023-03-28
RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
RHEL 8 RHOS 17.1: 2023-03-28
RHEL 8 RHOS 16.2: 2023-03-28
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
##### Downstream components:
tripleo: https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
octavia: https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
:::
## Tuesday 28 March
### Promotion Status
::: info
#### Upstream promotion
CentoS 9 Master: 2023-03-27
CentOS 9 Zed: 2023-03-26
CentOS 9 Wallaby: 2023-03-28 (Red mixed os bug ongoing)
CentOS 8 Wallaby: 2023-03-28 (Red mixed os bug ongoing)
CentOS 8 Train: 2023-03-

#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-03-25
RHEL 9 RHOS 17.1: 2023-03-28
RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
RHEL 8 RHOS 17.1: 2023-03-28
RHEL 8 RHOS 16.2: 2023-03-28
##### Downstream components:

:::

## Tuesday 27 March
### Promotion Status
::: info
#### Upstream promotion
CentoS 9 Master: 2023-03-24 (line still running)
CentOS 9 Zed: 2023-03-26
CentOS 9 Wallaby: 2023-03-25 (Red mixed os bug ongoing)
- https://review.rdoproject.org/r/c/testproject/+/40248
CentOS 8 Wallaby: 2023-03-24 (Red mixed os bug ongoing)
CentOS 8 Train: 2023-03-26

#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-03-25
RHEL 9 RHOS 17.1: 2023-03-22
- https://code.engineering.redhat.com/gerrit/c/testproject/+/435331(container build issue)
RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
- https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/current-tripleo/delorean.repo.md5
- https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/tripleo-ci-testing/delorean.repo.md5
RHEL 8 RHOS 17.1: 2023-03-27

RHEL 8 RHOS 16.2: 2023-03-27
##### Downstream components:
clients: https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
:::

# Thursday 30

* Promotion chase
* https://review.rdoproject.org/r/c/testproject/+/47975
* https://review.rdoproject.org/r/c/testproject/+/47974

# Wednesday 29 March

* promotion chase:
* master: https://review.rdoproject.org/r/c/testproject/+/47974
* zed: https://review.rdoproject.org/r/c/testproject/+/47975

# Tuesday, 28 march 2023

* Promotion chase
* master: https://review.rdoproject.org/r/c/testproject/+/47974
*

# Monday, 27 March 2023
* master fs35 has tempest failures (not the same sets of tests) - Ananya is checking

* post failures on 26th (looks better today)

* promotion chase:
* master: https://review.rdoproject.org/r/c/testproject/+/47974
* zed: should promote today (no failing jobs)
* wallabys: https://review.rdoproject.org/r/c/testproject/+/47975 (mixed rhel)
* train: should promote today (no failing jobs)

# Friday, 24 March 2023

## gate failure, following under recheck
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/873474~~

## Upstream Integration line
* master:
* fs35: https://review.rdoproject.org/r/c/testproject/+/47974
* zed:
* ~~https://review.rdoproject.org/r/c/testproject/+/47975~~
* train:
* ~~https://review.rdoproject.org/r/c/testproject/+/47976~~

* Components
* master: https://review.rdoproject.org/r/c/testproject/+/47980
* zed: Green
* wallaby c9: https://review.rdoproject.org/r/c/testproject/+/47981
* wallaby c8 : green
* train : green

**TODO** : Need to find out what is wrong with wallaby c8 and zed tripleo - the jobs are green but the last promotion is long back. Maybe the changes are recent and the line did not run. Ananya will check on monday.

## Downstream Integration line

* 16.2 on rhel8:
* re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431140 **SUCCESS**

* 17.1 on rhel8:
* re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803

* 17.1 on rhel9:
* re-running failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803
* Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2181513
* Fix: https://review.opendev.org/c/openstack/tripleo-common/+/878524