# Ruck Rover 2023-05-11 - 2023-05-18

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/0iGbL43uTPa9YtjWtdWfjQ

##### ruck & rover: doug (dviroel)
##### ruck & rover alias: @ruckroverciframework

:::info
**Fixes and Improvements needed**
* periodic-tripleo-ci-build-containers-ubi-9-quay-push-* is not building packages from *Depends-On*
* Testing it here: https://review.rdoproject.org/r/c/testproject/+/46046
* periodic-tripleo-ci-build-containers-ubi-9-quay-push-* is not working with *dlrn_hash_tag* var - it is always using *tripleo-ci-testing* hash.
* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
* https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
* periodic build-image jobs were failing because of that

:::

:::danger
**BUGS**
* Internal DLRN is returning **500 Internal Server Error** for some promotions, need to reach reldel if persists.

* https://bugs.launchpad.net/tripleo/+bug/2019507 - featureset020-wallaby is failing tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps -failed to reach VERIFY_RESIZE
* Fix: https://review.opendev.org/c/openstack/tripleo-common/+/883156
* This now shows up in 17.1 https://sf.hosted.upshift.rdu2.redhat.com/logs/f6/f6cb4cde9cd049884965cec6b57d8e0b8cd6125b/openstack-periodic-integration-rhos-17.1-rhel9/periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17.1/7790488/logs/undercloud/var/log/tempest/stestr_results.html.gz

* **NEW** ~~[RHOS-17][RHEL-9] periodic-tripleo-build-containers-rhel-9-internal-build-push-upload-rhos-17 is failing to build the base container "Installed package which-2.21-28.el9.x86_64 not available"~~
* https://bugzilla.redhat.com/show_bug.cgi?id=2203218

* train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0
* https://bugs.launchpad.net/tripleo/+bug/2019192
* not a blocker, passing on internal
* ~~Content providers: https://bugzilla.redhat.com/show_bug.cgi?id=2196905~~
* Fixed with https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/443000
:::

:::info
## Handoff

* 17.1 on RHEL 9 is out since 2023-04-30
* QE Jenkins jobs are failing on ntp/chrony/cert issue. Attila is working on it
* sc002 is failing due to https://bugzilla.redhat.com/show_bug.cgi?id=2196805
* Hash which has only these 2 missing: **daac453466807b1bd6c835fd903afb52**
* All 17.1 jobs are failing due to https://bugzilla.redhat.com/show_bug.cgi?id=2196805
* https://redhat-internal.slack.com/archives/C042U5CT6TH/p1683716005503589
* Components to check after 17.1 issue clear: cinder, octavia, clients, common, network
* Content providers: https://bugzilla.redhat.com/show_bug.cgi?id=2196905
:::

:::info
#### Upstream promotion
* CentOS 9 Wallaby 2023-05-09
* needs fix for https://bugs.launchpad.net/tripleo/+bug/2019507
* CentOS 8 Wallaby 2023-05-18
* CentOS 8 Train 2023-05-16
* no new hash (18th)

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-04-23
* (no new hash after 66dcadfdb45c891c5d4fd1f097f832fb)
* RHEL 9 RHOS 17.1: 2023-05-12
* Components promoted, needs a new promotion
* RHEL 9 RHOS 17.0: 2023-05-17
* no new hash (18th)
* RHEL 8 RHOS 17.1: 2023-05-18
* RHEL 8 RHOS 16.2: 2023-05-17

:::
#### 2023-05-18
* some promotion jobs are failing in dlrnapi promote operation with: 500 Internal Server Error.

#### 2023-05-17

#### 2023-05-16

#### 2023-05-15
* new wallaby bug: https://bugs.launchpad.net/tripleo/+bug/2019507
* fix? https://review.opendev.org/c/openstack/tripleo-common/+/883156
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds POST_FAILURE in downstream

#### 2023-05-11
* new train cix filed
* https://bugs.launchpad.net/tripleo/+bug/2019192 train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0