# Ruck Rover 2023-05-05 - 2023-05-10
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)
*
###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/rk9X0WJKQMSE_oUyobSDyw

##### ruck & rover: anbanerj
##### ruck & rover alias: @ruckroverciframework

:::info
## Handoff

* 17.1 on RHEL 9 is out since 2023-04-30
* QE Jenkins jobs are failing on ntp/chrony/cert issue. Attila is working on it
* sc002 is failing due to https://bugzilla.redhat.com/show_bug.cgi?id=2196805
* Hash which has only these 2 missing: **daac453466807b1bd6c835fd903afb52**
* All 17.1 jobs are failing due to https://bugzilla.redhat.com/show_bug.cgi?id=2196805
* https://redhat-internal.slack.com/archives/C042U5CT6TH/p1683716005503589
* Components to check after 17.1 issue clear: cinder, octavia, clients, common, network
* Content providers: https://bugzilla.redhat.com/show_bug.cgi?id=2196905

:::

:::danger
#### Open Bugs
* https://bugzilla.redhat.com/show_bug.cgi?id=2196905 (Content providers)
* https://bugzilla.redhat.com/show_bug.cgi?id=2196808 (cert 404 17.1 compute)
* https://bugzilla.redhat.com/show_bug.cgi?id=2196805 (libunbound all 17.1)
* https://bugs.launchpad.net/tripleo/+bug/2018384 (ceph failure on 17 and 17.1 lines)
* https://redhat-internal.slack.com/archives/C04HN4L5XHV/p1683297924545939
:::

:::success
#### Closed Bugs
* https://bugs.launchpad.net/tripleo/+bug/2018588 (C8 standalone deploy) **GATE BLOCKER**
* https://redhat-internal.slack.com/archives/C04F192QB37/p1683287107847279
* https://bugs.launchpad.net/tripleo/+bug/2018356 (Kernel panic on c9 wallaby and 17.1)
* https://redhat-internal.slack.com/archives/C042U5CT6TH/p1683291687263269
:::

:::info
#### Upstream promotion
* CentOS 9 Wallaby 2023-05-09
* CentOS 8 Wallaby 2023-05-11
* CentOS 8 Train 2023-05-09

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-04-23
* (no new hash after 66dcadfdb45c891c5d4fd1f097f832fb)
* RHEL 9 RHOS 17.1: 2023-04-30
* Hitting virt customize
* RHEL 9 RHOS 17.0: 2023-05-08
* RHEL 8 RHOS 17.1: 2023-05-10
* RHEL 8 RHOS 16.2: 2023-05-08
* (no new hash after d805226bf739bd964fd1e4f4a5e702e7)
:::
#### 2023-05-11
* ~~17.1/9 image builds failing dependencies might be related to yesterday issues~~ (seems transient?)
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-9-buildimage-overcloud-hardened-uefi-full-rhos-17.1/0cfb293/job-output.txt
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-rhel-9-internal-build-push-upload-rhos-17.1/9cfa612/job-output.txt
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/ec6f961eea2940cf914a44ede2af97d3
* rerun containers to sanity check before filing bug https://code.engineering.redhat.com/gerrit/c/testproject/+/442966
* PASSED
* new train cix filed
* https://bugs.launchpad.net/tripleo/+bug/2019192 train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0

#### 2023-05-10

* TP: 17.1: https://code.engineering.redhat.com/gerrit/c/testproject/+/430999
* 17.1 on 9 components to check
* cinder, octavia, clients, common, nw,
* I dont see any node failure this morning
* Rebuilt https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.2-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/
* Failed overcloud deploy
* failed undercloud deploy - No package pcp-system-tools available
* Zed TP: https://review.rdoproject.org/r/c/testproject/+/48510

* nothing provides libunbound.so.8()(64bit) needed by openvswitch3.1-3.1.0-14.el9fdp.x86_64
* https://sf.hosted.upshift.rdu2.redhat.com/logs/99/430999/51/check/periodic-tripleo-ci-rhel-9-scenario002-standalone-rhos-17.1/6b0134d/job-output.txt
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-compute-rhos-17.1/67a8d83/job-output.txt
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-compute-rhos-17.1/acafcd3/job-output.txt

* Setup repos on live host: 404 for http://hdn.corp.redhat.com/rhel8-csb-stage/RPMS/noarch/redhat-internal-cert-install-0.1-31.el7.noarch.rpm
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal/9a63f84/job-output.txt
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal/ad8272d/job-output.txt

#### 2023-05-09
* Train TP https://review.rdoproject.org/r/c/testproject/+/48498
* Move OVBs to nested virt https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/442875
* TP: https://code.engineering.redhat.com/gerrit/c/testproject/+/442794
*
##### Components
* RHEL 9 RHOS 17.1:
* hitting ceph and ovb virt customize timeouts
* periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17.1
* ERROR: Can't run container container-puppet-ironic, Undefined variable '::deploy_config_name
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17.1/c64d07a/logs/undercloud/home/zuul/undercloud_install.log

### 2023-05-08

* Node hold for sc004 ceph: https://code.engineering.redhat.com/gerrit/c/testproject/+/442794
* Testing fix for c8 ansible: https://review.rdoproject.org/r/c/testproject/+/48498
*

### 2023-05-05

##### TPs:
* tripleo component of 17/17.1: https://code.engineering.redhat.com/gerrit/c/testproject/+/442699
* 17, 17.1 cephs: https://code.engineering.redhat.com/gerrit/c/testproject/+/442691
* 16.2: https://code.engineering.redhat.com/gerrit/c/testproject/+/442696
* wallaby c8: https://review.rdoproject.org/r/c/testproject/+/47981

* Node hold for virt customize:
* https://review.rdoproject.org/r/c/testproject/+/48491
* https://code.engineering.redhat.com/gerrit/c/testproject/+/442726