# Ruck Rover 2023-04-28 - 2023-05-04
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

#### Upstream promotion
* ~~CentoS 9 Master~~ R.I.P. Master
* CentOS 9 Zed 2023-04-30
* CentOS 9 Wallaby 2023-04-30
* CentOS 8 Wallaby 2023-04-30
* CentOS 8 Train 2023-04-30

#### Downstream Promotion
* RHEL 9 RHOS 18.0:
* 2023-04-26 (current-podified) (podified-ci-testing has new content: 2023-04-27)
* 2023-04-23 (current-tripleo) (tripleo-ci-testing has no new content: 2023-04-23 66dcadfdb45c891c5d4fd1f097f832fb )
* RHEL 9 RHOS 17.1: 2023-04-30
* RHEL 9 RHOS 17.0: 2023-04-25 (no new content current-tripleo = tripleo-ci-testing 72a2c3a9dd3e42308907b19e710b5089)
* RHEL 8 RHOS 17.1: 2023-04-29
* RHEL 8 RHOS 16.2: 2023-04-28

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/j737nuJYTH20cG__Z-keuQ ([index there](https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ))

##### ruck & rover: cjeanner
##### ruck & rover alias: @ruckroverciframework

### 2023-05-08
:::warning
- 'Templar' object has no attribute 'set_temporary_context' https://bugs.launchpad.net/tripleo/+bug/2018588
- virt-customize segfaulting in cs9/wallaby jobs https://bugs.launchpad.net/tripleo/+bug/2018356
- also now on 17.1
- seems to be IBM cloud but not vexxhost
- workaround for RDO: https://review.rdoproject.org/r/c/rdo-jobs/+/48497 Run OVB jobs on vexxhost
- downstream osp-17.1 on el9: ceph issue, `--ntp-server` option not recognized - also hits osp-17.0 since both are wallaby.
- related to https://trello.com/c/dvh0Jhv0
- downstream change requests:
- https://code.engineering.redhat.com/gerrit/c/python-tripleoclient/+/442662
- [RHOS-17.1][RHEL-9] tripleo-component: periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17.1is failing with " Could not retrieve information from environment production source(s) file:///boot/efi/EFI/centos/grubx64.efi" https://bugzilla.redhat.com/show_bug.cgi?id=2196054
:::

#### Upstream
* CentOS9 Zed: 2023-05-08 (OVB running on vexxhost)
* CentOS9 Wallaby: 2023-05-08
* fs001 rerun - https://review.rdoproject.org/r/c/testproject/+/47875
* CentOS8 Wallaby: 2023-04-28
* needed the template fix - line in rerun
* CentOS8 Train: 2023-04-27
* Network and common train components are out of date
#### Downstream
* el9-osp18: 2023-05-08 current-podified
* el9-osp17.1 2023-04-26 - issues still here
* el9-osp17.0 2023-05-08
* el8-osp17.1 2023-04-27 -
* el8-osp16.2 2023-05-08

* All 16.2 and 17 components are passing. 17,1 has some failing components

NOTE: for OVB on downstream - used nested-virt nodeset like: https://code.engineering.redhat.com/gerrit/c/testproject/+/430999/47/.zuul.yaml or https://code.engineering.redhat.com/gerrit/c/testproject/+/440877/42/.zuul.yaml

#### Podified Container lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9) - 2023-05-08
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0) - 2023-05-08
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0) - 2023-05-08

### 2023-05-05
:::warning
- 'Templar' object has no attribute 'set_temporary_context' https://bugs.launchpad.net/tripleo/+bug/2018588
- virt-customize segfaulting in cs9/wallaby jobs https://bugs.launchpad.net/tripleo/+bug/2018356
- also now on 17.1
- seems to be IBM cloud but not vexxhost
- workaround for RDO: https://review.rdoproject.org/r/c/rdo-jobs/+/48497 Run OVB jobs on vexxhost
- downstream osp-17.1 on el9: ceph issue, `--ntp-server` option not recognized - also hits osp-17.0 since both are wallaby.
- related to https://trello.com/c/dvh0Jhv0
- downstream change requests:
- https://code.engineering.redhat.com/gerrit/c/python-tripleoclient/+/442662
- [RHOS-17.1][RHEL-9] tripleo-component: periodic-tripleo-ci-rhel-9-containers-multinode-tripleo-rhos-17.1is failing with " Could not retrieve information from environment production source(s) file:///boot/efi/EFI/centos/grubx64.efi" https://bugzilla.redhat.com/show_bug.cgi?id=2196054
:::

### 2023-05-04

:::warning
- NODE_FAILURE is still hitting downstream
- https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/osp-internal/builds?skip=0 (not our tenant - not our problem)
- downstream osp-17.1 on el9: ceph issue, `--ntp-server` option not recognized - also hits osp-17.0 since both are wallaby.
- related to https://trello.com/c/dvh0Jhv0
- downstream change requests:
- https://code.engineering.redhat.com/gerrit/c/python-tripleoclient/+/442662
:::

* wallaby/CS9: 2023-04-30
* re-running missing wallaby jobs: https://review.rdoproject.org/r/c/testproject/+/46698
* wallaby/CS8: 2023-05-04
* train/CS8: 2023-04-30 (no new content)
* zed/CS9: 2023-04-30
* re-running missing jobs: https://review.rdoproject.org/r/c/testproject/+/46694
* OSP-16.2/EL8: 2023-05-01
* re-running missing job: https://code.engineering.redhat.com/gerrit/c/testproject/+/442654
* OSP-17.1/EL8: 2023-05-03
* OSP-17.1/EL9: 2023-04-30
* re-running missing jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/442480
* OSP-18/EL9:
* current-tripleo: 2023-04-23 (no new content)
* current-podified: 2023-05-03

### 2023-05-03
* virt-customize CIX should be solved soon (cs9/wallaby)
* https://bugs.launchpad.net/tripleo/+bug/2018356
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/882138

### 2023-05-02
* CAs issues downstream seem to be solved for now
* NODE_FAILURE on internal CS jobs is sorted out now

### 2023-05-01
* centos9 mirrors all failing - out of sync - see discussion on #opendev and rhos-ops. Impacting all c9 jobs and content-providers - https://bugs.launchpad.net/tripleo/+bug/2018265 - posted out on openstack-discuss mailing list
* NODE_FAILURE on internal centos jobs https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=check-rdo&skip=0
* new issue @ osp 16.2 line periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2
* https://bugzilla.redhat.com/show_bug.cgi?id=2192314 periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 fails during modify-image role for repo-setup

### 2023-04-28

#### Upstream
* CentOS9 Zed: failing jobs (tempest mostly), rekick: https://review.rdoproject.org/r/c/testproject/+/46694
* CentOS9 Wallaby: failing job, rekick: https://review.rdoproject.org/r/c/testproject/+/46698
* CentOS8 Wallaby: 2023-04-28
* CentOS8 Train: 2023-04-27
#### Downstream
* el9-osp18: 2023-04-23
* looks like this is still having dependency issues. Will check with Alfredo when he returns on Tuesday
* https://issues.redhat.com/browse/PCTOOLING-710
* el9-osp17.1 2023-04-26 - mixed-os failing (line will rekick shortly)
* el9-osp17.0 2023-04-25 - new content landed today, we'll see
* el8-osp17.1 2023-04-27 - mixed-os failing
* el8-osp16.2 2023-05-01 - baremetal fix - needs merge