###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/p0GEY1m5ToagXucq7FhlHg

##### ruck & rover: Sandeep
##### ruck & rover alias: @ruckroverciframework

[CIX](https://issues.redhat.com/secure/RapidBoard.jspa?rapidView=17762)

[Next Gen cockpit](https://prometheus.monitoring.softwarefactory-project.io/grafana/d/5JC-ru3Vz/github-projects-workflow-status?orgId=1)
* [doc](https://docs.google.com/document/d/14CcHuMEACfXpAy1HkSc8s_CpN1KNC_POG2lzLnbKYLM/edit)

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

[Modernization CI](https://hackmd.io/f9JzEVjMQC-hkJu1L9eolw)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
* [Operators - Check/Gate status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*)
* [pre-commit jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check&state=failure)
* [pre-commit jobs status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check)
* [unit tests jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-main-unit&state=failure)
* [kuttl tests Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-kuttl&state=failure)
* [Tempest Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-tempest&state=failure)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)

---
:::danger

## Bugs

- https://issues.redhat.com/browse/OSPCIX-63 - tripleo-build-containers-rhel-[8|9] failing in "Execute build-report" task
- slow container upload
- Ticket opened with IT: INC2724809
- https://issues.redhat.com/browse/OSPCIX-64 - wallaby intermittent ovb failure - overcloud don't boot
- this issue is still under investigation with Harald's help. We are using this test project for debug: https://review.rdoproject.org/r/c/testproject/+/46960
- https://issues.redhat.com/browse/OSPCIX-46 - envB node provisioning issue on 17/9
- Possible hardware issue, IT ticket opened: RITM1579280
:::

:::info
## Closed bugs
* ~~https://issues.redhat.com/browse/OSP-27549~~ - EDPM/Baremetal Jobs fails randomly with no endpoints available for service "nmstate-webhook
* ~~https://issues.redhat.com/browse/OSPCIX-55~~ - cs8 train: os_tempest ping router ip address task failing with Destination Host Unreachab
* in the end, iptables is pinned in release file and no fix will land for stable/train. **The last train is about to leave**.

:::

## Promotion Status

:::warning
#### Upstream
* Podified Master: 2023-09-14 (Green - no new hash)
* Podified Antelope: 2023-09-14 (Green - no new hash)
* CentOS 9 Wallaby : 2023-09-14 (Green - no new hash)
* CentOS 8 Wallaby: 2023-09-14 (Green - no new hash)
* CentOS 8 Train: 2023-09-14 (Green - no new hash)

#### Downstream
* Podified RHEL 9 RHOS 18.0: 2023-09-13 (Greeen - no new hash)
* RHEL 9 RHOS 17.1: 2023-09-12 (Greeen - no new hash)
* RHEL 9 RHOS 17.0: 2023-09-01 (Green - No new hash)
* RHEL 8 RHOS 17.1: 2023-09-07 (yellow - new hash available, last run failed due to registry issue)
* RHEL 8 RHOS 16.2: 2023-09-14 (Green - no new hash available)
---
:::

# Sept 14

* Outage on prow today: https://redhat-internal.slack.com/archives/CBN38N3MW/p1694670193577319
* we got the workaround
* RHEL 8 RHOS 17.1
* train have some failed jobs
* ~~wallaby - fs020 rerunning~~

# Sept 13
* 16.2 failed on promote job - rerunning
* Will keep an eye on today's antelope run as didn't promote yesterday
* failed yesterday on adoption job on retry
* wallaby/c9 in rerun: https://review.rdoproject.org/r/c/testproject/+/39420
* Need to test if correct jobs are triggering against OVB repo now after yesterday patches
* Continue investigation envB issue - IT ticket opened

# Sept 12
* Remove master jobs from ovb repo
* Patches:
* https://review.rdoproject.org/r/c/rdo-jobs/+/49953 Add wallaby ovb job definition for ovb repo
* https://review.rdoproject.org/r/c/config/+/49954 Update which job we trigger against ovb repo
* Debug envb issue
* 17.1/9 some scenarios failed on tempest rerunning
* https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/446183
* 16.2
* Bm job failed - tempest
* let's wait for today's run

# Sept 11
* Downstream zuul cert issue - We have informed rhos-ops
* Wallaby C9 - ovb issue continue to bite us
* Need to focus on Master/Antelope promotion

# Sept9

* wallaby c9 -
* Rerunning ovb: https://review.rdoproject.org/r/c/testproject/+/39420 - trying to reprodue node provision issue
* Wallaby c9 promoted

* Images slow upload: Created IT ticket: INC2724809