###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/oEM6x1zCS0y9WhH2XGQI0w

##### ruck & rover: Cedric
##### ruck & rover alias: @ruckroverciframework

[Next Gen cockpit](https://prometheus.monitoring.softwarefactory-project.io/grafana/d/5JC-ru3Vz/github-projects-workflow-status?orgId=1)
* [doc](https://docs.google.com/document/d/14CcHuMEACfXpAy1HkSc8s_CpN1KNC_POG2lzLnbKYLM/edit)

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

[Modernization CI](https://hackmd.io/f9JzEVjMQC-hkJu1L9eolw)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
* [Operators - Check/Gate status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*)
* [pre-commit jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check&state=failure)
* [pre-commit jobs status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check)
* [unit tests jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-main-unit&state=failure)
* [kuttl tests Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-kuttl&state=failure)
* [Tempest Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-tempest&state=failure)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)

---
## Bugs
- https://issues.redhat.com/browse/OSPCIX-40 - test_server_basic_ops tempest test failing on EDPM antelope Zuul job
- https://issues.redhat.com/browse/OSPCIX-41 - OSP-18 EDPM image build is failing with cp: error writing No space left on device
- https://issues.redhat.com/browse/OSP-27549 - EDPM/Baremetal Jobs fails randomly with no endpoints available for service "nmstate-webhook
- https://issues.redhat.com/browse/OSPCIX-42 - The ssl_verify_client parameter is required when setting ssl_ca (file: /etc/puppet/modules/horizon/manifests/wsgi/apache.pp on cs9 wallaby
- https://issues.redhat.com/browse/OSPCIX-45 - openstack-baremetal-operator-crc-podified-edpm-baremetal fails when /spec/roles/edpm-compute/baremetalSetTemplate/ctlplaneInterface is set to enp1s0

---
## Promotion Status

### Upstream
- Podified Master: 2023-08-17 (recovering from vexx)
- Podified Antelope: 2023-08-24
- CentOS 9 Wallaby : 2023-08-15 (recovering from vexx)
- CentOS 8 Wallaby: 2023-08-18 (recovering from vexx)
- CentOS 8 Train: 2023-08-18 (no new hash - but should get new hash on Fri Aug 25)

### Downstream
* Podified RHEL 9 RHOS 18.0: 2023-08-16 (blocked on ospcix-41)
* RHEL 9 RHOS 17.1: 2023-08-20 (no new hash)
* RHEL 9 RHOS 17.0: 2023-08-09 (bm_envB still failing)
* RHEL 8 RHOS 17.1: 2023-08-17 (no new hash)
* RHEL 8 RHOS 16.2: 2023-08-20 (no new hash)
---
## Aug 24
- RDO seems to recover from Vexx outage
- re-kicked Wallaby lines: https://softwarefactory-project.io/zuul/t/rdoproject.org/status/change/272138c
- (re-kick failed in 2 seconds with RETRY/RETRY_LIMIT...)
- Re-kicked Train promotion, we should have new content (Networking at least)
- https://review.rdoproject.org/r/c/testproject/+/46686
## Aug 23
- RDO Zuul faces mirror issues - probably related to Vexx changes.
- https://issues.redhat.com/browse/RHOSZUUL-1423
- envB issue for 17.0 on el9: apparently actual env issue, now all controllers are OK, but compute is flapping (timeout while waiting for it to boot)
- https://issues.redhat.com/browse/OSPCIX-46
- IMHO we may indeed discard that job and promote as-is. IIRC envB already proved unstable in the past. (PR pushed downstream against tripleo-environment)
## Aug 22
- RDO Zull still hitting node_failures, retry and related issues. NOT re-kicking lines yet.
## Aug 21
- RHEL 9 RHOS 17.0 re-run failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/446068
- RDO ZUUL down: https://redhat-internal.slack.com/archives/C042U5CT6TH/p1692592842118939