[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
* [Operators - Check/Gate status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*)
* [pre-commit jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check&state=failure)
* [pre-commit jobs status](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-main-precommit-check)
* [unit tests jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-main-unit&state=failure)
* [kuttl tests Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-kuttl&state=failure)
* [Tempest Jobs](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*-operator-build-deploy-tempest&state=failure)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Uakv4hhKQxO9D4JKlbMUEw

##### ruck & rover: Bhagyashris (slack - bshewale)
##### ruck & rover alias: @ruckroverciframework

https://review.rdoproject.org/r/c/testproject/+/48811
https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/884993

:::info
**Fixes and Improvements needed**

* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
* https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
* periodic build-image jobs were failing because of that

:::

:::danger

**OPEN BUGS**

:::

:::info
### Promotion status

#### Upstream promotion

* Podified Master: 2023-07-06
* Podified Antelope: 2023-07-06
* CentOS 9 Wallaby 2023-07-06

* CentOS 8 Wallaby 2023-07-05
* CentOS 8 Train 2023-07-03

#### Downstream Promotion
* Podified RHEL 9 RHOS 18.0: 2023-07-05
* RHEL 9 RHOS 17.1: 2023-07-03
* RHEL 9 RHOS 17.0: 2023-06-22
* NNH after 3876eba14585e0a14a0597a43c5e5e6c
* https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/tripleo-ci-testing/delorean.repo.md5
* RHEL 8 RHOS 17.1: 2023-06-30
* NNH after: 986929b258c826ba337d7bcd49bcb9a4
* https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17-1/tripleo-ci-testing/delorean.repo.md5
* RHEL 8 RHOS 16.2: 2023-06-30
* NNH after: 44bd2351fb386fb2698ce09abb9d6c51
* https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/tripleo-ci-testing/delorean.repo.md5
:::

## July 6, 2023

### Upstream

### Downstream:

* Error: Unable to match mock;
* https://issues.redhat.com/browse/RHELBLD-13214
* Plz check/read all the discussion on : https://redhat-internal.slack.com/archives/C042U5CT6TH/p1688377167621249 **Almost Resolved this issue from integration line results**

## July 5, 2023

### Upstream

* cs9 wallaby:
* Bug: https://bugs.launchpad.net/tripleo/+bug/2025747
* talkedked on rhos-ops channel: https://redhat-internal.slack.com/archives/C042U5CT6TH/p1688364774281039
* https://review.rdoproject.org/r/c/rdo-jobs/+/49169 Run cs9 and cs8 wallaby jobs on vexxhost node [NEW] **Plz check the commit message for more info**

### Downstream:

* NODE_FAILURE issue:
* https://redhat-internal.slack.com/archives/C04MLRF641W/p1688547539504809
* https://redhat.service-now.com/help?id=rh_ticket&table=incident&sys_id=60f2129e97372514f5d07c7e6253afe7&view=ess
* https://issues.redhat.com/browse/RHOSZUUL-1375
* **The ISSUE is RESOLVED**

* Error: Unable to match mock;
* https://issues.redhat.com/browse/RHELBLD-13214
* Plz check/read all the discussion on : https://redhat-internal.slack.com/archives/C042U5CT6TH/p1688377167621249

## July 4, 2023

### Upstream

* cs9 wallaby:
* Bug: https://bugs.launchpad.net/tripleo/+bug/2025747
* talkedked on rhos-ops channel: https://redhat-internal.slack.com/archives/C042U5CT6TH/p1688364774281039

* cs8 train:
* Note We are facing same issue as above in the cs9 wallaby

* Component:
* cs9 wallaby:
* re-run the failed jobs here: https://review.rdoproject.org/r/c/testproject/+/47232

### Downstream:

* rhos17.1 on rhel9:
* https://issues.redhat.com/browse/RHELBLD-13214
* https://redhat-internal.slack.com/archives/C042U5CT6TH/p1688377167621249

## July 3, 2023

### Upstream
* cs9 wallaby:
* Almost all the jobs are stuck and failed at "External deployment step 1" not sure is it timed out issue
* for timeout issue... failing on container image prepare - logs here
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/opens[…]s/undercloud/var/log/tripleo-container-image-prepare.log
* re-ruuning jobs here: https://review.rdoproject.org/r/c/testproject/+/49103 - most of the jobs **PASSED** in the testproject, only sc012 and mixed-os jobs are way from promotion so re-running those in the same testproject.

* cs8 wallaby:
* re-running failing jobs here: https://review.rdoproject.org/r/c/testproject/+/44874 [DNM] Test failing cs8 wallaby job

* cs8 train:
* re-running failing job here: https://review.rdoproject.org/r/c/testproject/+/49104 [DNM] Test failing cs8 train jobs

### Downstream:
* rhos17.1 on rhel9.2:
* jenkins jobs are failing - rekicked those jobs

## July 1, 2023

* looks like we have an issue with IBM cloud - component lines for train are scheduling very slowly. If not cleared by your morning - pls ping #rhos-ops on this
* **reply from : TalmoR** - there is an outage in brew that I know is affecting us. could be rlated for this issue too?

* wallaby still has not promoted ... trying https://review.rdoproject.org/r/c/testproject/+/47875/27/.zuul.yaml to rerun the one failed updates jobs from friday (fix only merged at 3am that night) and there are some failures from the current line kick that are rerunning
* Downstream rr tool is broken - you can see the error from the command line and in the downstream cockpit

## Handover notes

## Jun 30, 2023

### Upstream:
* cs9 wallaby:
* testing failed jobs here : https://review.rdoproject.org/r/c/testproject/+/49103 [DNM] test failing cs9 wallaby jobs
* Bug: https://bugs.launchpad.net/tripleo/+bug/2025473
* fix: https://review.opendev.org/c/openstack/tripleo-upgrade/+/887366
* testproject: https://review.rdoproject.org/r/c/testproject/+/49103

* cs8 wallaby:
* testing failed job here : https://review.rdoproject.org/r/c/testproject/+/49104 - **PASSED**