# Ruck Rover 2023-05-26 - 2023-06-02

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/0iGbL43uTPa9YtjWtdWfjQ

##### ruck & rover: soniya29 (svyas)
##### ruck & rover alias: @ruckroverciframework

:::info
**Fixes and Improvements needed**

* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
* https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
* periodic build-image jobs were failing because of that

:::

:::danger

**BUGS**

* https://bugzilla.redhat.com/show_bug.cgi?id=2209633 Tripleo Standalone jobs with ceph 6.0 failing with "msg": "JSONDecodeError: Expecting value: line 1 column 1 (char 0)"} and PermissionError: [Errno 13] Permission denied: '/var/lib/kolla/config_files/src-ceph/ceph.conf'
* Upstream Fixes
* ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/883825~~
* https://review.opendev.org/c/openstack/tripleo-ansible/+/884332
* Downstream
* ~~https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/443461~~
* https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/443477
* Tripleo-ansible fixes are failing , tht fixes got merged

* https://bugs.launchpad.net/tripleo/+bug/2019192 train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0

* rekicked here

* https://bugzilla.redhat.com/show_bug.cgi?id=2210225 17.1 Mixed rhel job fails with error 'Can't run container container-puppet-neutron

* Downstream is hitting RETRY_LIMIT issues - please check again in morning
:::
## Thursday 1/06/2023
#### Upstream promotion

* Master: Green 2023-06-01
* Antelope: Green 2023-05-
* periodic-centos-9-podified-edpm-integration-antelope is failing
* https://logserver.rdoproject.org/a0/a03f8684d8e855ba4c92c18bf827604e7596094b/openstack-periodic-container-antelope-centos9/periodic-centos-9-podified-edpm-integration-antelope/4326458/job-output.txt
* CentOS 9 Wallaby 2023-06-01
* CentOS 8 Wallaby 2023-06-01
* CentOS 8 Train 2023-05-31
* https://review.rdoproject.org/r/c/testproject/+/46082

### Upstream components:

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
* no new hash
* RHEL 9 RHOS 17.1: RED 2023-05-22
* fs20 internal and full_tempest_Scenario - https://bugzilla.redhat.com/show_bug.cgi?id=2211604
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443590

* RHEL 9 RHOS 17.0: 2023-05-26
* no new hash after 26/05
* RHEL 8 RHOS 17.1: 2023-06-01

* RHEL 8 RHOS 16.2: 2023-05-25
* no new hash after 25
* https://code.engineering.redhat.com/gerrit/c/testproject/+/430284

#### Downstream components
17.1 rhel9 :- network
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
* sc10 is still blocked on ceph issue

## Wednesday 31/05/2023
#### Upstream promotion

* Master: Green 2023-05-31
* Antelope: Green 2023-05-31
* CentOS 9 Wallaby 2023-05-29
* https://review.rdoproject.org/r/c/testproject/+/40248
* CentOS 8 Wallaby 2023-05-31
* CentOS 8 Train 2023-05-26
* no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5

### Upstream components:
cs9 netowrk:
#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
* no new hash on 25th
* RHEL 9 RHOS 17.1: RED 2023-05-22
* blocked on ceph issues
* lots of node_failures - pinged on #rhos-ops
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443606
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443590

* RHEL 9 RHOS 17.0: 2023-05-26
* no new hash after 26/05
* https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 8 RHOS 17.1: 2023-05-26
* blocked on mixed_rhel which is facing node_Failure
* chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
* no new hash after 25

#### Downstream components
17.1 rhel9 :- tripleo/network/common
- https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430517
## Tuesday 30-05-2023
#### Upstream promotion

* Master: Green 2023-05-30

* Antelope: 2023-05-
* https://review.rdoproject.org/r/c/testproject/+/42692
* CentOS 9 Wallaby 2023-05-

* CentOS 8 Wallaby 2023-05-
* CentOS 8 Train 2023-05-26
* https://review.rdoproject.org/r/c/testproject/+/40248
* https://review.rdoproject.org/r/c/testproject/+/40102
* no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
* no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
* blocked on ceph issues
* lots of node_failures - pinged on #rhos-ops
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443606
* RHEL 9 RHOS 17.0: 2023-05-26
* no new hash after 26/05
* blocked with node_Failures
* RHEL 8 RHOS 17.1: 2023-05-26
* blocked on mixed_rhel which is facing node_Failure
* chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
* no new hash after 25

## Monday 29-05-2023
#### Upstream promotion

* Master: Green 2023-05-29

* Antelope: 2023-05-
* tried enqueing line but line disappears immediately.
* 500 internal server error
* https://logserver.rdoproject.org/openstack-periodic-container-antelope-centos9/review.rdoproject.org/rdo-jobs/master/periodic-centos-9-antelope-promote-consistent-to-podified-ci-testing/9a2ec13/job-output.txt
* https://review.rdoproject.org/r/c/testproject/+/46082
* CentOS 9 Wallaby 2023-05-29
* CentOS 8 Wallaby 2023-05-29(green)
* CentOS 8 Train 2023-05-26
* no new hash after 26 https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
* no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
* Newer hash available on 26th
* blocked on ceph issues
* lots of node_failures - pinged on #rhos-ops
* https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 9 RHOS 17.0: 2023-05-26
* no new hash after 26/05
* RHEL 8 RHOS 17.1: 2023-05-26
* new hash available
* blocked on node_failures
* chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2: 2023-05-25
* no new hash after 25

##### Downstream components
* RH9 17.1: tripleo/network/promote
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443587
* https://code.engineering.redhat.com/gerrit/c/testproject/+/443590
* RH8 16.2:
* RH9 17.1:
## Friday 26-05-2023

#### Upstream promotion

* Master: Green 2023-05-26(green)
* Antelope: 2023-05-26(green)
* CentOS 9 Wallaby 2023-05-26(green)
* CentOS 8 Wallaby 2023-05-26(green)
* CentOS 8 Train 2023-05-21
* no new hash (25th)

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-23
* no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
* Newer hash available on 26th
* blocked on ceph issues
* https://code.engineering.redhat.com/gerrit/c/testproject/+/430284
* RHEL 9 RHOS 17.0: 2023-05-26
* RHEL 8 RHOS 17.1: 2023-05-26
* new hash available
* blocked on mixed rhel
* RHEL 8 RHOS 16.2: 2023-05-25
* no new hash

##### Downstream components
* RH9 17: green
* RH8 16.2: green
* RH9 17.1:
* compute blocked on mixed rhel
* network sc10
* tripleo sc04

* rr tool has some issues while showing results
* http://pastebin.test.redhat.com/1101315
* screenshot of pipeline -