# Ruck Rover 2023-05-19 - 2023-05-25

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting indexes](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

Prow Status
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

#### Podifed Container Build Lines
* [openstack-periodic-container-master-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-master-centos9)
* [openstack-periodic-container-antelope-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-antelope-centos9&skip=0)
* [openstack-periodic-container-zed-centos9](https://review.rdoproject.org/zuul/buildsets?pipeline=openstack-periodic-container-zed-centos9&skip=0)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/0iGbL43uTPa9YtjWtdWfjQ

##### ruck & rover: doug (dviroel)
##### ruck & rover alias: @ruckroverciframework

:::info
**Fixes and Improvements needed**
* periodic-tripleo-ci-build-containers-ubi-9-quay-push-* is not building packages from *Depends-On*
* Testing it here: https://review.rdoproject.org/r/c/testproject/+/46046
* periodic-tripleo-ci-build-containers-ubi-9-quay-push-* is not working with *dlrn_hash_tag* var - it is always using *tripleo-ci-testing* hash.
* role *oooci-build-images* needs to use *discover-latest-image* to always get latest CentOS image. So we can stop manual updates on role's vars.
* https://github.com/openstack/tripleo-ci/commit/6df518c8a98bfb73d585c6f6e238f1c2ad9053f1
* periodic build-image jobs were failing because of that

:::

:::danger
**BUGS**

* https://bugzilla.redhat.com/show_bug.cgi?id=2209633 Tripleo Standalone jobs with ceph 6.0 failing with "msg": "JSONDecodeError: Expecting value: line 1 column 1 (char 0)"} and PermissionError: [Errno 13] Permission denied: '/var/lib/kolla/config_files/src-ceph/ceph.conf'
* Upstream Fixes
* ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/883825~~
* https://review.opendev.org/c/openstack/tripleo-ansible/+/884332
* Downstream
* https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/443461
* https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/443477

* https://bugs.launchpad.net/tripleo/+bug/2020774 Centos8 train image build jobs are failing due to missing Centos image.
* Fix: https://review.opendev.org/c/openstack/tripleo-ci/+/884324
* Testing here: https://review.rdoproject.org/r/c/testproject/+/46960/12
* ovb earlier run passed the image build step: https://logserver.rdoproject.org/60/46960/11/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/0cf7ac3/job-output.txt

* train ovb jobs fail modify image (repo-setup) wrong fs type, bad option, bad superblock on /dev/nbd0
* https://bugs.launchpad.net/tripleo/+bug/2019192
* not a blocker, passing on internal

* https://bugs.launchpad.net/tripleo/+bug/2020304
* Wallaby c9 jobs failing during "TASK [multi-node-bridge : Install openvswitch]" with Failed to download packages: openvswitch3.1-3.1.0-23.el9s.x86_64: Cannot download, all mirrors were already tried without success
* fixed now after fixing temporary repo

* Internal DLRN is returning **500 Internal Server Error** for some promotions, need to reach reldel if persists.

* ~~https://bugs.launchpad.net/tripleo/+bug/2020618 Centos8/RHEL8 jobs are failing on TASK [os_tempest : Gather variables for each operating system] with {"msg": "An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"}~~
* fix proposed: https://review.opendev.org/c/openstack/tripleo-ci/+/884000

* ~~https://bugs.launchpad.net/tripleo/+bug/2019507 - featureset020-wallaby is failing tempest.scenario.test_network_advanced_server_ops.TestNetworkAdvancedServerOps -failed to reach VERIFY_RESIZE~~
* Fix: https://review.opendev.org/c/openstack/tripleo-common/+/883156
* This now shows up in 17.1 https://sf.hosted.upshift.rdu2.redhat.com/logs/f6/f6cb4cde9cd049884965cec6b57d8e0b8cd6125b/openstack-periodic-integration-rhos-17.1-rhel9/periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17.1/7790488/logs/undercloud/var/log/tempest/stestr_results.html.gz

:::

:::info
#### Upstream promotion
* Master: Green(Passed 25th) https://review.rdoproject.org/zuul/buildset/af809a2fe70a4aec8d3d72e8b94e35a2
* Antelope: Green(Passed 25th)https://review.rdoproject.org/zuul/buildset/af809a2fe70a4aec8d3d72e8b94e35a2

* CentOS 9 Wallaby 2023-05-25
* No new hash (25th)
* CentOS 8 Wallaby 2023-05-25
* no new hash (25th)
* CentOS 8 Train 2023-05-21
* no new hash (25th)

#### Downstream Promotion
* RHEL 9 RHOS 18.0: 2023-05-24
* no new hash on 25th
* RHEL 9 RHOS 17.1: 2023-05-22 (Red)
* Newer hash available on 25th
* blocked on ceph issues
* RHEL 9 RHOS 17.0: 2023-05-24
* no new hash (25th)
* RHEL 8 RHOS 17.1: 2023-05-23
* new hash available
* RHEL 8 RHOS 16.2: 2023-05-23
* today's run killed due to post failure, retrigger

:::

### May 25

~~Post failure in downstream: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?result=POST_FAILURE&skip=0 - pinged #rhos-ops - fixed space issue~~

#### Check/gate
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/863201 failed yesterday due to https://bugs.launchpad.net/tripleo/+bug/2020618, In rerun now.

#### Integration lines
* CentOS 9 Wallaby
* ~~fs020 failed~~
* node provision failure:
* ~~https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby/7aa55b2/job-output.txt~~
* rerunning here:
* https://review.rdoproject.org/r/c/testproject/+/39420
* CentOS 8 Wallaby - All green
* CentOS 8 Train
- Image build issue
* https://bugs.launchpad.net/tripleo/+bug/2020774 Centos8 train image build jobs are failing due to missing Centos image.
* Fix: https://review.opendev.org/c/openstack/tripleo-ci/+/884324
* Testing here: https://review.rdoproject.org/r/c/testproject/+/46960/12
* ovb earlier run passed the image build step: https://logserver.rdoproject.org/60/46960/11/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-train/0cf7ac3/job-output.txt
* RHEL 9 RHOS 18.0 - All green
* RHEL 9 RHOS 17.1
* Blocked on Ceph bug
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1
* periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1
* rerunning: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 8 RHOS 16.2
* Post failure killed the line

#### Component lines
* CentOS 9 Wallaby
* compute:
* rerunning here:
* https://review.rdoproject.org/r/c/testproject/+/39420
* CentOS 8 Wallaby - All green
* CentOS 8 Train
* network component failed yesterday due to tempest issue
* rerunning: https://review.rdoproject.org/r/c/testproject/+/46960
* RHEL 9 RHOS 18.0 - we don't have components
* RHEL 9 RHOS 17.1
* network - ceph issue
* tripleo - ceph issue
* RHEL 9 RHOS 17.0
* tripleo component jenkins job failed
* reran: https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_component-tripleo-pcci-17_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/
* RHEL 8 RHOS 17.1 - we don't have components
* RHEL 8 RHOS 16.2 - All green

### May 24

#### Check/gate

#### Integration lines
* CentOS 9 Wallaby - All green
* CentOS 8 Wallaby
* ffu job failing
* https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-upgrade-ffu-wallaby/af7c9bc/job-output.txt
* fatal: [undercloud]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"
* Bug reported: https://bugs.launchpad.net/tripleo/+bug/2020618
* CentOS 8 Train - All green
* RHEL 9 RHOS 18.0
- Need to check via zuul
* RHEL 9 RHOS 17.1
* sc01/04/10 failing due to ceph bump, investigation ongoing with ceph team.
* Image build job failed , seems like a mirror issue:
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-9-buildimage-overcloud-full-rhos-17.1/11b0e84/logs/build.log.txt.gz
* rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1 - All green
* RHEL 8 RHOS 16.2
* Line completly red - need to check back in few hours.
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-rhos-16.2/52cfbd6/job-output.txt
* 2023-05-23 22:37:17.000827 | primary | TASK [os_tempest : Gather variables for each operating system] *****************
2023-05-23 22:37:17.001147 | primary | Tuesday 23 May 2023 22:37:17 -0400 (0:00:00.625) 0:42:27.851 ***********
2023-05-23 22:37:17.054962 | primary | fatal: [undercloud]: FAILED! => {"msg": "An unhandled exception occurred while running the lookup plugin 'first_found'. Error was a <class 'ansible.errors.AnsibleLookupError'>, original message: No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"}
* Bug reported: https://bugs.launchpad.net/tripleo/+bug/2020618

#### Component lines

* 17/9 - again wrong hash promoted

### May 23

#### Check/gate
~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/881169 in recheck~~

#### Integration lines

* CentOS 9 Wallaby - All green
* CentOS 8 Wallaby - All green
* CentOS 8 Train - All green
* RHEL 9 RHOS 18.0 - Hash is really old, need to check
* RHEL 9 RHOS 17.1 - All green
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1 - All green
* RHEL 8 RHOS 16.2
* Need to check back line in few hours
* fs035 failed: rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540

#### Component lines
* CentOS 9 Wallaby - All green
* CentOS 8 Wallaby - All green
* CentOS 8 Train - All green
* RHEL 9 RHOS 18.0 - we don't have components
* RHEL 9 RHOS 17.1
* Security
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-security-rhos-17.1/c081b5b/job-output.txt
* Rerunning Here: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1 - we don't have components
* RHEL 8 RHOS 16.2 - All green

### May 22

#### Check/gate

#### Integration lines
* CentOS 9 Wallaby
* Seeing some failure on retry_limit, need to debug
* CentOS 8 Wallaby - All green
* CentOS 8 Train - All green
* RHEL 9 RHOS 18.0 - All green
* RHEL 9 RHOS 17.1
* 1 FTBFS on tht otherwise all green
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1 - All green
* RHEL 8 RHOS 16.2
* Need to check back line in few hours

#### Component lines

* CentOS 9 Wallaby - All green
* CentOS 8 Wallaby - All green
* CentOS 8 Train - All green
* RHEL 9 RHOS 18.0 - we don't have components
* RHEL 9 RHOS 17.1
* network - tempest failure
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-network-rhos-17.1/d719a45/logs/undercloud/var/log/tempest/failing_tests.log
* rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643
* RHEL 9 RHOS 17.0 - All green
* RHEL 8 RHOS 17.1 - we don't have components
* RHEL 8 RHOS 16.2 - All green

### May 19

#### Check/gate
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/861100 - recheck
* https://review.opendev.org/c/openstack/tripleo-ansible/+/883413 - recheck

#### Integration lines
* Wallaby - yellow
* periodic-tripleo-ci-centos-9-8-multinode-mixed-os
* periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby
* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-wallaby
rerunning here: https://review.rdoproject.org/r/c/testproject/+/36356
* train - Green
* 17.1 - yellow, new hash available
* DLRN wrong promotion issue where delorean.repo md5sum don't matches.
* 17/9 - green
* 16.2/8 - green
* 17.1/8 - blocked due to dlrn issue

#### Component lines
* Wallaby - yellow
* Compute have new content and have a failing job
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compute-wallaby

rerunning here: https://review.rdoproject.org/r/c/testproject/+/36356

* train - yellow
* Tripleo - ovb job failed
* tempest: https://logserver.rdoproject.org/56/36356/91/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/1be47d8/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

reruning here: https://review.rdoproject.org/r/c/testproject/+/46960

* 17.1
* tripleo
* rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/438532

* 17.0 - Green
* 16.2 - green