# Ruck Rover 2023-03-17 - 2023-03-23
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

:::warning
**Prow Status**
* [Operators - Check/Gate failures](https://prow.ci.openshift.org/?job=pull-ci-openstack-k8s-operators*&state=failure)
* For now, we just look (and investigate) if a test has multiple failures across different PRs
:::
[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Gjde-nceQZaXZSb-1F3cGA

##### ruck & rover: Sandeep (sandeep yadav), Arx (Arx Cruz)
##### ruck & rover alias: @ruckroverciframework

:::warning
### New Bugs

### Ongoing bugs

#### Upstream check
* https://bugs.launchpad.net/tripleo/+bug/2008822 tripleo-ansible "tripleo_compute_node" molecule job failing on "tripleo_iscsid" task

#### Upstream blockers
* https://bugs.launchpad.net/tripleo/+bug/2009488 tripleo modify-image role failed with format: raw error
* https://bugs.launchpad.net/tripleo/+bug/2012240 mixed-os job failing on all TestNetworkBasicOps tempest tests.
#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2170553 [RHOS-17.1][RHEL-9.2] Container image prepare is failing with No module named 'ansible'
- need to monitor
* https://bugzilla.redhat.com/show_bug.cgi?id=2179806 17.1/8 Container build is failing with deps issue: nothing provides network-scripts-openvswitch3.1 needed by rhosp-network-scripts-openvswitch-3.1-1.el8osttrunk.noarch

### Solved bugs.
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2180319 RHOSP17.1 Mixed os job is failing with TypeError: relationship() got an unexpected keyword argument 'sync_backref'~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012363 Featureset39 train fails with libibverbs-44.0-2.el8.1.x86_64.rpm: signature hdr data: BAD, no. of bytes(15176) out of range~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2012489 Diskimage-builder failing to install in train job~~
:::

### Promotion Status
::: info
#### Upstream promotion
CentoS 9 Master: 2023-03-21
CentOS 9 Zed: 2023-03-22 (no new hash)
CentOS 9 Wallaby: 2023-03-13 (Red mixed os bug ongoing)
CentOS 8 Wallaby: 2023-03-14 (Red mixed os bug ongoing)
CentOS 8 Train: 2023-03-21 (Last bug on image build solved yesterday)
#### Downstream Promotion
RHEL 9 RHOS 18.0: 2023-03-22
RHEL 9 RHOS 17.1: 2023-03-22
RHEL 9 RHOS 17.0: 2023-03-07 (no new content)
RHEL 8 RHOS 17.1: 2023-03-21 (no new content)
RHEL 8 RHOS 16.2: 2023-03-22
:::

# Thursday, 23 March 2023

## gate failure, following under recheck
* https://review.opendev.org/c/openstack/tripleo-ansible/+/873474

## Upstream Integration line
## Downstream Integration line

# Wednesday, 22 March 2023

## Gate failures, following under recheck
* https://review.opendev.org/c/openstack/tripleo-ansible/+/873474/

## Upstream Integration lines
* Master - new hash available - line running right now
* Zed - All green
* Wallaby - mixed os bug still blocking promotion
* Train
* New failure in image build job: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-buildimage-ironic-python-agent-train/0111b86/job-output.txt

#### Downstream Integration lines
* 18/9 - all green
* 17.1/9 - waiting for jenkins job
* 17.1/8 - all green
* 17/9 - no new content
* 16.2
* full-tempest
* random tempest
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2/a0f7b32/logs/undercloud/var/log/tempest/failing_tests.log
* jenkins

# Tuesday, 21 March 2023

#### Upstream Integration lines
* Master
* ipa - random ssh issue
* https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/32d1d2e/job-output.txt
* fs020 - random tempest failure
* https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/173e170/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
* rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
* Zed
* All green
* Wallaby - mixed Os bug under investigation
* Train
* fs039 - internal job- debugging.
* https://bugs.launchpad.net/tripleo/+bug/2012363 Featureset39 train fails with libibverbs-44.0-2.el8.1.x86_64.rpm: signature hdr data: BAD, no. of bytes(15176) out of range

#### Downstream Integration lines

# Monday, 20 March 2023

##### Gate failures
Following under recheck
* https://review.opendev.org/c/openstack/tripleo-ansible/+/877305/
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/858845/
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/869961/

New failure in zed/master
https://zuul.opendev.org/t/openstack/build/a57134cf29aa46488415eb54388808a3
~~~
Last metadata expiration check: 0:02:28 ago on Mon 20 Mar 2023 08:11:15 AM UTC.
No package containers-common-1-40.el9 available.
Error: No packages marked for downgrade.
~~~

Fixes are up:-
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/877761~~
* ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/877762~~

#### Upstream Integration lines
* Master - All green
* ~~fs002 - random failure~~
* ~~rerunning: https://review.rdoproject.org/r/c/testproject/+/39420~~
* Zed - All green
* Wallaby - periodic-tripleo-ci-centos-9-8-multinode-mixed-os tempest failure
* https://bugs.launchpad.net/tripleo/+bug/2012240
* Train - line run got skipped due to infra issue (waiting for 09:00 when line retriggers today)

#### Downstream Integration lines
* 18/9 - All green
* 17.1/9
* Blocked due to periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1
* Which skips because 17.1/8 container build fails
* 17.1/8
* Failing container build failing
* Filed: https://bugzilla.redhat.com/show_bug.cgi?id=2179806 17.1/8 Container build is failing with deps issue: nothing provides network-scripts-openvswitch3.1 needed by rhosp-network-scripts-openvswitch-3.1-1.el8osttrunk.noarch
* 17/9 - All green - no new content
* 16.2/8 - All green - no new content

#### Downstream component lines
* 17.1/9 - All green
* 18.9 - All green - we don't have jobs in component line
* 17.1/8 - All green - we don't have jobs in component line
* 17/9 - All green
* 16.2/8 - tempest - jenkins jobs - retriggered

# Thursday, 17 March 2023

#### Upstream/RDO Issues
* ~~All the rdo jobs failing with POST FAILURE and no logs captured~~ Issue fixed now
* ~~"There was an issue creating /var/www/logs/90/47890 as requested: [Errno 28] No space left on device: '/var/www/logs/90/47890~~
* ~~We reached out to #rhos-ops on slack.~~

#### Upstream integration lines
* Master
* Tracking hash: Promoted today
* rr --release master --aggregate_hash tripleo-ci-testing/58/42/58426a1999597e7ac2b17b4f0a6a0caf
* ~~fs01~~
* random tempest failure: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/9e4b408/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
* ~~fs020~~
* Consistent tempest failure: we have a bug: https://bugs.launchpad.net/tripleo/+bug/2011824
* https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/4a1e013/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
* Posted a skiplist patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/877749
* Testing here: https://review.rdoproject.org/r/c/testproject/+/46960

* Zed:
* ~~[sc04](https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario004-standalone-zed/3183628/job-output.txt) - random ssh failure~~
* ~~[container multinode](https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-zed/e194436/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz) - random tempest~~
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39357

* Wallaby C9:
* periodic-tripleo-ci-centos-9-8-multinode-mixed-os: logs not available - post_failure issue in morning - rerunnin again
* testproject: https://review.rdoproject.org/r/c/testproject/+/28537

* Train - All green

#### Upstream Component lines
* Master - All green
* Zed - All green
* Wallaby - All green
* Train - All green

#### Downstream Integration lines
* 18/9 - All green
* 17.1/9
* periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1: tempest failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1/1fc08f0/logs/undercloud/var/log/tempest/failing_tests.log
* rerunning: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540