# Ruck Rover 2023-01-06 - 2023-01-12
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/CeFztoYvSqO-nwjxNYtKhQ

##### ruck & rover: Marios and Sandeep

### NOTE: Ongoing bugs

:::info

Check/Gate blockers

* ~~https://bugs.launchpad.net/tripleo/+bug/2002469~~ Master: Node provisioning failure with "ironic.common.exception.ConfigInvalid: Auth plugin requires parameters which were not given: auth_url"
* ~~https://bugs.launchpad.net/tripleo/+bug/2002112~~ openstack-tox-cover/tox-sanity-py39 job failure in openstack/tripleo-repos

* ~~https://bugs.launchpad.net/tripleo/+bug/2001602~~ tripleo-ansible openstack-tox-linters fails with NoInterpreter
* ~~https://bugs.launchpad.net/tripleo/+bug/2002035~~ openstack-tox-* are failing in opendev with "env name py38 conflicting with base python python3
* https://bugs.launchpad.net/tripleo/+bug/2002526 openstack-tox-linters failure against tripleo-repos

Intergation line blockers
* https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops

* ~~https://bugs.launchpad.net/tripleo/+bug/2001916 periodic standalone sc10 wallaby broken~~
* ~~Fix: https://review.rdoproject.org/r/c/rdo-jobs/+/46536~~
* ~~https://bugs.launchpad.net/tripleo/+bug/2000226~~ : xfs_growfs: /dev/mapper/vg-lv_srv is not a mounted XFS filesystem
* Workaround in place
* https://bugs.launchpad.net/tripleo/+bug/2000061 : CS-9 OVB fs001 tripleo master is unstable

Downstream blockers:
* https://bugzilla.redhat.com/show_bug.cgi?id=2154179 periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy

:::

## 12th Jan

## New Bug:
https://bugzilla.redhat.com/show_bug.cgi?id=2160392 OSP17.1/Rhel-9 Ceph based standalone jobs failing on Ceph Deploy with error "FATAL | Wait for expected number of osds to be running"
cephadm latest package corresponds to 5.3, container tag is still 5.2 - mismatch is causing issue - Ceph team checking internally about right tag for 5.3.

Gate failures:
https://review.opendev.org/c/openstack/tripleo-quickstart/+/869820/ - tempest failure and sc000 install failure - under recheck
https://review.opendev.org/c/openstack/tripleo-ansible/+/868277/ - content-provier - "Failed to pull following containers: ['quay.io/prometheus/prometheus:v2.18.1', 'quay.io/prometheus/node-exporter:v0.18.1']"} - under recheck
* internal jobs getting aborted e.g. https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train
* e.g uploaded v119 and then before zuul reported, uploaded v120 https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/46510/
* might need to tweak the time or have different patch for each branch
* posted https://review.rdoproject.org/r/c/config/+/46584

#### **Upstream Integration line promotion status**

- Master CS9 - 2023-01-12
- Zed - 2023-01-12
- Wallaby c9 - 2023-01-12
- Wallaby c8 - 2023-01-12 - Green
- Train C8 - 2023-01-12 - Green

#### **Upstream Component line promotion status**
- Master CS9
- compute
- network
- octavia
- tempest
- Zed CS9
- network
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-zed/da346df/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- Wallaby CS9
- network
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/7bcbfd0/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/b1555ed/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- compute
- all tempest failure
- https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-compute-wallaby/a3ce8c9/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-wallaby/c4ef9a6/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby/3a297ae/job-output.txt
- Wallaby CS8 - All green
- Train CS8 - All green
- Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/39357

- **Downstream Integration line promotion status**
- RHOS-17.1 RH9 - 2023-01-10 - Green
- newer hash available
- sc01: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1/bb557d4/logs/undercloud/var/log/tempest/failing_tests.log
- fs01: node provision failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1/162dcff/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz
- fs035: node provision failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1/ffbde0c/logs/undercloud/home/zuul/overcloud_node_provision.log.txt.gz
- Rerrunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
- RHOS-17.1 RH8 - 2023-01-11 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-11 - Green
- Jobs running right now, will check back in few hours

- **Downstream Component line promotion status**
- RHOS-17.1 RH9
- cinder - - sc01/sc04 - failure could be legit - rerunning before opening a bug
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-cinder-rhos-17.1/10459c9/job-output.txt
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario004-standalone-cinder-rhos-17.1/f45d8ac/job-output.txt
- network - ovb
- introspection failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-network-rhos-17.1/8df040a/job-output.txt
- RHOS-17.1 RH8 - Green - no component jobs
- RHOS-17 RH9
- network :
- standalone https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-network-rhos-17/3527070/logs/undercloud/home/zuul/standalone_deploy.log
- tempest-api https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-network-rhos-17/9d8bb33/logs/undercloud/var/log/tempest/failing_tests.log
- RHOS-16-2 RH8 - All green
- Rerunning failed jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643

## 11th Jan

#### gate
https://review.opendev.org/c/openstack/tripleo-ansible/+/869360/ - tox-py310 failure - ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/community/crypto/versions/2.3.2/': The read operation timed out - rechecked -> passed on recheck
* please check https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-linters&project=openstack/tripleo-quickstart
* we have 2 there (otherwise/other projects green) lets see if recheck still hits https://review.opendev.org/c/openstack/tripleo-quickstart/+/865238
* reported: https://bugs.launchpad.net/tripleo/+bug/2002526
#### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/2002469 Master: Node provisioning failure with "ironic.common.exception.ConfigInvalid: Auth plugin requires parameters which were not given: auth_url"
* https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops

* https://bugs.launchpad.net/tripleo/+bug/2002526 openstack-tox-linters failure against tripleo-repos

#### **Upstream Integration line promotion status**

- Master CS9 - 2023-01-10 - Green
- New hash available
- All ovb jobs failing - reported https://bugs.launchpad.net/tripleo/+bug/2002469
- sc07/ovn-provider - rerunning here to confirm if consistent failure: https://review.rdoproject.org/r/c/testproject/+/42657
- Zed CS9 - 2023-01-11 - Green
- Wallaby CS9 - 2023-01-11 - Green
- Intermittent failure for sc10
- ovn-provider: tempest failure - same test failed - Bug reported: https://bugs.launchpad.net/tripleo/+bug/2002478
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/5ca6c98/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/bd78e32/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/20/39420/23/check/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-wallaby/d689423/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
- Wallaby CS8 - 2023-01-11 - Green
- Train CS8 - 2023-01-09 - Green
- Newer hash available
- [fs035](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/8fcbf84/job-output.txt)/[fs039](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/7b96731/job-output.txt)
- wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.",
- ~~full-tempest-api : [tempest failure](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/946a1ce/logs/undercloud/var/log/tempest/failing_tests.log)~~
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537

#### **Upstream Component line promotion status**
- Master CS9
- baremetal - same ovb bug as integration line
- network - same ovb bug as integration line
- tripleo - same ovb bug as integration line
- Zed CS9 - All green
- Wallaby CS9 - All green
- Wallaby CS8 - All green
- Train CS8 - All green

- **Downstream Integration line promotion status**
- RHOS-17.1 RH9 - 2023-01-10 - Green
- RHOS-17.1 RH8 - 2023-01-10 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-11 - Green

- **Downstream Component line promotion status**
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - Green - no component jobs
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All green

## 10th Jan
#### Gate
https://review.opendev.org/c/openstack/tripleo-ansible/+/865620/ - random ubi image pull issue - in recheck

#### **Upstream Integration line promotion status**

- Master CS9 - 2023-01-10 - Green
- Zed CS9 - 2023-01-09 - Green
- Wallaby CS9 - 2023-01-09 - Green
- Wallaby CS8 - 2023-01-10 - Green
- Train CS8 - 2023-01-09 - Green

#### **Upstream Component line promotion status**
- Master:
- Baremetal
- ovb - tempest failure: https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/330ed91/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- https://logserver.rdoproject.org/37/28537/75/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master/b893f54/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537
- Zed: All Green
- Wallaby C9: All Green
- Wallaby C8 - All Green
- Train C8
- network - internal kvm job - mirror issue
- Why we are using Vexxhost mirror for job running in internal? Need to debug further
- https://sf.hosted.upshift.rdu2.redhat.com/logs/10/46510/91/check-rdo/periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train/f19e28a/logs/undercloud/etc/yum.repos.d/quickstart-centos-appstreams.repo
- Rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643
- we are missing that https://code.engineering.redhat.com/gerrit/plugins/gitiles/openstack/tripleo-ci-internal-config/+/refs/heads/master/playbooks/tripleo-rdo-base/pre.yaml#7
- e.g. there https://sf.hosted.upshift.rdu2.redhat.com/logs/08/46508/16/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-internal-wallaby/ea7ea7d/job-output.txt but not there https://sf.hosted.upshift.rdu2.redhat.com/logs/10/46510/91/check-rdo/periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train/f19e28a/job-output.txt
- causes that https://sf.hosted.upshift.rdu2.redhat.com/logs/10/46510/91/check-rdo/periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train/f19e28a/logs/undercloud/etc/ci/mirror_info.sh export NODEPOOL_MIRROR_HOST=mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org (compare to https://sf.hosted.upshift.rdu2.redhat.com/logs/08/46508/16/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-internal-wallaby/ea7ea7d/logs/undercloud/etc/ci/mirror_info.sh.txt.gz)

- **Downstream Integration line promotion status**
- RHOS-17.1 RH9 - 2023-01-10 - Green
- ~~ovb fs-20/fs-20-rbac/fs-35 failed with retry_limit~~
- ~~Changed the network we use for ovb: https://review.rdoproject.org/r/c/config/+/46549~~
- RHOS-17.1 RH8 - 2023-01-07 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-09 - Green

- **Downstream Component line promotion status**
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - we don't have component jobs in this line.
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All Green

## 09th Jan

#### Gate failures
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/864167/ - tripleo-tox-molecule - timed out - rechecked as single failure

#### **Upstream Integration line promotion status**
- Master CS9 - 2023-01-08 - Green
- Zed CS9 - 2023-01-08 - newer hash available
- fs039 - [tempest failure](https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-zed/e1d64d2/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz)
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/42657
- Wallaby CS9 - 2023-01-08 - Green
- Wallaby CS8 - 2023-01-07 - Green
- Train CS8 - 2023-01-06- Yellow
- [fs001](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/1b7dc0f/job-output.txt)
- [fs039](https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train/ae15308/logs/supplemental/var/log/ipaserver-install.log.txt.gz)
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
- also need c-8-scenario010-kvm-internal-standalone-train 4defb2bf4d2d5225896e23af590... https://code.engineering.redhat.com/gerrit/c/testproject/+/438475

#### **Upstream Component line promotion status**
- Master: All green
- Zed - All green
- Wallaby C9 - All green
- Wallaby C8 - All green
- Train C8 - All green

- **Downstream Integration line promotion status**
- RHOS-17.1 RH9 - 2023-01-08 - Green
- RHOS-17.1 RH8 - 2023-01-07 - Green
- RHOS-17 RH9 - 2023-01-04 - Yellow
- Bm job - consistent failure on Node provisioning -
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/6ed8de8/logs/undercloud/home/zuul/overcloud_node_provision.log
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/3f09f34/logs/undercloud/home/zuul/overcloud_node_provision.log
- https://privatebin.corp.redhat.com/?64c5d4cfdd88a141#C6WUL9dqhSH6bTRUKuhG7Fo97CgCHecWPwYSdxKchx44
- 1 node failed to provision - Failed command: WRITE DMA EXT - Looks like issue with Hard disk.
- created cix directly no BZ https://trello.com/c/O7jqHzjK
- RHOS-16-2 RH8 - 2023-01-08 - Green
- Newer hash running right now - Bm job still pending - Will check back in few hours

- **Downstream Component line promotion status**
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - We don't have component jobs anymore
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All green

## 06th Jan

### New bugs:
https://bugs.launchpad.net/tripleo/+bug/2002112 openstack-tox-cover/tox-sanity-py39 job failure in openstack/tripleo-repos

#### Gate
* https://review.opendev.org/c/openstack/tripleo-ansible/+/865136/ tox failures in wallaby
* also tht/wallaby https://zuul.opendev.org/t/openstack/build/253f20d226f948a29f18e7a8cd02f84a rabi fixing with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/868464

#### **Upstream Integration line promotion status**
- Master CS9 - 2023-01-06 - Green
- Zed CS9 - 2023-01-06 - Green
- Wallaby CS9 - 2023-01-06 - Green
- Wallaby CS8 - 2023-01-06 - Green
- Train CS8 - 2023-01-05- Green

#### **Upstream Component line promotion status**
- Master: All green
- Zed
- network
- ovb
- Wallaby c9 : All green
- Wallaby c8 - all green
- Train c8 - all green

Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/28537

- **Downstream Integration line promotion status**
- RHOS-17.1 RH9 - 2023-01-~~05~~06 (Green)
- RHOS-17.1 RH8 - 2023-01-~~04~~06
- mixed os job failing (NODE_FAILURE https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1&project=openstack/tripleo-ci)
- rerunning ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/438405~~
- RHOS-17 RH9 - 2023-01-04 (Green no new content)
- RHOS-16-2 RH8 - ~~2022-12-25 (Yellow)~~ 2022-01-06
- Line running right now with new content but lot of node failures
- Need to ping on #rhos-ops again when more folks are around, pinged once already
- re-running NODE_FAILURE jobs ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/438408~~

- **Downstream Component line promotion status**
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - we don't have jobs in component
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 : All green