# Ruck Rover - 20th May 2022 - 26th May 2022

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/S6LEv7rFS82q3Ma2RKHHdw?both
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

## Thursday May 26th

Check/Gate
* https://review.opendev.org/c/openstack/python-tripleoclient/+/841286/ - mirror issue - posted recheck
* https://review.opendev.org/c/openstack/python-tripleoclient/+/841205/4#message-dc78079bf746f4e124f4b431fc5c8236d78ed1a9 - may be legit - rerunning to confirm if issue is consistent.

* Master, last_promotion=26th May
* Master promoted today
* ~~[Master jobs running on vexx cloud failing during task - Install deps from upper-constraints, ERROR: Cannot install -r /home/zuul/src/opendev.org/openstack/python-tripleoclient/requirements.txt (line 14) because these package versions have conflicting dependencies. ](https://bugs.launchpad.net/tripleo/+bug/1975784)~~
* cleared after pypi cache was cleared
https://review.rdoproject.org/r/c/testproject/+/28446 - testing if jobs are back to green.

* Master Component line, following components are older than 1 week.
* clients : https://review.rdoproject.org/r/c/testproject/+/43135
* ~~security~~
* tripleo : https://review.rdoproject.org/r/c/testproject/+/43135 (fs001 failed last due to NTP sync bug)
Because of missing containers - some jobs failed - should clear in today's run - not rerunning as we down on resources.

* Wallaby C9 - Last promotion 2022-05-25 - All green (No new hash when I check this morning.) -

* Wallaby c8 - Last promotion 2022-05-23
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby failing

* Victoria c8 - Last promotion 2022-05-23 - All green (No new hash when I check this morning.)

* Train c8 - Last promotion 2022-05-16
* fs035 failing(tempest) - in rerun https://review.rdoproject.org/r/c/testproject/+/39960
* https://bugs.launchpad.net/tripleo/+bug/1972163 - again back - reopened the bug/cix.

* 16.2 - Last promotion 2022-05-16
* Legit bug: https://bugzilla.redhat.com/show_bug.cgi?id=2089304
* 17/8 - Last promotion 2022-05-19
* Legit bug: https://bugzilla.redhat.com/show_bug.cgi?id=2089304
* 17/9 - Last promotion 2022-05-19
* https://bugzilla.redhat.com/show_bug.cgi?id=2089945
* dib patch merged.

## Wednesday May 25th (and earlier ongoing things tracked here)

#### Check/gate

All green currently

#### Promotion pipelines:-

Master (promoted today) :-)

* https://bugs.launchpad.net/tripleo/+bug/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
* Vexxhost have reduced the cpu overcommit from 4.5 to 1.5
* Reduced max-servers to 60, but we can see issue in ovb stack creation - ovb jobs spin up more than 1 node because it starts a job which launches a heat stack
* Situation seems better - We promoted master/wallaby without skipping anything - Need more observation.

* [Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest](https://bugs.launchpad.net/tripleo/+bug/1973223)
* Not consistent failure as per nova team, this is because of slow node.. Can we run this job on a bigger node?

Master component line:-
* [Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503](https://bugs.launchpad.net/tripleo/+bug/1973568)
* Security component promoted today.
* We can revert the [skiplist](https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/841934) now.

Wallaby c9 (Promoted yesterday) - Line triggered in afternoon - no new hash till morning

Wallaby c8 (Promoted on 23rd may)
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* Waiting for neutron team

Victoria - 2022-05-23 - no new hash all green

Train - 16th May
* https://bugs.launchpad.net/tripleo/+bug/1972163 - tempest cinder tests (reopened this bug today)
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43171 - skipping fs035 in criteria

Downstream

16.2 - Last promotion 2022-05-16 / 17/8 - Last promotion 2022-05-19
* [s020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle](https://bugzilla.redhat.com/show_bug.cgi?id=2089304)
* Neutron team (Yatin working on this)

17/9

* https://bugzilla.redhat.com/show_bug.cgi?id=2089945 RHEL-9 GA: periodic-tripleo-rhel-9-buildimage-overcloud-hardened-uefi-full-rhos-17 fails with "wrong fs type, bad option, bad superblock on /dev/mapper/loop0p3"
* Dib patch merged
* Need to check if fix have merged downstream and rerun the job.
* Trying to promote wallaby common component - https://review.rdoproject.org/r/c/testproject/+/42657

* https://bugzilla.redhat.com/show_bug.cgi?id=2089724 RHEL-9 tripleo_nodes_validation failing with packet loss in the Network availability validation block
* Waiting for newer ovn

--
## ***STOP (all tracked bugs duplicated above stop scrolling) STOP***

---

## Tuesday May 25th

* ~~[RDO Third party check jobs are failing because master containers are not available.](https://bugs.launchpad.net/tripleo/+bug/1975671)~~
* ananya/daniel have restored the hash, this is fixed now

* ~~[Master: Image build failing due to dependency issues, Problem: package fence-agents-rhevm-4.10.0-23.el9.noarch requires fence-agents-common = 4.10.0-23.el9, but none of the providers can be installed](https://bugs.launchpad.net/tripleo/+bug/1975551)~~
* Resolved itself after mirror sync
* permanent fix also merged.

* ~~[Master: Overcloud deployment failing on task "FATAL | Ensure system is NTP time synced"](https://bugs.launchpad.net/tripleo/+bug/1975591)~~
* ~~Fix is up https://review.opendev.org/c/openstack/python-tripleoclient/+/843196~~

* ~~https://bugs.launchpad.net/tripleo/+bug/1975588 Wallaby: Node provisioning failing, Overcloud nodes not booting properly - System BootOrder not found.~~
* last run passed - probably transient

---
## Tuesday May 24th

#### Check/gate
* Looks good today, below patches are in rerun
* ~~https://review.opendev.org/c/openstack/puppet-tripleo/+/842958/~~
* ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/834594/~~
* New bugs:
* https://bugs.launchpad.net/tripleo/+bug/1975646 -
tripleo tests are failing os_tempest with "openstacksdk library MUST be >=0.99."
* https://bugs.launchpad.net/tripleo/+bug/1975623 -
tripleo-ci-centos-9-undercloud-upgrade is failing check/gate with "cannot install both NetworkManager-1:1.39.2-1.el9.x86_64 and NetworkManager-1:1.39.5-1.el9.x86_64" (hoping this is just a mirror sync issue)

#### Master integration line,
* Image build failing - reported below bug
* [Master: Image build failing due to dependency issues, Problem: package fence-agents-rhevm-4.10.0-23.el9.noarch requires fence-agents-common = 4.10.0-23.el9, but none of the providers can be installed](https://bugs.launchpad.net/tripleo/+bug/1975551)
*

#### Wallaby integration line, last_promotion=21stMay
* [Wallaby: Node provisioning failing, Overcloud nodes not booting properly - System BootOrder not found.](https://bugs.launchpad.net/tripleo/+bug/1975588)

#### Wallaby c8 integration line- last_promotion - 23rd - No new hash

#### Victoria integration line- All green (No new hash when I checked this morning)

#### Train integration line, node import fix merged - rerunning jobs
* https://review.rdoproject.org/r/c/testproject/+/36256
* putting in patch to skip fs001 and fs035 on train to promo - based on https://review.rdoproject.org/r/c/testproject/+/36256- the current run failed on openstacksdk bug.
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43124 Temp remove fs001 and fs035 to promo train

### rhos-17 rhel-9:
apevec brew list-tagged --latest rhos-17.0-rhel-9-trunk-candidate rhosp-openvswitch
apevec need to untag/retag rhosp-openvswitch2.17
lon|eod ah --latest

### Component line

#### Master

* Glance - https://review.rdoproject.org/r/c/testproject/+/43099 (passed)
* Clients - https://review.rdoproject.org/r/c/testproject/+/43099
* Tripleo - https://review.rdoproject.org/r/c/testproject/+/43099
* Baremetal and others (to run after fence agent is fixed) - https://review.rdoproject.org/r/c/testproject/+/43113

#### Wallaby c9

* Compute
* periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby - tempest test_volumes_snapshots.VolumesSnapshotTestJSON
* Network
* periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby - tempest test_volumes_snapshots.VolumesSnapshotTestJSON
* Tripleo
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby - tempest test_network_basic_ops.TestNetworkBasicOps

### Wallaby c8
errors are inconsistent
* baremetal - https://review.rdoproject.org/r/c/testproject/+/43115
* tripleo - https://review.rdoproject.org/r/c/testproject/+/43115
---

---

#### May 23rd Monday

## New bug:

#### Check/Gate
* ~~[Victoria upgrade jobs failing with dict object' has no attribute 'ussuri']~~(https://bugs.launchpad.net/tripleo/+bug/1975456)
* Fix: https://review.opendev.org/c/openstack/tripleo-ci/+/842915
* ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/839789/ - ipa job failed - rechecked - passed in rerun~~

#### Promotion status

* Master integration lines - last_promotion=2022-05-16 - Need to check failing jobs
* ovb jobs ans sc010-kvm failing
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39357
* ovb issues still persist
* Ananya to check sc-10-kvm bug: https://bugs.launchpad.net/tripleo/+bug/1973223
This is what ananya found: https://trello.com/c/TP9yQsFM/2513-cixlp1973223tripleociproa-master-sc010-kvm-job-is-failing-on-octavia-related-tempest-test-octaviatempestplugintestsscenariov2tes - will try to find out what the delete is about before updating in the bug
* sandeep - To fix fs035 in downstream - ipv6 related issues.

* Wallaby c9 integration line - last_promotion=2022-05-21 Need to check failing jobs.
* ovb and full-tempest-api/sc07-mulitnode
* rerunning here https://review.rdoproject.org/r/c/testproject/+/26273

* Wallaby c8 integration lines - last_promotion=2022-05-16 Need to check failing jobs.
* fs001/fs35 failing
* fs001 passed in internal
* fs035 - only 1 run so far - failed on tempest - in rerun: https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/43047
* Let's check in few hours if issue persists
* We will promote c8 wallaby if fs035 run on psi passes
* Victoria c8 integration lines - last_promotion=2022-05-23 - All green

* Train c8 integration lines - last_promotion=2022-05-16 - Need to check failing jobs.
* Real bug: https://bugs.launchpad.net/tripleo/+bug/1975462

* 16.2 integration lines - last_promotion=2022-05-16 - Need to check failing jobs
* ysandeep to check and report bug
* 17/8 Integration line jobs - last_promotion=last_promotion=2022-05-19 - Need to check failing jobs
* ysandeep to check and report bug

* 17/9 Integration line jobs - last_promotion=last_promotion=2022-05-19 - Line just triggered will recheck in few hours.
* Ananya to check and report bug if any

Component lines

* master/wallaby/wallaby-c8/victoria - Ananya

* Master: testproj: https://review.rdoproject.org/r/c/testproject/+/43082
* Security
periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master - fails overcloud deploy with different errors connected yo keystone_db_sync
* Tripleo: periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master - inconsistent errors

* train/16.2/17 on 8/17 on 9 - Sandeep