# Ruck Rover - 03rd June 2022 - 09th June 2022

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/E7lFF86iSyWwViYolD6g_Q?edit
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

## Thu Jun 09
### Promotions
* CS9 master & wallaby: https://review.rdoproject.org/r/c/testproject/+/41367
* ~~RHOS17 RHEL8 & RHEL9 network: https://code.engineering.redhat.com/gerrit/c/testproject/+/407601~~
* ~~RHOS17 RHEL9 glance: https://code.engineering.redhat.com/gerrit/c/testproject/+/407599/7/.zuul.yaml~~
* RHOS17 RHEL9 baremetal: https://code.engineering.redhat.com/gerrit/c/testproject/+/398308

## Active Bugs
### LibSELinux related issues
* **FB Mirror is missing packages**
* All repos are synced from fb mirror: http://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/Packages/
* waiting on a sync to http://mirror.dfw.rax.opendev.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/?C=M;O=D from
* after that, it needs to go to: http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org/centos-stream/9-stream/BaseOS/x86_64/os/Packages/

* Gate blocker: https://bugs.launchpad.net/tripleo/+bug/1977873
* Needs libselinux-3.4-2 on AFS mirror to clear this bug
* If all not works, merge this https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137
* fb mirror still not updated - expected tomorrow - can't get patch in - fails on same issue

* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ - need to recheck this patch after libselinux fix

* https://bugs.launchpad.net/tripleo/+bug/1977888 : CS8 Train diskimage builder failure due to Python dep
* https://review.opendev.org/c/openstack/tripleo-ci/+/845081
* it depends on libselinux issue
* **used this to try promote train**
* https://review.rdoproject.org/r/c/testproject/+/41367 (passed image builds - now running skipped OVB tests)
* Hash_under_test=https://trunk.rdoproject.org/api-centos8-train/api/civotes_agg_detail.html?ref_hash=a1d37ee0eb1cd07011e801784b237678
* promote tempest component for wallaby needed to clear downstream phase 2 tests
* ~~https://review.rdoproject.org/r/c/rdoinfo/+/43475~~
* It's in https://trunk.rdoproject.org/centos9-wallaby/component/tempest/component-ci-testing/ but not yet in https://trunk.rdoproject.org/centos9-wallaby/component/tempest/promoted-components/
* kicked off: https://review.rdoproject.org/r/c/testproject/+/42319

* **promoted victoria**
* **promoted train**
* **promoted rhos-17 on rhel-9**
* 16-2 had no new hash
* trying to promote 17 on rhel-8 - see QE test lagging on compute component (pinged attila) - rerunning scenario010

### Other
* https://bugs.launchpad.net/tripleo/+bug/1977716 - The conditional check ''negative_results' in name.value ' failed during set_fact
* master fix is merged; started validation for master: https://review.rdoproject.org/r/c/testproject/+/42319
* master passed all except image builds (can probably close this out)
* All tripleo component jobs passes for master and wallaby

* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

* https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.

* https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

* https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block
* RHOS 17 RHEL9 promotion will clear it
* left note in bug - pls close out bug and CIX is ok with release team

* https://bugzilla.redhat.com/show_bug.cgi?id=2093406 Container image prepare fails with Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist
* RHOS 17 RHEL9 promotion will clear it
* left note in bug - pls close out bug and CIX is ok with release team

* https://bugzilla.redhat.com/show_bug.cgi?id=2094257 Additional partition /dev/sda4 in rhel-guest-image-9.0-20220420.0.x86_64.qcow2 guest image
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952
* if bm job failes, use it as a depends on in TP
* BM RHOS-17 RHEL-9 will be fixed
* used depends-on to run BM job - will close this when patch merges

### Promotions
* cs8 train: https://review.rdoproject.org/r/c/testproject/+/41367
* cs9 master/wallaby -- waiting on fb mirror update
*

## Thu 09

### Handover Bugs

CS9 Gate Blockers:
* https://bugs.launchpad.net/tripleo/+bug/1977873 master/wallaby deployments are failing with "SELinux boolean os_enable_vtpm does not exist."
* We need master and wallaby promotion
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/845229 to unblock upstream check and gate
* https://bugzilla.redhat.com/show_bug.cgi?id=2094683 with associated patch https://review.opendev.org/c/openstack/diskimage-builder/+/845189 to unblock image builds (failing right now)

Tempest Bugs:
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
* topic:wait_until_sshable_pingable testing
* https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

RHOS-17 RHEL-9 Bugs
* https://bugzilla.redhat.com/show_bug.cgi?id=2094257 Additional partition /dev/sda4 in rhel-guest-image-9.0-20220420.0.x86_64.qcow2 guest image
* https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block
* https://bugzilla.redhat.com/show_bug.cgi?id=2093406 Container image prepare fails with Error validating CNI config file /etc/cni/net.d/87-podman-bridge.conflist
* https://bugzilla.redhat.com/show_bug.cgi?id=2094811 Collectd base container build failure with Error: Unable to find a match: collectd-dbi
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2089945 DIB fails to mount RHEL9 images on RHEL9 machines: periodic-tripleo-rhel-9-buildimage-overcloud-hardened-uefi-full-rhos-17 fails with "wrong fs type, bad option, bad superblock on /dev/mapper/loop0p3~~

CentOS 8 integration line Bugs
* https://bugs.launchpad.net/tripleo/+bug/1977888 : CS8 Train diskimage builder failure due to Python dep
* https://review.opendev.org/c/openstack/diskimage-builder/+/845217 - py36 revert

CentOS 9 Validation Components
* https://bugs.launchpad.net/tripleo/+bug/1977716 - The conditional check ''negative_results' in name.value ' failed during set_fact

Reviews in order
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137 - Once it merges - (Not sure it is needed)
* then recheck following patches
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ - cs8 component quay issue
* https://review.opendev.org/c/openstack/tripleo-ci/+/845081 - cs8 train py36 issue
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952 - Downstream RHEL guest image workaround
* https://review.opendev.org/q/f73c56eff4c17d3f22e2b71ddba8bd530b41941c - collectd missing packages
* https://review.opendev.org/c/openstack/tripleo-ansible/+/845107 - Downstream cni issue (check with sandeep before reviewing)

## Wed 08

* Master gate blocker: Needs libselinux-3.4-2 on AFS mirror to clear this bug
* https://github.com/redhat-openstack/openstack-selinux/pull/91 - merged
* working on promoting it
* https://review.rdoproject.org/r/c/testproject/+/43532 0 done
* promote to promoted component - https://review.rdoproject.org/r/c/testproject/+/43535 - don
* Not sure it is linked:
* /home/zuul/workspace/dib_build.CvdH2Jsv/hooks/cleanup.d/99-selinux-fixfiles-restore:main:95 : sudo runcon -t setfiles_mac_t -- chroot /home/zuul/workspace/tmp.3Hu52mHHxe /usr/sbin/setfiles -F /etc/selinux/targeted/contexts/files/file_contexts /
2022-06-08 01:11:02.832 | /usr/sbin/setfiles: Could not set context for /: No such file or directory
* https://logserrrver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-9-buildimage-overcloud-full-master/a6a45ad/build.log
* testing here: https://review.rdoproject.org/r/c/testproject/+/43533
* If all not works, merge this https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137

* Unblock rhos-17 on rhel-9
* Need to kick RHOS-17 RHEL-9 line
* https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-common/+/414155 got merged recentky

* promote tempest component for wallaby needed to clear downstream phase 2 tests
* https://review.rdoproject.org/r/c/rdoinfo/+/43475
* @TzBFFuLTQUS6wCI1QP-4BQ Please have a look, thanks!

* https://bugs.launchpad.net/tripleo/+bug/1977888
* https://review.opendev.org/c/openstack/tripleo-ci/+/845081
* https://review.rdoproject.org/r/c/testproject/+/43512

* https://bugzilla.redhat.com/show_bug.cgi?id=2094257
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952
* https://code.engineering.redhat.com/gerrit/c/testproject/+/211643

* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ - unblock component quay 401 registry issue

## Please Ignore anything below this line

## Tue 07

~~* Bunch of node_failures: https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE&skip=0 fbo is look0ing into that
* The issue is connected to wrong kernel behavior. infra is preparing an update~~

* Unblock rhos-17 on rhel-9
* ~~need to promote wallaby tripleo component to get https://review.opendev.org/c/openstack/tripleo-ansible/+/844478~~
* ~~promoting tripleo wallaby component here: https://review.rdoproject.org/r/c/testproject/+/43512~~
* Promote tripleo component downstream
* https://code.engineering.redhat.com/gerrit/c/testproject/+/398473
* **It failed!**
* then rekick rhos-17 rhel-9 line
* https://trunk.rdoproject.org/centos9-wallaby/component/tripleo/promoted-components/ promoted on 06/06
periodic-tripleo-centos-9-wallaby-component-tripleo-promote-to-promoted-components testproject master check 43512,2
1 min 58 secs 2022-06-07 15:22:48 SUCCESS

tripleo wallaby c9 component is still failing: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-standalone-tripleo-rhos-17/b82d4f4/logs/undercloud/var/log/tripleo-container-image-prepare.log
https://trello.com/c/jjzPJsTA/2557-cixftbfsosp-170-rhel-9
pinged jon.

* https://bugzilla.redhat.com/show_bug.cgi?id=2094257
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952
* https://code.engineering.redhat.com/gerrit/c/testproject/+/211643
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/ - unblock component quay 401 registry issue

* CS8 Wallaby promo line:
* Removed fs35 from promo line https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43524

## Mon 06
* network component is failing due to:
> Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9bfd74a3-6701-428c-9501-a43abb5278ef.'}. Server boot request ID: req-c43ea003-9233-43af-ab62-ce00636ea9e9
* Running testproject to see if that can be reproduced: https://review.rdoproject.org/r/c/testproject/+/42319
* It passed: network is off: 8 days ago | 41 minutes ago

~~* https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest ~~

### Promotions:
* cs-8 wallaby: https://review.rdoproject.org/r/c/testproject/+/42374

## Friday 03 June

### topic:wait_until_sshable_pingable testing

* Master: https://review.rdoproject.org/r/c/testproject/+/43400
* Wallaby:
* https://review.rdoproject.org/r/c/rdoinfo/+/43405 from master with unmerged patches
* https://review.rdoproject.org/r/c/rdoinfo/+/43459 - with tempest-31.0.0
* https://review.rdoproject.org/r/c/rdoinfo/+/43460 - with tempest-31.0.0 and unmerged patches

Things to check:
https://bugs.launchpad.net/tripleo/+bug/1972163
- tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest
- tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body
- tempest.api.volume.test_volumes_extend.VolumesExtendTest
- Job:
- periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master

https://bugs.launchpad.net/tripleo/+bug/1960310
- tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_rebuild_server_with_volume_attached
- tempest.api.compute.servers.test_server_rescue_negative.ServerRescueNegativeTestJSON.test_rescued_vm_detach_volume
- Job:
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master
- periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master
- periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-wallaby
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

## Friday 03 June

* CS9 wallaby promotion
* FS01, 035 and 064 re-run: https://review.rdoproject.org/r/c/testproject/+/43399
* 064 run ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success
* https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/8b7ec41/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz
* https://logserver.rdoproject.org/99/43399/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby/f37b1d6/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

* topic:wait_until_sshable_pingable testing
* Master: https://review.rdoproject.org/r/c/testproject/+/43400
* Wallaby: https://review.rdoproject.org/r/c/rdoinfo/+/43405