# Ruck Rover - 06th May 2022 - 12th May 2022

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/ITStP2VvSYmxgNVJf56s5w?both
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

---

---
## Wed 12 May (and earlier ongoing things tracked here)

check/gate
* https://bugs.launchpad.net/tripleo/+bug/1973115 Wallaby/Victoria/Train Tripleo Content-provider job failing, Unable to pull ceph related containers because quay.ceph.io is not accessible
* Fixes are up

Master integration line
* [Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest](https://bugs.launchpad.net/tripleo/+bug/1973223)
* Latest run passed under observation - rerunning sc010-kvm again to confirm all okay - https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/42801
* https://bugs.launchpad.net/tripleo/+bug/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
* Under observation

Master Component line
* https://bugs.launchpad.net/tripleo/+bug/1973038 Master Node provisioning failing because nodes failing to boot and entering Emergency shell
* We are waiting for latest diskimage-builder to promote till current-tripleo to fix component lines.

* https://bugs.launchpad.net/tripleo/+bug/1971586 periodic ovb fs39 component security fails "Assign admin role to admin project - Internal Server Error"
* Was failing on node provisoining earlier, Test running right now with latest dib - Need to confirm if issue exists yet.

* https://bugs.launchpad.net/tripleo/+bug/1972163 Master cs9 standalone full tempest api failing for volume tempest tests and with error - 'Volume failed to detach from server'

* https://bugs.launchpad.net/tripleo/+bug/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
* https://bugs.launchpad.net/tripleo/+bug/1970899 master component network fs 1 fails provision - host not connected to any segments on routed provider network

* (cix degraded) https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

---

## ***STOP (all tracked bugs duplicated above stop scrolling) STOP***

---
## Thursday 12 May

ceph container issue merge order:-

https://review.opendev.org/q/topic:ceph_monitoring_containers

Test projects in same topic.

1. ~~[Train](https://review.opendev.org/c/openstack/tripleo-common/+/841550)~~
2. [Victoria](https://review.opendev.org/c/openstack/tripleo-common/+/841549)
3. [Wallaby](https://review.opendev.org/c/openstack/tripleo-common/+/841547) - Once train patch merged
4. [Remove Victoria Upgrade job](https://review.opendev.org/c/openstack/tripleo-ci/+/841568)

* still failing rhos-17 on rhel-8 container builds - pinged lon:
https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/b5495cb/logs/container-builds/eb5cb506-0571-45d5-8864-49d94b768b3d/base/os/gnocchi-base/gnocchi-base-build.log

components are clearing though

**NEW BUG**

[Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest](https://bugs.launchpad.net/tripleo/+bug/1973223)

* Its nova bug as per gthiemonge and johnsom, requested them to add their investigation on bug.
~~~
<johnsom> ysandeep|rover It's a nova issue, so maybe collect the nova RPM diffs or haproxy config changes.
<gthiemonge> there's a huge backtrace in nova-api.log, it looks like nova cannot fetch a security-group that was created a few seconds before in neutron
<ysandeep|rover> johnsom, gthiemonge could you please also add your investigation comments on https://bugs.launchpad.net/tripleo/+bug/1973223 as well, it would help to reach nova team.
gthiemonge> ysandeep|rover: ack
<johnsom> Yeah, nova-api: 2022-05-12 14:24:47.095 6 ERROR nova.api.openstack [req-1b96272e-da6a-4025-b37a-2c073c789cce 7527fbfe4d204aafaff987d944a25019 516d2e21db7043e8aac29216b12572e0 - default default] Caught error: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.: OSError: Apache/mod_wsgi request data read error: Partial results are valid but processing is incomplete.
ysandeep|rover>30 https://paste.openstack.org/show/beA9TQcrWxSqwX9F6Cz0/ - all the packages rpm diff
20<ysandeep|rover>30 johnsom, gthiemonge thanks for investigation help, Its late for me (2200 here), if you need any help from us please ping rlandy 
~~~

---
## May 11 Wednesday
* https://bugs.launchpad.net/tripleo/+bug/1973115
* content provider jobs failing on https://quay.ceph.io/ - not accessible (hopefully that will be gone in your morning)
* (doug): just in case it does not come back:
* https://review.opendev.org/c/openstack/tripleo-common/+/841512
* same registries configured in master branch
* question: is there a difference between 'grafana' and 'ceph-grafana' containers? they have the same tag
* testing ceph jobs: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841513
* same issue hits **stable/train**, will need a similar fix
* rhos-17 on rhel-8 - still waiting on deps respin
lon rlandy: it's assuming we're gonna hit more of what you were hitting due to all the deps I pulled back from rhel9 to rhel8
lon rlandy: none of them had autogeneration
lon the new issue here is that the version

* https://bugs.launchpad.net/tripleo/+bug/1973043 - new gate blocker:
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/841464 workaround fix (on hold)
* We reverted back bump of python-smmap in, rerunning testproject to confirm if that solves the issue.

[1] https://review.rdoproject.org/r/c/rdoinfo/+/42757
[2] https://review.rdoproject.org/r/c/testproject/+/36256
* testproject https://review.rdoproject.org/r/c/testproject/+/36256
* impacting image builds in master line
* Trying to promote node provisioning fix: https://review.rdoproject.org/r/c/testproject/+/28537
* ~~[openstack-tox-pep8 is getting timed_out on wallaby in tripleo-heat-templates](https://launchpad.net/bugs/1972940)~~
* ~~https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841282(fixed)~~

* See hardware prov chat re: https://review.opendev.org/c/openstack/diskimage-builder/+/838792 - harold said it does not have latest diskimage builder ... we need to promo there

---
## May 10 Tuesday
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2083390~~
* ~~[Master jobs failing during "Install deps from upper-constraints" / "Install openstack clients in venv" with stderr: ERROR: Cannot install oslo.concurrency>=3.26.0 because these package versions have conflicting dependencies.](https://bugs.launchpad.net/tripleo/+bug/1972765)~~ resolved after vexx pypi mirror cache clean
* Intermittent error in check - monitoring further
* https://d1a27a951c381e771447-3c8db7d9a3092adc2fa7c0c039f7a5c8.ssl.cf2.rackcdn.com/841223/1/check/tripleo-ci-centos-9-content-provider-wallaby/508441c/job-output.txt
~~~
ERROR! Error when getting the collection info for ansible.netcommon from default (https://galaxy.ansible.com/api/) (HTTP Code: 502, Message: Bad Gateway Code: Unknown)
~~~
* Wallaby - only fs001 failed - known bug https://bugs.launchpad.net/tripleo/+bug/1964940 , remove fs001 from criteria?

Pipeline monitor work divide:-
* Sandeep:-
Master
c9 wallaby
c8 wallaby
rhel9/17 intergration

* Soniya
c8 Victoria
C8 train
rhel8/16.2 intergration

---
### Monday May 09th

* Promotion status:

Master: 2022-05-06, Wallaby: 2022-05-06 , Victoria: 2022-05-09, Train: 2022-05-03
* Master - Line running - failed - ssh issue to overcloud - vexxhost is slow - sshd is taking time to come up
* Wallaby - rerunning failed jobs - Duplicate of yatin's bug
* https://review.rdoproject.org/r/c/testproject/+/28446
* Rerunning Train failed jobs here: https://review.rdoproject.org/r/c/testproject/+/39357

### Bugs
* https://bugzilla.redhat.com/show_bug.cgi?id=2083390 - [RHOS17][RHEL8]tempest and stackviz are failing - "mimeparse module missing" and "stackviz-export" errors (dd not CIX yet) Soniya pls review version and CIX if it's not just missing a patch/version update
* ~~[qdrouterd container build is failing on "nothing provides libwebsockets = 2.4.2 needed by qpid-dispatch-router-1.9.0-6.el8.x86_64](https://bugzilla.redhat.com/show_bug.cgi?id=2083067)~~ fixed
* ~~Reopened https://bugs.launchpad.net/tripleo/+bug/1961811 , hitting in ovb check - shared example in bug.~~
* Switched to ipxe-uefi for checks: https://review.rdoproject.org/r/c/rdo-jobs/+/42673 is merged.
* cs9 standalone full tempest api fails different failures https://bugs.launchpad.net/tripleo/+bug/1972163
* ~~https://bugs.launchpad.net/tripleo/+bug/1972660 - Wallaby fs001 failed with different tempest failures with error in neutron server logs: "ERROR neutron_lib.callbacks.manager TypeError: _handle_lb_on_ls() got an unexpected keyword argument 'context'"~~ - duplicate

---

## Friday May 06th

Promotion status:

Master: 2022-04-28
Wallaby: 2022-05-06
Train: 2022-05-03

From weekend:
* 17 on rhel-8: openstack-periodic-integration-rhos-17 failing - bug: https://bugzilla.redhat.com/show_bug.cgi?id=2083067: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-8-internal-rhel-8-build-push-upload-rhos-17/fc1254d/logs/container-builds/4135c3c2-f20b-4277-8b48-65901397dd9d/base/qdrouterd/qdrouterd-build.log
* 17 on rhle-9 https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel9-osp17/ - promoted
* 16.2: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/72be3e8afaee4ecf8ee7b1f197f3a43c - promoted
* RDO - nothing promoted over weekeknd - still struggling with OVB (latest testproject patch: https://review.rdoproject.org/r/c/testproject/+/39960)

### New/Transient/No bug yet
* .

### Bugs
* https://bugzilla.redhat.com/show_bug.cgi?id=2082578 [RHOS-17] jobs running on rhel-8 are failing component lines with 'nothing provides python3-gitdb >= 4.0.0 needed by python3-GitPython-3.1.14-1.el8osttrunk.noarch'

---