Ruck Rover - 27th May 2022 - 02nd June 2022

tags: `ruck_rover`

Previous RR notes: https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ?view

Cockpit

Downstream cockpit

Thursday 02 June

new/transient/no bug yet

https://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-wallaby/8dade15/job-output.txt
- ~~because of https://review.rdoproject.org/r/c/rdo-jobs/+/42881~~ ?
  - trying/test with https://review.rdoproject.org/r/c/testproject/+/43374 Depends-On: https://review.rdoproject.org/r/c/rdo-jobs/+/43360
  - not this same result https://logserver.rdoproject.org/78/43378/3/check/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master/ff217a2/job-output.txt
- because of this https://review.opendev.org/c/openstack/tripleo-ci/+/844389 ?
  - testing v4 @ https://review.rdoproject.org/r/c/testproject/+/43378 https://review.rdoproject.org/r/c/testproject/+/43374 looks good so far

Bugs:

https://bugs.launchpad.net/tripleo/+bug/1976614 victoria standalone & undercloud upgrade jobs broken at undercloud-setup package deps

Wednesday 01 June (and earlier ongoing things tracked here)

Bugs:

https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block
~~https://bugs.launchpad.net/tripleo/+bug/1973568~~ Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
~~https://bugzilla.redhat.com/show_bug.cgi?id=2089304~~ fs020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle (edit)

STOP (all tracked bugs duplicated above stop scrolling) STOP

Tuesday 31 May

Bugs:

https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
https://bugzilla.redhat.com/show_bug.cgi?id=2089304 fs020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle (edit)
https://bugs.launchpad.net/tripleo/+bug/1973568 Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block

downstream:
- rhos17 on rhel9 :
  - build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17 failing due [1]
    - [1] https://bugzilla.redhat.com/show_bug.cgi?id=2091816
    - fix: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/412327 (works)
- TO Do's:
  - for 16.2:
    - keep eye on fs020 results on testproject patch: https://code.engineering.redhat.com/gerrit/c/testproject/+/315285
    - PROMOTED
  - for rhos17 on rhel8: waiting for next integration run result
    - (rlandy) fix for failing test - https://review.opendev.org/c/openstack/neutron/+/843763/ is not yet passed in the network component line and has not reached downstream. Rerunning failed jobs: https://review.rdoproject.org/r/c/testproject/+/36254
  - for rhos17 on rhel9: waiting for tesproject patch result: https://code.engineering.redhat.com/gerrit/c/testproject/+/412202 once it pass need to rekick the whole line
    - (rlandy) containers and images builds pass - tests are still failing on https://bugzilla.redhat.com/2089724 (new ovn did not help) NEEDS INVESTIGATION TOMORROW

Promotions upstream handoff:

1 TRAIN

latest buildset https://review.rdoproject.org/zuul/buildset/0eb15e91b17d49dca312ae072d972aea - many fails stackviz thing https://bugs.launchpad.net/tripleo/+bug/1976247
digging for candidate @ http://promoter.rdoproject.org/promoter_logs/centos8_train.log - https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/81/64/ from 27th :/
rekicked line manually => decent result https://review.rdoproject.org/zuul/buildset/87f1a28cf802437f8823f57f9d2efe4c only fs35 timeout
posted chaser fs35 rerun https://review.rdoproject.org/r/c/testproject/+/43298
(rlandy) rerun twice - second run still in progress (if not same tempest tests, suggestion to comment this test out and promoted)

2 ~~WALLABY 9~~ promoted

nice build @ https://review.rdoproject.org/zuul/buildset/bc85c2bb90bd4b0696a407f79c40fa84
brief dig inconsistent fails @ https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby
posted chaser @ https://review.rdoproject.org/r/c/testproject/+/43284
recheck tempest :/ https://review.rdoproject.org/r/c/testproject/+/43284/1#message-6da3a4d7a2b099ba15941eaae92637a645f0af94 => green & promoted

3 WALLABY/8

buildset https://review.rdoproject.org/zuul/buildset/0d2c138a8d4d4a87a4c8fc92f6d55783 - all 3 fails stackviz issue. line currently running no test yet
(rlandy) only missing fs035 - you can decide to skip and promo if no new failures

4 MASTER

buildset https://review.rdoproject.org/zuul/buildset/0771b066f0364ba18ad462f5883ba7eb 20/35/39 (2x tempest 1x mirror)
chasing with https://review.rdoproject.org/r/c/testproject/+/43286
dig @ http://promoter.rdoproject.org/promoter_logs/centos9_master.log another candidate from 29th https://trunk.rdoproject.org/centos9-master/tripleo-ci-testing/41/f9/ chasing with https://review.rdoproject.org/r/c/testproject/+/43285
(rlandy) https://review.rdoproject.org/r/c/testproject/+/43286 failed twice - with what looks like a legit failure - same test neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest. BUT … the in the next hash fs020 passes (failed fs039 and fs064). So skipping seems not needed - we have a choice to ignore that failure and promo or try chase later hash

5 VICTORIA

https://review.rdoproject.org/zuul/buildset/237623aa98514f9a924981868450b10b
new bug standalone/undercloud upgrade package conflict ? https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-undercloud-upgrade-victoria/87354dd/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz only seen once
posted chaser https://review.rdoproject.org/r/c/testproject/+/43296

new/transient/no bug yet

possible breakages in ceph jobs until these are all merged https://review.opendev.org/q/topic:ceph_promotion_pipeline

Monday 30 May

Bugs:

https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
https://bugzilla.redhat.com/show_bug.cgi?id=2089304 fs020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle (edit)
https://bugs.launchpad.net/tripleo/+bug/1973568 Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
~~https://bugzilla.redhat.com/show_bug.cgi?id=2091502~~ ERROR: Cannot install stackviz because these package versions have conflicting dependencies
- downstream testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/315285 [DNM] Test rhos16.2 failing jobs
- upstream bug: ~~https://bugs.launchpad.net/tripleo/+bug/1976247~~ wallaby gate blocker tripleo-ci-centos-8-standalone ERROR: Cannot install stackviz
~~https://bugs.launchpad.net/tripleo/+bug/1976251~~ [CI] tox-ansible-test-sanity doesn't take the "ignore" anymore
~~https://bugs.launchpad.net/tripleo/+bug/1975917~~ AttributeError: 'Service' object has no attribute 'enabled'

~~https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/411213~~

<bhagyashris> slaweq, hey can you help us to merge this one  https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/411213
<bhagyashris> we are blocked due to this ^
<slaweq> bhagyashris: sure, looking
<slaweq> bhagyashris: done
<bhagyashris> slaweq, thanks

regarding the curl error:

pinged migarcia

<bhagyashris> migarcia, hey
 <bhagyashris> around?
 <migarcia> bhagyashris: I am, what's up?
 <bhagyashris> we are facing one issue build push upload image 
 <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/1c70911/logs/container-builds/d200f2a7-c38e-49ad-b9ef-e83cabfa5fc0/base/base-build.log
 <bhagyashris> lon is out today 
 <bhagyashris> issue is: Errors during downloading metadata for repository 'osptrunk-candidate-deps':
 <bhagyashris>   - Curl error (28): Timeout was reached for http://download.eng.bos.redhat.com/brewroot/repos/rhos-17.0-rhel-9-trunk-candidate/latest/x86_64/repodata/f6d120c5ebe86676cd598a6c179f7bef99e4aa3fc54f9291e27708b502d1f7fc-primary.xml.gz
 <migarcia> bhagyashris: I would rekick, could be a network blip or that the repo was regenerated while the job was running
 <migarcia> rhos-17.0-rhel-9-trunk-candidate/latest symlink gets updated regularly as new builds are tagged in
 <bhagyashris> migarcia, ack thanks ! will check the result in the rekick 
 <migarcia> cool, let me know
 <bhagyashris> migarcia, thanks 
 <bhagyashris> let me know once you re kicked 
 <bhagyashris> migarcia, hey you are rekicking or should i rekicked 
 <migarcia> bhagyashris: please do
 <bhagyashris> migarcia, ack
 <bhagyashris> migarcia, hey we are still with same issue on the recent run 
 <bhagyashris>  https://sf.hosted.upshift.rdu2.redhat.com/logs/94/947e8a93a865e16481d14a1dd9fe1f91216e1a8d/openstack-periodic-integration-rhos-17-rhel9/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/45338ea/logs/container-builds/a3e81dbf-42ba-42e9-bf72-d5aeb0e65b4f/base/base-build.log
 <migarcia> bhagyashris: huh, I can download that file just fine.
 <migarcia> and it looks like the job was also downloading it fine, but very slow for some reason
 <migarcia> osptrunk-candidate-deps                         506  B/s |  64 kB     02:10
 <ysandeep> bhagyashris: could you hold a node and check mtu on cni-podman bridge

 <ysandeep> bhagyashris, sounds similiar to https://bugzilla.redhat.com/show_bug.cgi?id=2060932 

 <bhagyashris> ysandeep, let me hit the testproject patch

Update: recently podman version updated from 2:4.0.2-6.el9_0 to 2:4.0.2-7.el9_0 which basically includes the new dependencies "netavark" and that creates the "podman0" bridget from here https://github.com/containers/netavark/blob/02e031fdd9f7cd849c4fdd18cdd1ecb1a135485f/src/test/config/setupopts2.test.json#L14-L22 and takes mtu value as 1500 which basically take more time to download metadata for repository 'osptrunk-candidate-deps' and get timed and failed. Will debug more tomorrow on that and will file bug if required

new/transient/no bug yet

c9 broken container build https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-build-containers-centos-9-push-master/a0d5760/logs/build.log
- not a blocker because https://github.com/rdo-infra/rdo-jobs/blob/c72a5465ae3b5425bb983f8a1744af8b3e839c34/zuul.d/integration-pipeline-main.yaml#L10 we are pushing to quay now that job is OK

Friday 27 May

Bugs:

https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
https://bugs.launchpad.net/tripleo/+bug/1973568 Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
https://bugzilla.redhat.com/show_bug.cgi?id=2089304 fs020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle (edit)
https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block
https://bugs.launchpad.net/tripleo/+bug/1975917 AttributeError: 'Service' object has no attribute 'enabled'
downstream notes:
- TO Do's:
  - @rlandy didn't get any reply so far from lon so will need to check with lon and rekicked the line for ovn/ovs issue downstream: https://trello.com/c/GZtIK9Tb/2539-cixbz2089724rhos-17rhel-9-tripleonodesvalidation-failing-with-packet-loss-in-the-network-availability-validation-block
    - Rekicked - new base container issue - was failing on Friday due to mirror issue
    - Same issues today (Sunday) … pls see log from https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-17/1c70911/logs/container-builds/d200f2a7-c38e-49ad-b9ef-e83cabfa5fc0/base/base-build.log. USA (lon/jon) will be out … pls ask migarcia id he can fix: the curl error
  - for 16.2 https://trello.com/c/10kvMPrS/2529-cixbz2089304osp17osp162rhel8rhel9fs020-and-full-tempest-scenario-job-failing-on-tempest-test-neutrontempestpluginscenariotesttru / https://bugzilla.redhat.com/show_bug.cgi?id=2089304 waiting for patch https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/411213 merged and get it promoted till integration
- rhos16.2 on rhel8
  - (from rlandy) https://code.engineering.redhat.com/gerrit/c/networking-ovn/+/411213 should be mergeable - pls ask Yatin/Rodolfo to merge … then will need to be promoted up network component
  - Integration line: standalone and scenarios jobs failing with below error
    - log: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-rhos-16.2/fbb5d0f/job-output.txt
    - error: ERROR! Error when getting the collection info for ansible.posix from default (https://galaxy.ansible.com/api/) (HTTP Code: 530, Message: Code: Unknown)
    - 530 Site is frozen re-running the jobs here https://code.engineering.redhat.com/gerrit/c/testproject/+/315285
      - maximum node node_failure on testproject
- rhos17 on rhel9
  - node_failure on most of the jobs in the current run:
    - pinged on rhos-ops - no reply yet

<bhagyashris> #rhos-ops Hi, we are faing the node_failure on rhos17-rhel9
<bhagyashris> facing*
<bhagyashris> currently running integration line - rhos17 on rhel9
<bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status
<bhjf> Title: Zuul (at sf.hosted.upshift.rdu2.redhat.com)
<bhagyashris> psedlak|ruck, ^

<bhagyashris> dpawlik, ^
<bhagyashris> facing node failure on downstream 

<dpawlik> kforde: hey, all is fine with the infra?
<dpawlik> kforde: ah, just horizon does not work. Was thinking that something happend with one vm

<dpawlik> bhagyashris: can I deque your job and recheck?
<bhagyashris> yeah

<bhagyashris> dpawlik, yeah 

<dpawlik> bhagyashris: "Global Service Outage Ongoing: RDU2 DC Impact"
<dpawlik> it can be related
<bhagyashris> dpawlik, ack
<dpawlik> we got network flappings between services like zookeeper, DNS does not work...

Thursday 26

(previous ruck|rover pad: https://hackmd.io/uiv6iiN5QR-Z3mfFyKWeqA)

https://bugs.launchpad.net/tripleo/+bug/1973223 Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest
https://bugs.launchpad.net/tripleo/+bug/1973568 Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
https://bugs.launchpad.net/tripleo/+bug/1972163 cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
https://bugs.launchpad.net/bugs/1971465 fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
https://bugzilla.redhat.com/show_bug.cgi?id=2089304 fs020 and full-tempest-scenario job failing on tempest test neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle (edit)
https://bugzilla.redhat.com/show_bug.cgi?id=2089724 tripleo_nodes_validation failing with packet loss in the Network availability validation block
~~https://bugs.launchpad.net/tripleo/+bug/1975671~~ Component lines jobs and RDO Third party check jobs are failing because master containers are not available.

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover - 27th May 2022 - 02nd June 2022

tags: ruck_rover

Previous RR notes: https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ?view

Thursday 02 June

new/transient/no bug yet

Bugs:

Wednesday 01 June (and earlier ongoing things tracked here)

Bugs:

STOP (all tracked bugs duplicated above stop scrolling) STOP

STOP (all tracked bugs duplicated above stop scrolling) STOP

Tuesday 31 May

Bugs:

Promotions upstream handoff:

new/transient/no bug yet

Monday 30 May

Bugs:

new/transient/no bug yet

Friday 27 May

Bugs:

Thursday 26

tags: `ruck_rover`