Ruck Rover - 13th May 2022 - 19th May 2022

tags: ruck_rover

Previous RR notes: https://hackmd.io/zjCkCnVeQ06bnBq836iF8g

Cockpit

Downstream cockpit

Thursday 19 May

rlandy to check below failures:-

periodic-tripleo-ci-centos-9-scenario004-standalone-master	TIMED_OUT 3h 09m 09s
periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master	FAILURE 2h 28m 48s
periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master 

• periodic-tripleo-ci-centos-9-scenario004-standalone-master openstack/tripleo-ci SUCCESS in last two runs
• periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master openstack/tripleo-ci master openstack-periodic-integration-main 2 hrs 16 mins 44 secs 2022-05-19 09:25:35 SUCCESS
• periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master - failed - will rerun
• reopened https://bugs.launchpad.net/tripleo/+bug/1973223 for scenario010 kvm - job now passed in rerun

Check/gate

• https://review.opendev.org/c/openstack/tripleo-ci/+/835460/

Master integration line - last_promotion=2022-05-16

Rerunning failed jobs: https://review.rdoproject.org/r/c/testproject/+/39357 https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/42965

New bug:

Master overcloud deployment failing with 'Node' object has no attribute 'uuid'

• reverting patch that caused this issue: https://review.opendev.org/c/openstack/tripleo-ansible/+/842399 (-W)
• Holding above for proper fix:
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/842473
• https://bugs.launchpad.net/tripleo/+bug/1974244 openstack-tox-py36 is failing on tripleo-heat-templates master tests - oslo-log (possible mirror issue)

Master Component line

• tripleo (last_promotion=2022-05-16)- ovb fs001 failed - new failure - rerunning to confirm if failure is consitent
• security (last_promotion=2022-05-13)- ovb fs039 failed - failing with known bug - should be resolved now
• network (last_promotion=2022-05-18)- ovb fs001 failed - tempest failure

Rerunning failed jobs here

Wallaby c9 integration line - last_promotion=2022-05-17

• Only fs001 - failing - need to check

Rerunning here https://review.rdoproject.org/r/c/testproject/+/26273

wallaby c9 component line - tripleo

• tripleo(last_promotion=2022-05-18) - container-multinode failing - tempest failure - rechecking to confirm if conistent
• https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=c736812dd676ac4f0b15f70f9bc668cb Promotable - internal fs001 in tempest now - rest passed in rerun

Rerunning failed jobs here

Wallaby c8 integration line - last_promotion=2022-05-16

• fs001/fs035 - failing - need to check
• promoted

wallaby c8 component line - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/26273

Victoria integration line - last_promotion=2022-05-16

• Rerun 1
• Rerun 2+3

Bunch of jobs failed:

• periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria fails on different tempest errors all the time, sometimes it fails on LP bug #1960310

• periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria fails on different tempest and non-tempest errors all the time

• periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria dito

• periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria dito

• Jobs keep failing because of MySQL errors in various/random services, example

• Job periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria might be failing because of an real issue:

  ​​2022-05-18T05:59:27.276336835+00:00 stderr F 2022-05-18T05:59:27Z|00021|patch|WARN|Bridge 'br-tenant' not found for network 'tenant'
  

  log example 1, log example 2. But maybe it is also related to MySQL issues mentioned above because errors are showing up here as well: https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/986574c/logs/undercloud/var/log/containers/neutron/server.log.txt.gz

  Same for periodic-tripleo-ci-centos-8-standalone-upgrade-victoria.

• Jobs periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria and periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria failed twice with:

  ​​2022-05-21 08:44:18.734207 | primary | ERROR: Cannot install oslo.utils because these package versions have conflicting dependencies.
  ​​2022-05-21 08:44:18.734262 | primary |
  ​​2022-05-21 08:44:18.734277 | primary | The conflict is caused by:
  ​​2022-05-21 08:44:18.734291 | primary |     The user requested oslo.utils
  ​​2022-05-21 08:44:18.734303 | primary |     The user requested (constraint) oslo-utils===4.6.1
  ​​2022-05-21 08:44:18.734314 | primary |
  ​​2022-05-21 08:44:18.734324 | primary | To fix this you could try to:
  ​​2022-05-21 08:44:18.734335 | primary | 1. loosen the range of package versions you've specified
  ​​2022-05-21 08:44:18.734347 | primary | 2. remove package versions to allow pip attempt to solve the dependency conflict
  ​​2022-05-21 08:44:18.734361 | primary |
  ​​2022-05-21 08:44:18.734373 | primary | ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
  

  https://logserver.rdoproject.org/78/42978/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria/8539a90/job-output.txt

victoria c8 component line - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/26273

Train integration line - last_promotion=2022-05-16 - Green (No new hash when I checked this morning)

Train component line - Only tripleo lagging

• tripleo - passed in rerun

Analysis, 18th and 16th failed on same tempest test - 17th had different tempest failure - rerunning to confirm if issue is consistent

https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/319b011/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
https://logserver.rdoproject.org/73/26273/20/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/1de3186/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/58fce8a/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz

Rerunning failed jobs here

16.2 integration line - last_promotion==2022-05-16 - Green (No new hash when I checked this morning

16.2 component line - All green

17/8 integration line - last_promotion=2022-05-18

• fs035 failed in last run - rerunning here

17/8 component line - All green

17/9 integration line - last_promotion=2022-05-18 - All green

• fs035 failed in last run passed in rerun

17/9 component line - All green

Wednesday 18 May (and earlier ongoing things tracked here)

• Check/gate Master fs035 and fs064 failing on overcloud deploy on task "Render keystone-db-sync container definitions" with error - can only concatenate list (not "AnsibleUnicode") to list. Unexpected templating type error occurred on (image: "{{ tripleo_keystone_image }}"

Master integration line

• Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503

  • Need to revert tempest

• Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest

  • Latest run passed twice under observation

• https://bugs.launchpad.net/tripleo/+bug/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Under observation

Master Component line

• https://bugs.launchpad.net/tripleo/+bug/1973038 Master Node provisioning failing because nodes failing to boot and entering Emergency shell

  • We are waiting for latest diskimage-builder to promote till current-tripleo to fix component lines.

• https://bugs.launchpad.net/tripleo/+bug/1972163 Master cs9 standalone full tempest api failing for volume tempest tests and with error - 'Volume failed to detach from server'

Wallaby

https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

STOP (all tracked bugs duplicated above stop scrolling) STOP

Wednesday 18 May

Promotion status

Master integration line - last_promotion=2022-05-16

• New bug:

  • Master fs035 and fs064 failing on overcloud deploy on task "Render keystone-db-sync container definitions" with error - can only concatenate list (not "AnsibleUnicode") to list. Unexpected templating type error occurred on (image: "{{ tripleo_keystone_image }}"
    • Cedric proposed a patch.
      • https://review.opendev.org/c/openstack/tripleo-ansible/+/842303
      • rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537

• ovb jobs failed in rerun here

  • Need to investigate other failures

Master Component line - last_promotion=2022-05-16

Component which have failing jobs:-

• Baremetal
• Compute
• Network
• security
• tempest
• tripleo

Security and tripleo failing because of above known bug.

security failure looks wrong that mean we don't have proper isolication because we include delorean-current.

Wallaby c9 integration line - last_promotion=2022-05-17

• random tempest failures - rerunning jobs here

wallaby c9 component line

• tripleo - ovb failed - rerunning here

Wallaby c8 integration line - last_promotion=2022-05-16

• fs001/fs035 failed - need to rerun

wallaby c8 component line

• tripleo - ovb failed - rerunning here

Victoria integration line - last_promotion=2022-05-16

• Line running right now - will recheck in few hours

victoria c8 component line - All green

Train integration line - last_promotion=2022-05-16 - All green no new hash to promote

Train component line

• tripleo - ovb and sc04 - reruning here

16.2 integration line - 2022-05-16 - All green no new hash

17/8 integration line - last_promotion=2022-05-17

• full-tempest-api/fs001 failed, rerunning here

17/9 integration line - last_promotion=2022-05-17

• Line just triggered, will recheck in few hours

Tuesday 17 May

• Random tempest issue is back

  • Running wallaby/master on psi to see if we see same tempest failures.
    • https://code.engineering.redhat.com/gerrit/c/testproject/+/209874

• Master integration line:

  • fs001/020/35/39/64/ipa/container-updates failed, rerunning here
  • afternoon hash - periodic-tripleo-ci-build-containers-centos-9-quay-master failed - rerunning.

• Master component line - rerunning here

  • baremetal - ovb fs001 - passed in testproject rerun
  • security - ovb fs039
  • validation - container multinode - passed in testproject rerun

• C9 wallaby integration line - 2022-05-17 - All green

  • fs001 rbac failed - not in promotion criteria - do we need to report?

• c9 wallaby component line - rerunning here

  • tripleo

• C8 wallaby integration line - last_promotion=2022-05-16 - All green - No new hash when I checked this morning.

• c8 wallaby component line - All green

• Victoria integration line - last_promotion=2022-05-16 - All green - No new hash when I checked this morning.

• Victoria component line - ALl greeen

• Train integration line- last_promotion=2022-05-16 - All green - No new hash when I checked this morning.

• Train component line - All green

• OSP16.2 integration line - last_promotion=2022-05-16 - All green

• OSP16.2 Component line - All green

• OSP17/9 Integration line - In rerun

  • full-tempest/fs020/bm failed
    • testproject
      • https://code.engineering.redhat.com/gerrit/c/testproject/+/190672
      • https://code.engineering.redhat.com/gerrit/c/testproject/+/211643

• OSP17/9 Component line - All green

  • network - full-tempest-api - cleared in rerun
  • tripleo - ipa/container-multinode - cleared in rerun

• OSP17/8 Integration line - last_promotion=2022-05-17 - All green

  • full-tempest-scenario - cleared in rerun
  • fs001 - cleared in rerun

• OSP17/8 Component line - All green

  • tripleo
    • ipa - cleared in rerun
    • container-multinode - cleared in rerun

Monday May 16th

• Check/Gate
  • rechecked - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841284

Status

Master integration line - last promotion - 2022-05-16 - Green

• only 1 failing job - sc02 and blocking master promotion
• Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503
  • Added in skiplist:
    • https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/841834
  • reruning in testproject: https://review.rdoproject.org/r/c/testproject/+/39357
  • Fix: https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/42876
  • Promoting fix via component line: https://review.rdoproject.org/zuul/status#28537

Master component line - All green

Wallaby c9 integration line - 2022-05-15

• Ovb and full-tempest-api failing - need to dbeug

Wallaby c9 component line - only tripleo lagging

• tripleo

Wallaby c8 integration line - 2022-05-11

• fs01 and fs035 failing
• all fs passed independently today, promotion in progress.

Wallaby c8 component line - 2022-05-12

• last tripleo promotion on thursday passed after rerun
• last network promotion on friday lets try to reproduce this re-popping error
• Reruns

Victoria integration line - last_promotion=2022-05-16 - Green

• Promoted today

Victoria component line - baremetal

• baremetal - ovb - rlandy force promoted.

train integration line , last_promotion=2022-05-16 - All green

train component lines - All green

16.2 integration line, last_promotion=2022-05-13

• Bm job failed - rerunning here

16.2 component lines - All Green

17/rhel-9 integration line, last_promotion=2022-05-15 (Green)

• Scheduled run is running, lets see in few hours

17/rhel-9 component lines - All green

• common - passed in rerun

17/rhel-8 integration line - last_promotion=2022-05-16 (green)

17/rhel-8 component lines - All green

• tempest - passed in testproject
• tripleo - passed in testproject

Friday 13 May

Status:-

Check

• https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841284/ (Under observation)
  • Validation rpm build failed - invesitgate why we are build validation rpm in tripleo job
• https://review.opendev.org/c/openstack/tripleo-validations/+/841378/
  • https://0fc95984b662eca8c7af-61a732188f1643f755755f84f6310584.ssl.cf2.rackcdn.com/841378/1/gate/tripleo-ci-centos-8-content-provider/e4b1117/logs/delorean_logs/component/validation/cc/ce/ccce4761654f0bc8477a5c14f1adba8bcf85ac67_dev/index.html

Pinged jpdovin about that - We are rechecking https://review.opendev.org/c/openstack/tripleo-validations/+/841378 - to issue if issue is consistent.

Yatin found the issue, jiri is fixing it.

<ykarel> https://github.com/openstack/cliff/commit/584352dcd008d58c433136539b22a6ae9d6c45cc is in 2.18.0+
<ykarel> but in train we have 2.16.0
<ykarel> jpodivin, the issue is reproducable easily with tox
<ykarel> too
<ykarel> with cliff 2.16.0
<ykarel> also iirc it was said stable/1.6 branch was for train release, but still i see it's testing with master u-c
ykarel> shouldn't it be  using stable/train
<jpodivin> it should be
<ykarel> ok please fix then it should catch atleast error like this
<ykarel> https://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L285

Promotion

Master - 2022-05-06 - Out by 1 week - 2 known bugs (ovb instability and sc-10-kvm)

• Full tempest scenario failing - debug
• Rerunning failed jobs here

Master component line

• Components which are lagging - No known bugs - Rerunning jobs here:-
  • Network promoted via testproject
  • Security promoted via testproject
  • Validations
  • Tripleo promoted via testproject

Wallaby c9 - 2022-05-10 - Out by 3 days now - 2 known bugs(ovb instalbility and neutron tempest bug)

• Full-tempest-scenario - passed via testproject
• Rerunning the failing jobs here
• Need to discuss with Yatin and rodolfo - where are we with neutron bug.
  • Spoke with yatin this morning - Running a testproject with node hold and cleanup script disabled for debugging
    • https://review.rdoproject.org/r/c/testproject/+/36140/49/.zuul.yaml#9

Wallaby c9 components

• Components which are lagging
  • Network(08th may) - ovb failing - Rerunning here

Wallaby c8 - 2022-05-10 - 2 known bugs(ovb instalbility and neutron tempest bug)

• ovb fs001/035 failing
• Need to discuss with Yatin and rodolfo - where are we with neutron bug.
  • Spoke with yatin this morning - Running a testproject with node hold and cleanup script disabled for debugging
    • https://review.rdoproject.org/r/c/testproject/+/36140/49/.zuul.yaml#9

Wallaby c8 components

• Components which are lagging behind:-
  • baremetal - Red - 26th April - ovb failing - need to debug
    • https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/4b8b904/job-output.txt
    • Report and investigate
    • Rerun: https://review.rdoproject.org/r/c/testproject/+/42813 If it gets to tempest (instead of failing on redfish), then we will promote this line.
  • Promoted component

Victoria C8 - 2022-05-09

• Running victoria failed jobs here

That hash is out only fs001 and fs035 - can choose to promote it (7 days out)

• project: check: jobs: - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria: vars: force_periodic: true featureset_override: dlrn_hash_tag: 527fdcb7ab243130b6f360da7f56c4f9 - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria: vars: force_periodic: true featureset_override: dlrn_hash_tag: 527fdcb7ab243130b6f360da7f56c4f9

Victoria C8 components

• Components which are lagging
  • Baremetal - 30th April - rerunning here
  • Tripleo - promoted on 11th May - Can wait for tonight run
  • Have baremetal testproject in rerun
    • promoting with fs001 removed

Train C8 - 2022-05-09

• Line running right now - let's see in few hours
  • https://review.rdoproject.org/r/c/testproject/+/36254/98
  • fs35 on train timed out, full-tempest-api-train succeeded and sc1 failed
• Few jobs failed - need to investigate and report if consistent
• Train passed all jobs except fs035 (timed out - skipped to promote)
• Train promoted

Train C8 components - All green

16.2 - Promoted on 2022-05-13 - All Green

16.2 Components - All Green

17/9 - 2022-05-12 - Green

17/9 components - Rerunning failed jobs here

• Components which are lagging
  • tripleo
  • validations promoted via testproject

17/8 - Need to check yet

• containers build jobs passed but jobs are failing on ceph: fpantano rlandy: we merged a downstream patch yesterday https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/834352 is needed for quickstart fpantano https://code.engineering.redhat.com/gerrit/c/python-tripleoclient/+/406914 fpantano rlandy: fultonj the revert ^

New/Transient/No bug yet

Bugs