Ruck Rover 2022-11-25 - 2022-12-01

RDO Cockpit / RHOS Cockpit RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

• opendev.org:openstack
• rdoproject.org:rdoproject.org
• redhat.com:tripleo-ci-internal

Review List

tags: ruck_rover

Previous RR notes: https://hackmd.io/Vn_zjslARMG5Crs4YwnSqw

ruck & rover: pooja & ananya

Dec 01

Gate

No particular job is red but I see some Retry limits (TODO monitor)

New bugs

https://bugzilla.redhat.com/show_bug.cgi?id=2149963

Upstream promotion pipelines

• Master: 2022-11-28
  • https://bugs.launchpad.net/tripleo/+bug/1998227
    • tp: https://review.rdoproject.org/r/c/testproject/+/46302
• Zed: 2022-11-25
  • fs64:
    • https://bugs.launchpad.net/tripleo/+bug/1998211
    • tp: https://review.rdoproject.org/r/c/testproject/+/36254
• Wallaby c9: 30 nov
  • tp: TODO
• Wallaby c8: 30 nov
  • tp: TODO
• Train c8: 30 nov
  • tp: TODO

Upstream component pipelines

• c9:
  • fs001s : https://bugs.launchpad.net/tripleo/+bug/1998227
  • standalones: https://bugs.launchpad.net/tripleo/+bug/1998348 (this cleared on its own, will close by eod)
• c8:
  • tp: TODO

Downstream promotion pipelines

• 17/9 - 30th Nov (Promoted yesterday) 01st Dec (promoted today)
• 17.1/9 - 29th Nov (Blocked on jenkins jobs)
  • chasing promotions https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/55
  • reported a bug for jenkins job failure https://bugzilla.redhat.com/show_bug.cgi?id=2149963
• 17.1/8 - 30th Nov (Promoted yesterday)
• 16.2/8 - 30th Nov (Promoted yesterday) No new hash

Downstream component pipelines

• 17/9 - All green except network component
  • Rerun https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/5
• 17.1/9 - All green except network component
  • Rerun https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/5
• 17.1/8 - No jobs in criteria
• 16.2/8 - All green

Nov 30

Gate

​​​​No blockers

New bugs

Upstream promotion pipelines

• Master: 2022-11-28
  • https://bugs.launchpad.net/tripleo/+bug/1998227
    • tp: https://review.rdoproject.org/r/c/testproject/+/46302
• Zed: 2022-11-25
  • fs64:
    • https://bugs.launchpad.net/tripleo/+bug/1998211
• Wallaby c9: 2022-11-28 PROMOTED TODAY!!
  • https://review.rdoproject.org/r/c/testproject/+/46256
• c8 (wallaby 2022-11-23) (train 2022-11-24) : PROMOTED TODAY!! https://review.rdoproject.org/r/c/testproject/+/46301

Upstream component pipelines

• Master, Zed:
  • https://bugs.launchpad.net/tripleo/+bug/1998348
  • https://bugs.launchpad.net/tripleo/+bug/1998227
• Wallaby c9
  • https://bugs.launchpad.net/tripleo/+bug/1998348

Downstream promotion pipelines

• 17/9 - 30th Nov
  • rerunning sc12 standalone https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/54
• 17.1/9 - 29th Nov
• 17.1/8 - 29th Nov
• 16.2/8 - 28th Nov
  • rerunning tempest api https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/54

Downstream component pipelines

• 17/9 -
  • Security and network component
  • Rerunning https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/3
• 17.1/9 -
  • network component
  • Rerunning https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/3
• 17.1/8 - No jobs in criteria
• 16.2/8 - All green

Nov 29

Gate

New bugs

Upstream promotion pipelines

• Master:

  • https://bugs.launchpad.net/tripleo/+bug/1998227

• Zed:

  • https://review.rdoproject.org/r/c/testproject/+/46287
  • fs64:
    • https://bugs.launchpad.net/tripleo/+bug/1998211

• Wallaby c9:

  • https://review.rdoproject.org/r/c/testproject/+/46256

Upstream Component pipelines

• https://review.rdoproject.org/r/c/testproject/+/46292
• https://review.rdoproject.org/r/c/testproject/+/46293

• Master
  • validation
    • https://review.rdoproject.org/r/c/testproject/+/46282
    • https://review.rdoproject.org/r/c/testproject/+/33582
• Zed
  • validation
    • https://review.rdoproject.org/r/c/testproject/+/46283
  • security
    • fs39 failed with random failures
  • network
    • fs01 failed with random failures.
• Wallaby C9
  • common
    • standalone blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
• Wallaby C8
  • tripleo
    • standalone job blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
  • tempest
    • standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
  • security
    • standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
  • network
    • standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
  • https://review.rdoproject.org/r/c/testproject/+/40897
• Train C8
  • tripleo
    • fs01 blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
  • common
    • sc04 standalone blocked on https://bugs.launchpad.net/tripleo/+bug/1997939

Downstream promotion pipelines

• 17/9 - 28th Nov Line currently running
• 17.1/9 - 28th Nov fs35 and sc01 standalone failed
  • Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/52
• 17.1/8 - 28th Nov All green
• 16.2/8 - 28th Nov All green

Downstream component promotion pipelines

• 17/9 - All green except sc04 common component job
  • Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/2
• 17.1/9 - All green except tempest api network component job
  • Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/2
• 17.1/8 - No job in criteria
• 16.2/8 - All green

Nov 28

Gate:

New bugs:

• https://bugs.launchpad.net/tripleo/+bug/1997939 (Affects wallaby and train c8)
  • tp: https://review.rdoproject.org/r/c/testproject/+/46272

Upstream promotion pipelines

• Master: 28th
  • https://review.rdoproject.org/r/c/testproject/+/46253
• Zed: 25th:
  • https://review.rdoproject.org/r/c/testproject/+/46242
  • featureset064:
    • Could not create '/home/zuul/overcloud-hardened-uefi-full.raw': Permission denied
    • Overcloud dceploy failed
• Wallaby c9: 27th
  • https://review.rdoproject.org/r/c/testproject/+/46256
• Wallaby c8: RED
  • https://bugs.launchpad.net/tripleo/+bug/1997939
• Train: RED
  • https://bugs.launchpad.net/tripleo/+bug/1997939

Upstream Component pipelines

• Master
  • tripleo (tempest tests failed)
    • https://review.rdoproject.org/r/c/testproject/+/46257
  • validation
    • https://bugs.launchpad.net/tripleo/+bug/1998098
• Zed
  • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-zed
    • deploy-steps-tasks-step-0.yaml times out (TODO bug)
    • https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-zed/0953c50/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
• Wallaby c9:
  • https://review.rdoproject.org/r/c/testproject/+/46269
• Walllaby c8:
  • tempest, tripleo, network - standalone job failing with https://bugs.launchpad.net/tripleo/+bug/1997939
• Train c8:
  • common -
    • standalone - https://bugs.launchpad.net/tripleo/+bug/1997939
  • tripleo
    • fs001 - https://bugs.launchpad.net/tripleo/+bug/1997939
    • sc04 standalone - https://bugs.launchpad.net/tripleo/+bug/1997939

Downstream promotion pipelines

• 17/9 - 28th Nov fs35 by skipping https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/865797
• 17.1/9 - 28th Nov fs35 by skipping https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/865797
• 17.1/8 - 26th Nov
• 16.2/8 - 28th Nov(Today) - tempest-scenario and sc01 standalone failing with randaom tempest test.
  • Rerun (got sucess) https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/50

Downstream component promotion pipelines

• 17/9 - All green except tripleo component jenkins job
• 17.1/9 - All green
• 17.1/8 - No job in criteria
• 16.2/8 - All green exceot tripleo component jenkins job

Nov 25

Gate:

Rechecked on https://review.opendev.org/c/openstack/heat/+/802139/ (failed once)

New bugs:

• https://bugzilla.redhat.com/show_bug.cgi?id=2148354 periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 failing consistently with set of keystone_tempest_plugin tests with error "Unexpected content type provided"
• https://bugs.launchpad.net/tripleo/+bug/1997939 tripleo-ci-centos-8-standalone in stable/wallaby consistently fails because of the failure during creating galera resource

Upstream promotion pipelines

• Master -

  • https://review.rdoproject.org/r/c/testproject/+/46241

• Zed - promoted

  • https://review.rdoproject.org/r/c/testproject/+/46242

• C9 Wallaby -

  • running

• C8 wallaby -

  • https://review.rdoproject.org/r/c/testproject/+/46243 (rerunning the olds registry push job till Amol's patches merge)

• C8 Train -

Upstream Component pipelines

• Master

  • clients
    • fs001 ovb (random tempest tests failing)
  • network
    • fs001 ovb (random tempest tests failing)
  • Reruns (https://review.rdoproject.org/r/c/testproject/+/33582)

• Zed

  • Reruns (https://review.rdoproject.org/r/c/testproject/+/36719)

• C9 wallaby

  • tempest - random tests failing

• C8 wallaby -

  • Reruns (https://review.rdoproject.org/r/c/testproject/+/40897)

• C8 train -

  • Reruns (https://review.rdoproject.org/r/c/testproject/+/36478)

Downstream promotion pipelines

• 17/9 - 22nd Nov (Line running)
• 17.1/9 - 22nd Nov - fs35 failing with tempest tests
• 17.1/8 - 22nd Nov (All green)
• 16.2/8 - 23rd Nov
  • fs35 failed once [Rerun]https://code.engineering.redhat.com/gerrit/c/testproject/+/209874 (got a green run https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/ed489e014c584d8d833e9d070132ed6a)

Downstream component promotion pipelines

• 17/9 - All green
• 17.1/9 - network and tripleo [Rerun] https://code.engineering.redhat.com/gerrit/c/testproject/+/436232 (got a green run https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/1#message-c943b28d5be1c620a8053a5ef8142341f5ea8c40)
• 17.1/8 - No job in criteria
• 16.2/8 - All green