Ruck Rover 2022-07-15 to 2022-07-21

tags: ruck_rover

Previous RR notes: https://hackmd.io/hD7Q6AsZQQyki_v4BoMfxg

Cockpit

Downstream cockpit

OpenStack Program Meeting 2022

2022-07-22 - handover

All skip jobs patches are reverted.

CIX needs to be triaged:

https://trello.com/c/9Lc7Z8Xw/2634-cixbz2109495osp162rhel8hosts-unreachable-during-gathering-facts-leading-to-ovb-job-failure

• Train network and tripleo components are lagging
  • May be it will clean it up
• then promote rhos-16.2

Tracker bugs:

• https://trello.com/c/iEvKnb0H/2632-cixlp1982464tripleociproa-ipa-job-mirror-issue-tracker-for-master-and-wallaby
• https://trello.com/c/8cGNSfh8/2621-cixlp1981478tripleociproa-cs8-9-ovb-fs001-fs035-master-wallaby-train-failing-with-db-connection-errors

CS9 Master – last_promotion=2022-07-21 16:31:19

To watch network and security

• components: https://review.rdoproject.org/r/c/testproject/+/42374

Bugs

• https://bugs.launchpad.net/bugs/1982137

  • https://trello.com/c/qP0WWS6r/2629-cixlp1982137tripleociproa-error-grub-core-disk-diskfilterc916diskfilter-writes-are-not-supported

• https://bugs.launchpad.net/bugs/1981829

  • https://trello.com/c/7Ld4uihR/2627-cixlp1981829tripleociproa-octavia-tempest-tests-are-failing-with-policy-does-not-allow-this-request-to-be-performed-error
  • revert this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626

• https://bugs.launchpad.net/bugs/1982195

  • https://trello.com/c/7xp7x4hZ/2630-cixlp1982195tripleociproa-cic9master-python-sqlalchemy-addition-in-cs9-base-broke-container-barbicanapidbsync
  • verify promotions: clients and security
    • https://review.opendev.org/c/openstack/oslo.db/+/850550 - clients component promoted
    • https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 - security component promoted
    • the wallaby promotion
    • then revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442

CS9 Wallaby – last_promotion=2022-07-21 13:15:57

• components: https://review.rdoproject.org/r/c/testproject/+/44058

Bugs

• https://bugs.launchpad.net/bugs/1982137
  • https://trello.com/c/qP0WWS6r/2629-cixlp1982137tripleociproa-error-grub-core-disk-diskfilterc916diskfilter-writes-are-not-supported
• https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft
  • https://review.opendev.org/c/openstack/tempest/+/849562
  • need to merge this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850155
  • TODO: Need to test tempest fix on wallaby

CS8 Wallaby – last_promotion=2022-07-19 16:56:09

• components: https://review.rdoproject.org/r/c/testproject/+/41367

CS8 Train – last_promotion=2022-07-19 01:01:47

• components: https://review.rdoproject.org/r/c/testproject/+/44056
• components: https://code.engineering.redhat.com/gerrit/c/testproject/+/309187

RH8 OSP16.2 – last_promotion=2022-07-18 22:05:30

• promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

RH8 OSP17 – last_promotion=2022-07-18 11:04:22

rlandy is working on promoting that.

• promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/309187

RH9 OSP17 – ast_promotion=2022-07-21 02:10:54

rlandy is working on this line with ceph-5.2

• promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/408119

2022-07-21

CS9 Master

Pushing forward promotion for: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087

chasing recent hash: https://review.rdoproject.org/r/c/testproject/+/42319

component: https://review.rdoproject.org/r/c/testproject/+/42374 Chasing hash: fcf18e1661b4add7eb1b3b2c761e1c63 Skip jobs: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 - those jobs passed tempests in different executions. We're only missing fs020. Attempt to run it on PSI: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956

All criteria patches(skipped jobs) are reverted.

Gate blocker for master & wallaby

https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-ovn-provider-standalone https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-standalone Reusing same bug: https://bugs.launchpad.net/tripleo/+bug/1981829

• moving it to skip list - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626

CS9 Wallaby – last_promotion=2022-07-19 11:52:25

components: https://review.rdoproject.org/r/c/testproject/+/41367 re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44115 https://bugs.launchpad.net/tripleo/+bug/1982464 - https://bugs.launchpad.net/tripleo/+bug/1982464

CS9 master – last_promotion=2022-07-21 02:55:20

re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44080

CS8 Wallaby – last_promotion=2022-07-19 16:56:09

components: https://review.rdoproject.org/r/c/testproject/+/44056

RH8 RHOS16

• https://bugzilla.redhat.com/show_bug.cgi?id=2109495

2022-07-20

Current issues

• CS9 Master and Stable/Wallaby Check bug: https://bugs.launchpad.net/tripleo/+bug/1982195
• tripleo-ci-centos-9-standalone is failing on stable/wallaby - tempest scenario - failed to reach ACTIVE status and task state "None" within the required time https://bugs.launchpad.net/tripleo/+bug/1982227
• Workaround: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850428 Downgrade python3-sqlalchemy and libknet1.x86_64 (disclaimer: idk if we need both downgrades - but we can remove them one at a time and see. Both clears the gate)
  • https://review.opendev.org/q/topic:bug%252F1982227
    • https://zuul.opendev.org/t/openstack/status#850442 sqlalchemy downgrade seems to be fix the issue
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442

CS9 Master – lagging behind promotion

• components: https://review.rdoproject.org/r/c/testproject/+/42319
• jobs: https://review.rdoproject.org/r/c/testproject/+/44037
• Re-running master job: https://review.rdoproject.org/r/c/testproject/+/44080

CS9 Wallaby – last_promotion=2022-07-19 11:52:25

components: https://review.rdoproject.org/r/c/testproject/+/41367

CS8 Wallaby – last_promotion=2022-07-19 16:56:09

components: https://review.rdoproject.org/r/c/testproject/+/44056

CS8 Train – last_promotion=2022-07-19 16:56:09

components: https://review.rdoproject.org/r/c/testproject/+/42374

RH8 OSP16.2 – last_promotion=2022-07-18 22:05:30

• https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

RH8 OSP17 – last_promotion=2022-07-18 11:04:22

RH9 OSP17 – lagging behind promotion - fs035

• bugfix: https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/420543 - merged
• testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/420692 to promote tripleo component and kick RHOS-17 RHEL-9 line

2022-07-19

Current Issues

Important: SC010 OVN is failing for both CS9 master & wallaby SC010 RHEL 8 RHOS 17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-17&skip=0 https://bugs.launchpad.net/tripleo/+bug/1981829

CS9 master

status=Yellow last_promotion=2022-07-12 13:54:22 Re-Running failed job: https://review.rdoproject.org/r/c/testproject/+/44037

• https://bugs.launchpad.net/tripleo/+bug/1982137 - ovb node provisioning issue

Check failing on a bug: https://bugs.launchpad.net/tripleo/+bug/1982195

Bugs

• https://bugzilla.redhat.com/show_bug.cgi?id=2103118
  • Debug env: https://code.engineering.redhat.com/gerrit/c/testproject/+/420316 gthiemonge is looking into that
• https://bugs.launchpad.net/tripleo/+bug/1982137

Components

• https://review.rdoproject.org/r/c/testproject/+/42319

Tripleo promotion

Pushing forward tripleo promotion to get latest changes in.

• https://review.rdoproject.org/r/c/testproject/+/41367

CS9 wallaby!!!

status=Red last_promotion=2022-07-19 11:52:25 \o/ promoted today

Bugs

• https://bugs.launchpad.net/tripleo/+bug/1981829

CS8 wallaby

status=Red last_promotion=2022-07-15 17:10:42

• https://review.rdoproject.org/r/c/testproject/+/44056

CS8 train!!!

status=Green last_promotion=2022-07-19 01:01:47 \o/

RHEL 8 RHOS 16.2

status=Red last_promotion=2022-07-18 04:03:34

• https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

RHEL 8 RHOS 17!!!

status=Green last_promotion=2022-07-18 11:04:22 \o/

RHEL 9 RHOS 17

status=Red last_promotion=2022-07-14 10:54:02

• https://code.engineering.redhat.com/gerrit/c/testproject/+/408119
• FS035 - https://bugzilla.redhat.com/show_bug.cgi?id=2108580

STOP! All updates

18th Jul, 2022

Important: SC010 OVN is failing for both CS9 master & wallaby SC010 RHEL 8 RHOS 17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-17&skip=0 https://bugs.launchpad.net/tripleo/+bug/1981829

CS9 master

Last promotion: 12th July Bugs:

• https://bugzilla.redhat.com/show_bug.cgi?id=2103118
  • Debug env: https://code.engineering.redhat.com/gerrit/c/testproject/+/420316 gthiemonge is looking into that
• https://bugs.launchpad.net/tripleo/+bug/1981829
  • Moving test to skip list: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850137 (Not yet merged)
  • Proper fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850118
  • https://review.opendev.org/c/openstack/tripleo-ci/+/850225
    • TP: https://review.rdoproject.org/r/c/testproject/+/44038
• Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44037

Patches to Clear OVN issue for Master and wallaby

• master: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/776942
• Wallaby: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/850033
• We need to fix standalone jobs in tripleo-ci not in tripleo-quickstart (https://review.opendev.org/c/openstack/tripleo-quickstart/+/850118)
• https://review.opendev.org/c/openstack/tripleo-ci/+/850225

Once ovn gets cleared for master, fs035 is already busted with different tempest tests, fs01 is also failing at different issue but fs020 is passong may be we can remove fs01/fs035 from criteria and promote master?

CS9 Wallaby

Last promotion: 12th July

• Rabbitq bump coming: https://review.rdoproject.org/r/c/rdoinfo/+/44025
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439
• Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/41367

CS8 train

Last promotion: 15th July Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44045

RH9 RHOS-17

Last promotion: 14th July Re-running failed jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/420204

RH8 RHOS-17

Last promotion: 18th july

RH8 RHOS-16.2

Last promotion: 18th July Re-running failed jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

15th Jul, 2022

Upstream Gate Blocker

• https://bugs.launchpad.net/tripleo/+bug/1981634 needs to be marked duplicate in favor of https://bugs.launchpad.net/tripleo/+bug/1981467

CS9 master

Last promotion: 12th July Re-running soniya tp: https://review.rdoproject.org/r/c/testproject/+/42692 sc10 ovn blocker: https://bugs.launchpad.net/tripleo/+bug/1981829 fs035 - https://paste.centos.org/view/raw/09698f75

• trying with one tempest test: https://review.rdoproject.org/r/c/testproject/+/44020

• started fresh patch: https://review.rdoproject.org/r/c/testproject/+/42374
• downstream for kvm: https://code.engineering.redhat.com/gerrit/c/testproject/+/408119
• disabled kvm for promo: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44029

CS9 Wallaby

Last promotion: 12th July

• https://review.rdoproject.org/r/c/testproject/+/41367
• https://code.engineering.redhat.com/gerrit/c/testproject/+/309187
• components: https://review.rdoproject.org/r/c/testproject/+/42319
  • sc10 ovn blocker: https://bugs.launchpad.net/tripleo/+bug/1981829

CS8 train

Last promotion: 15th July

Re-running failed job: https://review.rdoproject.org/r/c/testproject/+/44018

CS8 Wallaby:

last_promotion=2022-07-15 17:10:42

• Revert of promotion skip: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43689

RH8 RHOS-16.2

Last promotion: 12th july * Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/420016

RH8 RHOS-17

Last promotion: 13th july * Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956

• components: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027

RH9 RHOS-17

Last promotion: 14th July

• components: https://code.engineering.redhat.com/gerrit/c/testproject/+/407601