Ruck Rover 2022-09-23 to 2022-09-29

tags: ruck_rover

Next RR notes: TODO

Previous RR notes: https://hackmd.io/a92u74hDQhe5afE_LUtbRw

ruck: Chandan, rover: Ananya

RDO Cockpit / RHOS Cockpit

RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

• opendev.org:openstack
• rdoproject.org:rdoproject.org
• redhat.com:tripleo-ci-internal

Handover Notes

Promotion Status: CS9 master - 2022-09-29 CS9 wallaby - 2022-09-28 CS8 wallaby - 2022-09-28 CS8 Train - 2022-09-26

RHOS16-2 - 2022-09-23 RHOS17 RHEL-9 - 2022-09-29 RHOS17-1 RHEL-9 - 2022-09-26 RHOS17-1 RHEL-8 - 2022-09-24

Active Bugs:

Random and Inconsistent

• https://trello.com/c/fMby586x/2679-cixlp1987092tripleociproa-pacemaker-performance-causes-intermittent-galera-issues-in-loaded-ci-env

RHOS16-2

• https://trello.com/c/471dXQoC/2734-cixbz2130109osp162rhel8ceph-inventory-linking-fails-with-permission-issues
• https://trello.com/c/hcnDrBhO/2736-cixbz2130504osp162rhel8rhos-162-component-pipeline-jenkins-automatic-triggers-are-not-working

RHOS17-1

• https://trello.com/c/bF4diFmr/2726-cixftbfsosp-171-rhel-9

Wallaby

• https://trello.com/c/VMBKZlwg/2735-cixlp1991076tripleociproa-python-openstackclient-600-conflicting-dependencies
• https://trello.com/c/Uh4VX3Uk/2722-cixlp1990359tripleociproa-kernel-does-not-provide-mount-namespace-no-such-file-or-directory-edit
• https://bugs.launchpad.net/tripleo/+bug/1991240
• https://bugs.launchpad.net/tripleo/+bug/1991232

train

• https://trello.com/c/wMFW4qdv/2737-cixlp1991093tripleociproa-compute-and-network-tempest-tests-failing-on-fs35-train

Sep 29, 2022

• Upstream gate/check
  • train: https://bugs.launchpad.net/tripleo/+bug/1991232
    • skip: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859825
    • https://review.opendev.org/c/openstack/tripleo-ci/+/859830
    • https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859796 - revert
    • tp: https://review.rdoproject.org/r/c/testproject/+/45378
    • tp train: https://review.rdoproject.org/r/c/testproject/+/45380
  • wallaby:
    • ffu: https://bugs.launchpad.net/tripleo/+bug/1991240

Integration:

• master:
  • tp: https://review.rdoproject.org/r/c/testproject/+/45271
  • will promote
• wallaby c9:
  • 2022-09-28
  • running
• wallaby c8
  • 2022-09-28
• train
  • 2022-09-26
  • Running

Sep 28, 2022

• Upstream gate: Green
  • [CS8][Train] Exclude container-selinux-2:2.189.0 | https://review.opendev.org/c/openstack/tripleo-quickstart/+/859516
  • https://bugs.launchpad.net/tripleo/+bug/1991076

ovb issues

• Holding node for the galera/mysql issue

Integration lines

• master
  • last promotion: 2022-09-26
  • fs001: mysql/galera Node held
  • fs020: mysql/galera Node held
  • fs064: mysql/galera Node held
  • tp for one time failure: https://review.rdoproject.org/r/c/testproject/+/45271
• wallaby c9
  • last promotion: 2022-09-26
  • fs020: https://bugs.launchpad.net/tripleo/+bug/1990359
  • fs001: mysql/galera
  • tp for one time failures: https://review.rdoproject.org/r/c/testproject/+/45276
• wallaby c8
  • last promotion: 2022-09-26
• train
  • last promotion: 2022-09-26
  • fs035: https://bugs.launchpad.net/tripleo/+bug/1991093
    • tp to see if internal fs35 runs: https://code.engineering.redhat.com/gerrit/c/testproject/+/429821
    • if ^ passes switch 35 to inetrnal in promo criteria:

Component

• wallaby
  • validations:
    • fs001: https://bugs.launchpad.net/tripleo/+bug/1991076

Downstream

• OSP16.2 - 23/09/2022
  • https://bugs.launchpad.net/tripleo/+bug/1990988
    • https://review.opendev.org/q/topic:lp%25231990988
  • https://bugzilla.redhat.com/show_bug.cgi?id=2130504
• OSP 17.1 on RHEL-9 - 26/09/2022
  • FTBFS opened currently
• OSP 17.1 on RHEL-8 - 24/09/2022
• OSP 17.0 on RHEL-9 - 27/09/2022

Sep 27, 2022

• Upstream gate : Green

ovb failures

• fs39

• fs64

galera/sql issue: https://review.rdoproject.org/r/c/testproject/+/45279 - I am holding node here but this error is not consistent

Integration lines

• master:

  • Last promotion: 2022-09-26
  • fs002: https://bugs.launchpad.net/tripleo/+bug/1990556

• wallaby c9:

  • last promotion: 2022-09-26
  • fs20:

• train:

  • sc000: package conflick cintainer selinux
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45240

Components

• master

  • baremetal
    • fs001: https://bugs.launchpad.net/tripleo/+bug/1987092
  • security
    • fs39: https://bugs.launchpad.net/tripleo/+bug/1987092 Node is held
  • tp: https://review.rdoproject.org/r/c/testproject/+/45279

• wallaby c9

  • Green

• wallaby c8

  • Green

• train

  • Green

• RHEL-9 RHOS-17.1

  • promoted on 26th Sep, 2022

• RHEL-9 RHOS-17.0

  • promoted on 27th Sep, 2022

• RHEL-8 RHOS-16.2

  • promoted on 23rd Sept, 2022
  • Bug: https://bugzilla.redhat.com/2130109

Sep 26, 2022

• chasing downstream promotion

  • https://code.engineering.redhat.com/gerrit/c/testproject/+/427532

• Upstream Check/Gate

  • https://bugs.launchpad.net/tripleo/+bug/1990810 - affecting undercloud ffu job
    • Fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/859213

      Testing here: https://review.rdoproject.org/r/c/testproject/+/45269

• cs9-master

  • Last promotion: 2022-09-20
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45271
  • 35:
    • ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed No such file or directory - journal_errors.txt
  • Skipping 35 to promote as 35 internal passd on the same hash: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45275
  • Revert of skip: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44766

• cs9-wallaby

  • Last promotion: 2022-09-22
  • 35
  • 1
  • testproj: https://review.rdoproject.org/r/c/testproject/+/45276

• cs8-wallaby

  • Last promotion: 2022-09-23
  • Should promote today

• cs8-train

  • Last promotion: 2022-09-21
  • 35:
  • Skip 35 and promote as 35 internal passed for 3a74: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45275

• rhel-8-rhos-16.2

  • Last promotion: 2022-09-23

• rhel-8-rhos-17.1

  • Last promotion: 2022-09-24

• rhel-9-rhos-17.0

  • Last Promotion: 2022-09-24

• rhel-9-rhos-17.1

  • Last promotion: 2022-09-23

Components

• master:

  • tripleo
  • compute
  • tp: https://review.rdoproject.org/r/c/testproject/+/45279

• wallaby c9

  • network
  • tripleo
  • tp: https://review.rdoproject.org/r/c/testproject/+/45277

• wallaby c8

  • Green

• train

  • Green

RR Handoff

Integration Lines

• master c9 - promoted 09/20

  • https://bugs.launchpad.net/bugs/1990556 OVB jobs on c9 are hitting RETRY status - Max retries exceeded with url: /openstack/requirements/stable/wallaby/upper-constraints.txt
    • https://review.rdoproject.org/r/c/config/+/45232 Replace constraint_file link - connection timeouts
    • testing: https://review.rdoproject.org/r/c/testproject/+/31165/105/.zuul.yaml#9
  • https://bugs.launchpad.net/tripleo/+bug/1987092 Pacemaker performance causes intermittent galera issues in loaded CI env
    • suspecting systemd update - tried patch to downgrade
  • fs35: Verify ssl certificate failed - unable to get local issuer certificate. Ananya will check if there are related bugs
  • fs39: Internal Server Error (HTTP 500) in Keystone. Ananya will check related bug.
  • fs20: neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failed and Failed containers: rabbitmq_bootstrap, mysql_bootstrap"
    • Rerun: https://review.rdoproject.org/r/c/testproject/+/44965 (with Ronelle's fix)

• wallaby c9 - promoted 09/22

  • fs20: tempest failed (tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion)
  • fs35: mutually exclusive tempest fails for hash 0202d46f0d8ea73bbbbc94892bf8659d - we can skip to promote
  • fs1: tempest fail
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45238

• wallaby c8 - promoted 09/21

  • https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
  • https://bugs.launchpad.net/tripleo/+bug/1990012
  • fs1: Node introspection failed
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45239

• train c8 - promoted 09/21

  • last promotion: 21 Sept
  • standalone-full-tempest-api: tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body failed
  • sc10 kvm: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest) failed
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45240 , https://code.engineering.redhat.com/gerrit/c/testproject/+/429481

• rhel-9 osp17-1

  • promoted 09/21

• rhel-9 osp17

  • promoted 09/22

• rhel-8 osp17-1

  • promoted 09/22

• rhel-8 osp16-2

  • tempest component promoted
  • rekicked 16.2 - only job missing from promotion is the baremetal job - line is starting again now

Component lines

• master

  • clients
  • common
  • network
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45264

• wallaby c9:

  • network:
  • tripleo:
    • multinode-tripleo-wallaby-validation: Failed to connect to the host via ssh: kex_exchange_identification - Saw this only once
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45254

• wallaby c8:

  • network: tempest tests
  • tripleo: tempest tests
  • Rerun: https://review.rdoproject.org/r/c/testproject/+/45253

• train : green