# Ruck Rover 2022-09-23 to 2022-09-29

###### tags: `ruck_rover`
###### Next RR notes: TODO
###### Previous RR notes: https://hackmd.io/a92u74hDQhe5afE_LUtbRw

##### ruck: Chandan, rover: Ananya

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

### Handover Notes

Promotion Status:
CS9 master - 2022-09-29
CS9 wallaby - 2022-09-28
CS8 wallaby - 2022-09-28
CS8 Train - 2022-09-26

RHOS16-2 - 2022-09-23
RHOS17 RHEL-9 - 2022-09-29
RHOS17-1 RHEL-9 - 2022-09-26
RHOS17-1 RHEL-8 - 2022-09-24

Active Bugs:

Random and Inconsistent
- https://trello.com/c/fMby586x/2679-cixlp1987092tripleociproa-pacemaker-performance-causes-intermittent-galera-issues-in-loaded-ci-env

RHOS16-2
- https://trello.com/c/471dXQoC/2734-cixbz2130109osp162rhel8ceph-inventory-linking-fails-with-permission-issues
- https://trello.com/c/hcnDrBhO/2736-cixbz2130504osp162rhel8rhos-162-component-pipeline-jenkins-automatic-triggers-are-not-working

RHOS17-1
- https://trello.com/c/bF4diFmr/2726-cixftbfsosp-171-rhel-9

Wallaby
- https://trello.com/c/VMBKZlwg/2735-cixlp1991076tripleociproa-python-openstackclient-600-conflicting-dependencies
- https://trello.com/c/Uh4VX3Uk/2722-cixlp1990359tripleociproa-kernel-does-not-provide-mount-namespace-no-such-file-or-directory-edit
- https://bugs.launchpad.net/tripleo/+bug/1991240
- https://bugs.launchpad.net/tripleo/+bug/1991232

train
- https://trello.com/c/wMFW4qdv/2737-cixlp1991093tripleociproa-compute-and-network-tempest-tests-failing-on-fs35-train

### Sep 29, 2022

* Upstream gate/check
* train: https://bugs.launchpad.net/tripleo/+bug/1991232
* skip: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859825
* https://review.opendev.org/c/openstack/tripleo-ci/+/859830
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/859796 - revert
* tp: https://review.rdoproject.org/r/c/testproject/+/45378
* tp train: https://review.rdoproject.org/r/c/testproject/+/45380
* wallaby:
* ffu: https://bugs.launchpad.net/tripleo/+bug/1991240

### Integration:

* master:
* tp: https://review.rdoproject.org/r/c/testproject/+/45271
* **will promote**
* wallaby c9:
* 2022-09-28
* **running**
* wallaby c8
* 2022-09-28
* train
* 2022-09-26
* **Running**

### Sep 28, 2022

* Upstream gate: Green
* [CS8][Train] Exclude container-selinux-2:2.189.0 | https://review.opendev.org/c/openstack/tripleo-quickstart/+/859516
* https://bugs.launchpad.net/tripleo/+bug/1991076

### ovb issues

* Holding node for the galera/mysql issue

### Integration lines

* master
* last promotion: 2022-09-26
* fs001: [mysql/galera](https://bugs.launchpad.net/tripleo/+bug/1987092) **Node held**
* fs020: [mysql/galera](https://bugs.launchpad.net/tripleo/+bug/1987092) **Node held**
* fs064: [mysql/galera](https://bugs.launchpad.net/tripleo/+bug/1987092) **Node held**
* tp for one time failure: https://review.rdoproject.org/r/c/testproject/+/45271
* wallaby c9
* last promotion: 2022-09-26
* fs020: https://bugs.launchpad.net/tripleo/+bug/1990359
* fs001: [mysql/galera](https://bugs.launchpad.net/tripleo/+bug/1987092)
* tp for one time failures: https://review.rdoproject.org/r/c/testproject/+/45276
* wallaby c8
* last promotion: 2022-09-26
* train
* last promotion: 2022-09-26
* fs035: https://bugs.launchpad.net/tripleo/+bug/1991093
* tp to see if internal fs35 runs: https://code.engineering.redhat.com/gerrit/c/testproject/+/429821
* if ^ passes switch 35 to inetrnal in promo criteria:

### Component

* wallaby
* validations:
* fs001: https://bugs.launchpad.net/tripleo/+bug/1991076

### Downstream

* OSP16.2 - 23/09/2022
* https://bugs.launchpad.net/tripleo/+bug/1990988
* https://review.opendev.org/q/topic:lp%25231990988
* https://bugzilla.redhat.com/show_bug.cgi?id=2130504
* OSP 17.1 on RHEL-9 - 26/09/2022
* FTBFS opened currently
* OSP 17.1 on RHEL-8 - 24/09/2022
* OSP 17.0 on RHEL-9 - 27/09/2022

### Sep 27, 2022

* Upstream gate : Green

#### ovb failures

* fs39

* fs64

galera/sql issue: https://review.rdoproject.org/r/c/testproject/+/45279 - I am holding node here but this error is not consistent
#### Integration lines

* master:
* Last promotion: 2022-09-26
* fs002: https://bugs.launchpad.net/tripleo/+bug/1990556

* wallaby c9:
* last promotion: 2022-09-26
* fs20:

* train:
* sc000: package conflick cintainer selinux
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45240
#### Components

* master
* baremetal
* fs001: https://bugs.launchpad.net/tripleo/+bug/1987092
* security
* fs39: https://bugs.launchpad.net/tripleo/+bug/1987092 **Node is held**
* tp: https://review.rdoproject.org/r/c/testproject/+/45279

* wallaby c9
* Green

* wallaby c8
* Green

* train
* Green

* RHEL-9 RHOS-17.1
- promoted on 26th Sep, 2022
* RHEL-9 RHOS-17.0
- promoted on 27th Sep, 2022
* RHEL-8 RHOS-16.2
- promoted on 23rd Sept, 2022
- Bug: https://bugzilla.redhat.com/2130109

### Sep 26, 2022

* chasing downstream promotion
- https://code.engineering.redhat.com/gerrit/c/testproject/+/427532

* Upstream Check/Gate
- https://bugs.launchpad.net/tripleo/+bug/1990810 - affecting undercloud ffu job
+ Fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/859213
> Testing here: https://review.rdoproject.org/r/c/testproject/+/45269

* cs9-master
+ Last promotion: 2022-09-20
+ Rerun: https://review.rdoproject.org/r/c/testproject/+/45271
+ 35:
+ ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed No such file or directory - [journal_errors.txt](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/81ecba2/logs/overcloud-novacompute-0/var/log/extra/journal_errors.txt.gz)
+ Skipping 35 to promote as 35 internal passd on the same hash: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45275
+ Revert of skip: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44766

* cs9-wallaby
+ Last promotion: 2022-09-22
+ 35
+ 1
+ testproj: https://review.rdoproject.org/r/c/testproject/+/45276

* cs8-wallaby
+ Last promotion: 2022-09-23
+ Should promote today

* cs8-train
+ Last promotion: 2022-09-21
+ 35:
+ Skip 35 and promote as 35 internal passed for 3a74: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45275

* rhel-8-rhos-16.2
+ Last promotion: 2022-09-23

* rhel-8-rhos-17.1
+ Last promotion: 2022-09-24

* rhel-9-rhos-17.0
+ Last Promotion: 2022-09-24

* rhel-9-rhos-17.1
+ Last promotion: 2022-09-23

### Components

* master:
* tripleo
* compute
* tp: https://review.rdoproject.org/r/c/testproject/+/45279

* wallaby c9
* network
* tripleo
* tp: https://review.rdoproject.org/r/c/testproject/+/45277

* wallaby c8
* Green

* train
* Green

## RR Handoff

#### Integration Lines

* master c9 - **promoted 09/20**
* https://bugs.launchpad.net/bugs/1990556 OVB jobs on c9 are hitting RETRY status - Max retries exceeded with url: /openstack/requirements/stable/wallaby/upper-constraints.txt
* https://review.rdoproject.org/r/c/config/+/45232 Replace constraint_file link - connection timeouts
* testing: https://review.rdoproject.org/r/c/testproject/+/31165/105/.zuul.yaml#9
* https://bugs.launchpad.net/tripleo/+bug/1987092 Pacemaker performance causes intermittent galera issues in loaded CI env
* suspecting systemd update - tried patch to downgrade
* fs35: Verify ssl certificate failed - unable to get local issuer certificate. Ananya will check if there are related bugs
* fs39: Internal Server Error (HTTP 500) in Keystone. Ananya will check related bug.
* fs20: neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failed and Failed containers: rabbitmq_bootstrap, mysql_bootstrap"
* Rerun: https://review.rdoproject.org/r/c/testproject/+/44965 (with Ronelle's fix)

* wallaby c9 - **promoted 09/22**
* fs20: tempest failed (tempest.scenario.test_minimum_basic.TestMinimumBasicScenario.test_minimum_basic_instance_hard_reboot_after_vol_snap_deletion)
* fs35: [mutually](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/215bc3f/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz) [exclusive](https://logserver.rdoproject.org/38/45238/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/bd2b2df/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz) tempest fails for hash 0202d46f0d8ea73bbbbc94892bf8659d - we can skip to promote
* fs1: [tempest fail](https://logserver.rdoproject.org/38/45238/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/ccfd757/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz)
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45238

* wallaby c8 - **promoted 09/21**
* https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
* https://bugs.launchpad.net/tripleo/+bug/1990012
* fs1: Node introspection failed
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45239

* train c8 - **promoted 09/21**
* last promotion: 21 Sept
* standalone-full-tempest-api: tempest.api.compute.admin.test_volumes_negative.VolumesAdminNegativeTest.test_update_attached_volume_with_nonexistent_volume_in_body failed
* sc10 kvm: setUpClass (octavia_tempest_plugin.tests.scenario.v2.test_pool.PoolScenarioTest) failed
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45240 , https://code.engineering.redhat.com/gerrit/c/testproject/+/429481

* [rhel-9 osp17-1](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline)
* **promoted 09/21**

* [rhel-9 osp17](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline)
* **promoted 09/22**

* rhel-8 osp17-1
* **promoted 09/22**

* [rhel-8 osp16-2](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline)
* tempest component promoted
* rekicked 16.2 - only job missing from promotion is the baremetal job - line is starting again now

## Component lines

* master
* clients
* common
* network
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45264

* wallaby c9:
* network:
* tripleo:
* multinode-tripleo-wallaby-validation: Failed to connect to the host via ssh: kex_exchange_identification - Saw this only once
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45254

* wallaby c8:
* network: tempest tests
* tripleo: tempest tests
* Rerun: https://review.rdoproject.org/r/c/testproject/+/45253

* train : green