# Ruck Rover 2022-07-22 to 2022-07-28

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/KrGAxws2QiaHuJmj8TH6Jg

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

---
## Handover
- Downstream:
- Bugs which are still hitting us
- https://bugzilla.redhat.com/show_bug.cgi?id=2105408
- https://bugzilla.redhat.com/show_bug.cgi?id=2110535 ( We have workaround for this and Ronelle have patches up, may be will get merged before rr sync.)
* **NOTE**: all check OVB failing on node provision: https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0. Ananya, Bhagyashri pls check in it

## 2022-07-28
* https://bugs.launchpad.net/tripleo/+bug/1983004

## 2022-07-27
* Closed: https://bugzilla.redhat.com/show_bug.cgi?id=2110245, https://bugzilla.redhat.com/show_bug.cgi?id=2109495

## 2022-07-26
* https://bugs.launchpad.net/tripleo/+bug/1982874 - Tempest test tempest.api.object_storage.test_object_services.ObjectTest.test_create_object_with_transfer_encoding is failing
* This test never ran before, but recently was enabled on tempest, and now it's failing, adding to skiplist
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/851079
* Downstream TripleO component promotion
* Blocked by: https://bugzilla.redhat.com/show_bug.cgi?id=2110535
migarcia rlandy: brew still broken, can't give you an updated repo
migarcia apparently the brew mount broke FAILED: GenericError: Repo directory missing: /mnt/brew/repos/rhos-17.0-rhel-9-trunk-candidate/6391114
* Created revert patch for https://review.opendev.org/c/openstack/tripleo-quickstart/+/850568 this will close https://trello.com/c/7xp7x4hZ/2630-cixlp1982195tripleociproa-cic9master-python-sqlalchemy-addition-in-cs9-base-broke-container-barbicanapidbsync
* Revert did not work, still getting the same issue, checking what's wrong

* Bug https://bugzilla.redhat.com/show_bug.cgi?id=2109495 can be closed once we will get green run of periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17
* Testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/300913/11/.zuul.yaml

## 2022-07-25

* Created revert patch for https://review.opendev.org/c/openstack/tripleo-quickstart/+/850568 this will close https://trello.com/c/7xp7x4hZ/2630-cixlp1982195tripleociproa-cic9master-python-sqlalchemy-addition-in-cs9-base-broke-container-barbicanapidbsync

#### Promotions
- Wallaby c9 and c8 promoted last_promotion=2022-07-25 13:39:03
- Train C8 promoted last_promotion=2022-07-25 15:04:29
### TODO
**Note:** https://bugzilla.redhat.com/show_bug.cgi?id=2109495#c3 - the issue impacting 16.2 OVB and 17 OVB is also blocking OVB in the Train TripleO component.
Amol, Arx, pls check in with Rabi about that in your morning

- As per the discussion with Rabi it is missing tripleo component.
- Tripleo jobs rerun: https://review.rdoproject.org/r/c/testproject/+/41465

Network component passed fs001 and is blocked on periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train. Downstream gerritt is down atm and can't rerun but I would skip promote this on monday if still failing - component is too old now

**Note:** Opened new bug: https://bugzilla.redhat.com/show_bug.cgi?id=2110245 and CIX - found durng ceph qualification but hits all rhos-17 rhe-9 container builds

* ~~Revert https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44140~~
* ~~Continue to work on components (check if train network component was updated)~~ - promoted last friday (without kvm internal though)
* Security component on wallaby need to get promoted
* Check tempest patch to tempest minimum basic scenario
* https://review.opendev.org/c/openstack/tempest/+/849562 - Patch updated, waiting for zuul and reviews
* ~~Check octavia skiplist~~
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850555 - Merged

## 2022-07-22
### CS9 Master -- last_promotion=2022-07-21 02:55:20

### CS9 Wallaby -- last_promotion=2022-07-21 13:15:57
- components: https://review.rdoproject.org/r/c/testproject/+/44058

### CS8 Wallaby -- last_promotion=2022-07-19 16:56:09
- components: https://review.rdoproject.org/r/c/testproject/+/41367

### CS8 Train -- last_promotion=2022-07-19 01:01:47
- components: https://review.rdoproject.org/r/c/testproject/+/44056
- components: https://code.engineering.redhat.com/gerrit/c/testproject/+/309187

### Bugs:

- https://bugzilla.redhat.com/show_bug.cgi?id=2109807 : Non integer values in LooseVersion('stable/1.0.0')

## Handover notes from previous rr

CS9 master and wallaby promotion will clear following bugs/CIX
- https://trello.com/c/7Ld4uihR/2627-cixlp1981829tripleociproa-octavia-tempest-tests-are-failing-with-policy-does-not-allow-this-request-to-be-performed-error
- https://bugs.launchpad.net/bugs/1981829
- revert this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626
- https://trello.com/c/qP0WWS6r/2629-cixlp1982137tripleociproa-error-grub-core-disk-diskfilterc916diskfilter-writes-are-not-supported
- https://bugs.launchpad.net/bugs/1982137
- CS9 wallaby got promoted
- master is lagging behind fs01.

https://trello.com/c/7xp7x4hZ/2630-cixlp1982195tripleociproa-cic9master-python-sqlalchemy-addition-in-cs9-base-broke-container-barbicanapidbsync
- wait for clients and security promotion and then wallaby promotion
- Need to verify
- https://review.opendev.org/c/openstack/oslo.db/+/850550 - clients component promoted
- https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 - security component promoted
- the wallaby promotion
- then revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442

https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft
- need to merge this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850155
- TODO: Need to test tempest fix on wallaby

CIX needs to triaged:
https://trello.com/c/9Lc7Z8Xw/2634-cixbz2109495osp162rhel8hosts-unreachable-during-gathering-facts-leading-to-ovb-job-failure
- Train network and tripleo components are lagging
- May be it will clean it up
- then promote rhos-16.2

Tracker bugs:
- https://trello.com/c/iEvKnb0H/2632-cixlp1982464tripleociproa-ipa-job-mirror-issue-tracker-for-master-and-wallaby
- https://trello.com/c/8cGNSfh8/2621-cixlp1981478tripleociproa-cs8-9-ovb-fs001-fs035-master-wallaby-train-failing-with-db-connection-errors

RHOS-17 RHEL-9
- Rlandy is messing with this line with ceph-5.2

RHOS-17 RHEL-8
- rlandy is working on promoting that.

Upstream Component
- to watch network and security

## 2022-07-21
### CS9 Master
***IMPORTANT***
Pushing forward promotion for: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087

chasing recent hash: https://review.rdoproject.org/r/c/testproject/+/42319

component: https://review.rdoproject.org/r/c/testproject/+/42374
Chasing hash: fcf18e1661b4add7eb1b3b2c761e1c63
Skip jobs: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 - those jobs passed tempests in different executions.
We're only missing fs020. Attempt to run it on PSI:
https://code.engineering.redhat.com/gerrit/c/testproject/+/419956

All criteria patches(skipped jobs) are reverted.

## Gate blocker for master & wallaby
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-ovn-provider-standalone
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-standalone
Reusing same bug: https://bugs.launchpad.net/tripleo/+bug/1981829
- moving it to skip list - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626

### CS9 Wallaby -- last_promotion=2022-07-19 11:52:25
components: https://review.rdoproject.org/r/c/testproject/+/41367
re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44115
https://bugs.launchpad.net/tripleo/+bug/1982464 - https://bugs.launchpad.net/tripleo/+bug/1982464

### CS9 master -- last_promotion=2022-07-21 02:55:20
re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44080

### CS8 Wallaby -- last_promotion=2022-07-19 16:56:09
components: https://review.rdoproject.org/r/c/testproject/+/44056

### RH8 RHOS16
- https://bugzilla.redhat.com/show_bug.cgi?id=2109495