# Ruck Rover 2022-07-15 to 2022-07-21

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/hD7Q6AsZQQyki_v4BoMfxg

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

---
## 2022-07-22 - handover
All skip jobs patches are reverted.

### CIX needs to be triaged:
https://trello.com/c/9Lc7Z8Xw/2634-cixbz2109495osp162rhel8hosts-unreachable-during-gathering-facts-leading-to-ovb-job-failure
- Train network and tripleo components are lagging
- May be it will clean it up
- then promote rhos-16.2

### Tracker bugs:
- https://trello.com/c/iEvKnb0H/2632-cixlp1982464tripleociproa-ipa-job-mirror-issue-tracker-for-master-and-wallaby
- https://trello.com/c/8cGNSfh8/2621-cixlp1981478tripleociproa-cs8-9-ovb-fs001-fs035-master-wallaby-train-failing-with-db-connection-errors

### CS9 Master -- last_promotion=2022-07-21 16:31:19
To watch network and security
- components: https://review.rdoproject.org/r/c/testproject/+/42374

#### Bugs
- https://bugs.launchpad.net/bugs/1982137
- https://trello.com/c/qP0WWS6r/2629-cixlp1982137tripleociproa-error-grub-core-disk-diskfilterc916diskfilter-writes-are-not-supported

- https://bugs.launchpad.net/bugs/1981829
- https://trello.com/c/7Ld4uihR/2627-cixlp1981829tripleociproa-octavia-tempest-tests-are-failing-with-policy-does-not-allow-this-request-to-be-performed-error
- revert this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626

- https://bugs.launchpad.net/bugs/1982195
- https://trello.com/c/7xp7x4hZ/2630-cixlp1982195tripleociproa-cic9master-python-sqlalchemy-addition-in-cs9-base-broke-container-barbicanapidbsync
- verify promotions: clients and security
- https://review.opendev.org/c/openstack/oslo.db/+/850550 - clients component promoted
- https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/44082 - security component promoted
- the wallaby promotion
- then revert https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442

### CS9 Wallaby -- last_promotion=2022-07-21 13:15:57
- components: https://review.rdoproject.org/r/c/testproject/+/44058

#### Bugs
- https://bugs.launchpad.net/bugs/1982137
- https://trello.com/c/qP0WWS6r/2629-cixlp1982137tripleociproa-error-grub-core-disk-diskfilterc916diskfilter-writes-are-not-supported
- https://trello.com/c/8tGYExhe/2603-cixlp1980255tripleociproa-tripleo-ci-centos-9-standalone-and-multinode-ipa-are-failing-the-testminimumbasicinstancehardrebootaft
- https://review.opendev.org/c/openstack/tempest/+/849562
- need to merge this https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850155
- TODO: Need to test tempest fix on wallaby

### CS8 Wallaby -- last_promotion=2022-07-19 16:56:09
- components: https://review.rdoproject.org/r/c/testproject/+/41367

### CS8 Train -- last_promotion=2022-07-19 01:01:47
- components: https://review.rdoproject.org/r/c/testproject/+/44056
- components: https://code.engineering.redhat.com/gerrit/c/testproject/+/309187

### RH8 OSP16.2 -- last_promotion=2022-07-18 22:05:30
- promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

### RH8 OSP17 -- last_promotion=2022-07-18 11:04:22
rlandy is working on promoting that.

- promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/309187

### RH9 OSP17 -- ast_promotion=2022-07-21 02:10:54
rlandy is working on this line with ceph-5.2

- promotion: https://code.engineering.redhat.com/gerrit/c/testproject/+/408119

---

## 2022-07-21
### CS9 Master
Pushing forward promotion for: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087

chasing recent hash: https://review.rdoproject.org/r/c/testproject/+/42319

component: https://review.rdoproject.org/r/c/testproject/+/42374
Chasing hash: fcf18e1661b4add7eb1b3b2c761e1c63
Skip jobs: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44087 - those jobs passed tempests in different executions.
We're only missing fs020. Attempt to run it on PSI:
https://code.engineering.redhat.com/gerrit/c/testproject/+/419956

All criteria patches(skipped jobs) are reverted.

## Gate blocker for master & wallaby
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-ovn-provider-standalone
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-scenario010-standalone
Reusing same bug: https://bugs.launchpad.net/tripleo/+bug/1981829
- moving it to skip list - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850626

### CS9 Wallaby -- last_promotion=2022-07-19 11:52:25
components: https://review.rdoproject.org/r/c/testproject/+/41367
re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44115
https://bugs.launchpad.net/tripleo/+bug/1982464 - https://bugs.launchpad.net/tripleo/+bug/1982464

### CS9 master -- last_promotion=2022-07-21 02:55:20
re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44080

### CS8 Wallaby -- last_promotion=2022-07-19 16:56:09
components: https://review.rdoproject.org/r/c/testproject/+/44056

### RH8 RHOS16
- https://bugzilla.redhat.com/show_bug.cgi?id=2109495

## 2022-07-20
### Current issues
* CS9 Master and Stable/Wallaby Check bug: https://bugs.launchpad.net/tripleo/+bug/1982195
* tripleo-ci-centos-9-standalone is failing on stable/wallaby - tempest scenario - failed to reach ACTIVE status and task state "None" within the required time https://bugs.launchpad.net/tripleo/+bug/1982227
* Workaround: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850428 Downgrade python3-sqlalchemy and libknet1.x86_64 (disclaimer: idk if we need both downgrades - but we can remove them one at a time and see. Both clears the gate)
* https://review.opendev.org/q/topic:bug%252F1982227
* https://zuul.opendev.org/t/openstack/status#850442 sqlalchemy downgrade seems to be fix the issue
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/850442

### CS9 Master -- lagging behind promotion
* components: https://review.rdoproject.org/r/c/testproject/+/42319
* jobs: https://review.rdoproject.org/r/c/testproject/+/44037
* Re-running master job: https://review.rdoproject.org/r/c/testproject/+/44080
### CS9 Wallaby -- last_promotion=2022-07-19 11:52:25
components: https://review.rdoproject.org/r/c/testproject/+/41367
### CS8 Wallaby -- last_promotion=2022-07-19 16:56:09
components: https://review.rdoproject.org/r/c/testproject/+/44056
### CS8 Train -- last_promotion=2022-07-19 16:56:09
components: https://review.rdoproject.org/r/c/testproject/+/42374
### RH8 OSP16.2 -- last_promotion=2022-07-18 22:05:30
* https://code.engineering.redhat.com/gerrit/c/testproject/+/317973
### RH8 OSP17 -- last_promotion=2022-07-18 11:04:22
### RH9 OSP17 -- lagging behind promotion - fs035
* bugfix: https://code.engineering.redhat.com/gerrit/c/tripleo-ansible/+/420543 - merged
* testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/420692 to promote tripleo component and kick RHOS-17 RHEL-9 line

## 2022-07-19
### Current Issues
**Important**:
SC010 OVN is failing for both CS9 master & wallaby
SC010 RHEL 8 RHOS 17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-17&skip=0
https://bugs.launchpad.net/tripleo/+bug/1981829

### CS9 master
`status=Yellow last_promotion=2022-07-12 13:54:22`
Re-Running failed job: https://review.rdoproject.org/r/c/testproject/+/44037
- https://bugs.launchpad.net/tripleo/+bug/1982137 - ovb node provisioning issue

Check failing on a bug: https://bugs.launchpad.net/tripleo/+bug/1982195

#### Bugs
- https://bugzilla.redhat.com/show_bug.cgi?id=2103118
- Debug env: https://code.engineering.redhat.com/gerrit/c/testproject/+/420316 gthiemonge is looking into that
- https://bugs.launchpad.net/tripleo/+bug/1982137

#### Components
- https://review.rdoproject.org/r/c/testproject/+/42319
#### Tripleo promotion
Pushing forward tripleo promotion to get latest changes in.
- https://review.rdoproject.org/r/c/testproject/+/41367

### CS9 wallaby!!!
`status=Red last_promotion=2022-07-19 11:52:25`
\o/ promoted today

#### Bugs
- https://bugs.launchpad.net/tripleo/+bug/1981829

### CS8 wallaby
`status=Red last_promotion=2022-07-15 17:10:42`
* https://review.rdoproject.org/r/c/testproject/+/44056

### CS8 train!!!
`status=Green last_promotion=2022-07-19 01:01:47`
\o/

### RHEL 8 RHOS 16.2
`status=Red last_promotion=2022-07-18 04:03:34`
* https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

### RHEL 8 RHOS 17!!!
`status=Green last_promotion=2022-07-18 11:04:22`
\o/

### RHEL 9 RHOS 17
`status=Red last_promotion=2022-07-14 10:54:02`
* https://code.engineering.redhat.com/gerrit/c/testproject/+/408119
* FS035 - https://bugzilla.redhat.com/show_bug.cgi?id=2108580

---
**STOP! All updates**

## 18th Jul, 2022
**Important**:
SC010 OVN is failing for both CS9 master & wallaby
SC010 RHEL 8 RHOS 17 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-octavia-rhos-17&skip=0
https://bugs.launchpad.net/tripleo/+bug/1981829

### CS9 master
Last promotion: 12th July
Bugs:
- https://bugzilla.redhat.com/show_bug.cgi?id=2103118
- Debug env: https://code.engineering.redhat.com/gerrit/c/testproject/+/420316 gthiemonge is looking into that
- https://bugs.launchpad.net/tripleo/+bug/1981829
- Moving test to skip list: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/850137 (Not yet merged)
- Proper fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/850118
- https://review.opendev.org/c/openstack/tripleo-ci/+/850225
- TP: https://review.rdoproject.org/r/c/testproject/+/44038
- Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44037

Patches to Clear OVN issue for Master and wallaby
- ~~master: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/776942~~
- ~~Wallaby: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/850033~~
- We need to fix standalone jobs in tripleo-ci not in tripleo-quickstart (https://review.opendev.org/c/openstack/tripleo-quickstart/+/850118)
- https://review.opendev.org/c/openstack/tripleo-ci/+/850225

Once ovn gets cleared for master, fs035 is already busted with different tempest tests, fs01 is also failing at different issue but fs020 is passong
may be we can remove fs01/fs035 from criteria and promote master?

### CS9 Wallaby
Last promotion: 12th July
- Rabbitq bump coming: https://review.rdoproject.org/r/c/rdoinfo/+/44025
- https://review.opendev.org/c/openstack/tripleo-quickstart/+/848439
- Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/41367

### CS8 train
Last promotion: 15th July
Re-running failed jobs: https://review.rdoproject.org/r/c/testproject/+/44045

### RH9 RHOS-17
Last promotion: 14th July
Re-running failed jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/420204

### RH8 RHOS-17
Last promotion: 18th july

### RH8 RHOS-16.2
Last promotion: 18th July
Re-running failed jobs:
https://code.engineering.redhat.com/gerrit/c/testproject/+/317973

## 15th Jul, 2022

### Upstream Gate Blocker
* https://bugs.launchpad.net/tripleo/+bug/1981634 needs to be marked duplicate in favor of https://bugs.launchpad.net/tripleo/+bug/1981467

### CS9 master
Last promotion: 12th July
Re-running soniya tp: https://review.rdoproject.org/r/c/testproject/+/42692
sc10 ovn blocker: https://bugs.launchpad.net/tripleo/+bug/1981829
fs035 - https://paste.centos.org/view/raw/09698f75
- trying with one tempest test: https://review.rdoproject.org/r/c/testproject/+/44020
* started fresh patch: https://review.rdoproject.org/r/c/testproject/+/42374
* ~~downstream for kvm: https://code.engineering.redhat.com/gerrit/c/testproject/+/408119~~
* disabled kvm for promo: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44029

### CS9 Wallaby
Last promotion: 12th July
* https://review.rdoproject.org/r/c/testproject/+/41367
* https://code.engineering.redhat.com/gerrit/c/testproject/+/309187
* components: https://review.rdoproject.org/r/c/testproject/+/42319
* sc10 ovn blocker: https://bugs.launchpad.net/tripleo/+bug/1981829

### CS8 train
Last promotion: 15th July

~~Re-running failed job: https://review.rdoproject.org/r/c/testproject/+/44018~~

### CS8 Wallaby:
last_promotion=2022-07-15 17:10:42
* ~~Revert of promotion skip: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43689~~

### RH8 RHOS-16.2
Last promotion: 12th july
* Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/420016

### RH8 RHOS-17
Last promotion: 13th july
* Re-running failed job: https://code.engineering.redhat.com/gerrit/c/testproject/+/419956
* components: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027

### RH9 RHOS-17
Last promotion: 14th July
* components: https://code.engineering.redhat.com/gerrit/c/testproject/+/407601