# Ruck Rover 2022-07-29 to 2022-08-04

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/lfpJ3NdRTXaTah6t3C1oGw

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

[Downstream promoter](http://10.0.110.143/promoter_logs/)

---
## Handover

### Downstream

* rhos17.1: issue with sc001, sc002 and sc010
* previous issue was: https://bugzilla.redhat.com/show_bug.cgi?id=2113846
* After fix merged test project patch https://code.engineering.redhat.com/gerrit/c/testproject/+/420272 waiting for result

* rhos17 on rhel9: promoted 3rd Aug
* re-running the failure jobs from recent run: https://code.engineering.redhat.com/gerrit/c/testproject/+/421970

* rhos17 on rhel8: Promoted yesterday
* we were having issue with multinode ipa job failure but after fix : https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/423386 it's cleared now.

* rhos16.2 : promoted yesterday
* sc010 is failing intermitently not sure it's infra issue or legit issue: https://bugzilla.redhat.com/show_bug.cgi?id=2114736

## component
* All the component line are doing good.

---
## 2022-08-04

Gate is green
* tripleo-ci-centos-8-9-multinode-mixed-os:
* https://bugs.launchpad.net/tripleo/+bug/1983582
* wallaby content provider:
* https://bugs.launchpad.net/tripleo/+bug/1983585

### Check jobs

* tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039
* Mirror issues + https://bugs.launchpad.net/tripleo/+bug/1982464
* Rerun to hold the node https://review.rdoproject.org/r/c/testproject/+/44412
* Stopped the cleanup script **todo: revert afterwards**

### master
* promoted 1 Aug
* ~~Bug https://bugs.launchpad.net/tripleo/+bug/1983467 - image mount~~
* Fix suggested:
* ~~https://review.opendev.org/c/openstack/tripleo-common/+/835774~~
* ~~https://review.opendev.org/c/openstack/tripleo-common/+/828412~~
* Testing fix: https://review.rdoproject.org/r/c/testproject/+/44397

### wallaby c9
* promoted 1 Aug
* https://review.rdoproject.org/r/c/testproject/+/31165
* **promoting**

### wallaby c8
* promoted 3 Aug

### train
* promoted Aug 1
* all tempest related failures but no specific test that fails
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train is red but does not have fixed failure reason
* Rerun: https://review.rdoproject.org/r/c/testproject/+/44410

### rhos17.1 on rhel9

* https://bugzilla.redhat.com/show_bug.cgi?id=2113846
* https://bugzilla.redhat.com/show_bug.cgi?id=2113846#c12 - we have few comments on bz from those : in the next run it should pass

### rhos17 on rhel9 - **Promoted 3rd Augest**

### rhos17 on rhel8 - ** Promoted today ***

* multinode ipa job is failing with tempest test failure
* fix: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/423386 retesting: https://code.engineering.redhat.com/gerrit/c/testproject/+/422387
* here is the regression test result: https://code.engineering.redhat.com/gerrit/c/testproject/+/422387/4#message-d7cab1e28e3d4370b7ba80f6282a6cbaa87e8a5f
* Note: only rhos17 on rhel8 multinode job is failing rhos17 on rhel9 is doing good. we don't have multinode job in rhos16.2 on rhel8 integration line
* **repoerted bug:** https://bugzilla.redhat.com/show_bug.cgi?id=2114736
* tempest skiplist patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/851968/ -W on it for investigation
* doing regression here https://code.engineering.redhat.com/gerrit/c/testproject/+/422387

### rhos16.2 on rhel8 - **Promoted today**

* ~~featureset020-internal-rhos-16.2 having node_failure re-ran job here and got SUCCESS: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/4#message-db7851d3bcd5f924ebb176f50ae54c9ead07b894~~
* rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 having disk_full issue re-running job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400

### Upstream components

### master
* common: 3 Aug
* compute: 2 Aug
* security: 1 Aug
* tripleo: 18 July
* puppet issue cleared but new failures:
* standalone deploy timed out
* checking https://review.rdoproject.org/r/c/testproject/+/44404
*
* validations:
* https://review.rdoproject.org/r/c/testproject/+/44413
* real issue: https://logserver.rdoproject.org/13/44413/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/a965b91/job-output.txt - pls bug and send to jiri or matbu

### wallaby c9
* baremetal: 29 july
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby is red
* checking
* network: 1 aug
* https://review.rdoproject.org/r/c/testproject/+/44414
* tripleo: 2 aug
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby is red

### wallaby c8
* tripleo : 3 aug

### train
* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train **is red**

### Downstream component

* rekicked the failing triggerd jobs
* rerunning the failing component job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421971 [DNM] test failing component jobs

* ** 16-2 components**: **all compoonents are doing good - GREEN**
* rhos17 on rhel8 and rhel9 component status:
* fs001 is failing on tripleo compnent rhos17 on rhel8 and rhel9: tempest test are failing re-running the jobs -
* consistent failure: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17&skip=0
* failed once: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17&skip=0 - **SUCCESS on testproject patch**
* rerunning jobs here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421971

* **NOW it's cleared** - common component jobs were failing on 16-2 and 17 on rhel8:
* ~~rhos17 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2112344 - **The issue is resolved**~~
* ~~rhos16.2 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2113948 - **Looks better now**~~

* **NOT in criteria**: failing consistently rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17 : https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17&skip=0

---
## 2022-08-03

Gate is green

### master cs9
* Promoted - 1st aug
* Rerun: https://review.rdoproject.org/r/c/testproject/+/44397 but there might be legit isssues
* Bug: https://bugs.launchpad.net/tripleo/+bug/1983467
* Probable fix: https://review.opendev.org/c/openstack/tripleo-common/+/851984
* Testing fix: https://review.rdoproject.org/r/c/testproject/+/44397
* ^ has passed the image mount task
* The fix works, but hjensas will update the patch as he has a comment on it. Ananya will run the testproj again **tomorrow**.

### wallaby cs9
* Promoted - 1st aug
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby failed a lot - **todo: find out why**

### wallaby cs8
* Promoted - 2nd aug
* rerun: https://review.rdoproject.org/r/c/testproject/+/44398
* tempest tests failed in one passed in other (f62f..)
* **todo skip and promote**

### train cs8
* Promoted - 1st aug

### rhos16.2 on rhel8
* promoted on 1st Augest
* rerunning failing jobs: sc010, sc012 and fs020 - https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 [DNM] Test rhos-16.2 failing jobs
* sc012 and fs020 passed
* sc010 is failing on tempest test
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2&skip=0
* https://sf.hosted.upshift.rdu2.redhat.com/logs/00/422400/2/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/08ba8f6/logs/undercloud/var/log/tempest/stestr_results.html
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/d7dbc5e/logs/undercloud/var/log/tempest/stestr_results.html
* dooing regression onver sc010 job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 if it fails then will file a bug
* **repoerted bug:** https://bugzilla.redhat.com/show_bug.cgi?id=2114736
* tempest skiplist patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/851968/

### rhos17 on rhel8
* rerunning rhos17 on rhel8 failing jobs: fs001 , sc002 and multinode-ipa job failing rerunning jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422387
* sc002 and fs001 passed
* multinode-ipa is failing with some tempest test
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17&skip=0
* https://sf.hosted.upshift.rdu2.redhat.com/logs/87/422387/3/check/periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17/59f34d4/logs/undercloud/var/log/tempest/stestr_results.html
* **repoerted bug:** https://bugzilla.redhat.com/show_bug.cgi?id=2114736
* tempest skiplist patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/851968/ -W on it for investigation
* doing regression here https://code.engineering.redhat.com/gerrit/c/testproject/+/422387

### rhos17 on rhel9
* currently integration line is running waiting for result
* promoted aug 2

### rhos17.1 on rhel9
* sc001 and sc002 is failing : https://bugzilla.redhat.com/show_bug.cgi?id=2113846

## Component

### master

* baremetal: 2 Aug
* Rerun: https://review.rdoproject.org/r/c/testproject/+/44390
* ["No volumetype with a name or ID of 'tripleo' exists. - **todo: ananya will check and file bug tomorrow**

* tripleo: 18 July
* https://bugs.launchpad.net/tripleo/+bug/1983281
* fix: https://review.rdoproject.org/r/c/rdoinfo/+/44394 '
* testing fix: https://review.rdoproject.org/r/c/testproject/+/44373
* needs promotion
* promote job: https://review.rdoproject.org/r/c/testproject/+/44373
* The puppet thing is cleared but **ch**: https://review.rdoproject.org/r/c/testproject/+/44404

* security: 1 Aug
* https://review.rdoproject.org/r/c/testproject/+/44393

### wallaby c9
* baremetal: 29 July
* network: 1 Aug
* tripleo: 2 Aug
*

### wallaby c8
green

### train
* tripleo: **todo**
*

### Downstream component

* rekicked the failing triggerd jobs
* rerunning the failing component job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421971 [DNM] test failing component jobs
* fs001 bm job is failing with error DISK_FULL: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17&skip=0
* common component jobs are failing on 16-2 and 17 on rhel8
* rhos17 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2112344 - **we have fix for this issue and hopefully it will come in next dlrn run.**
* rhos16.2 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2113948 - **Looks better now**

---
## 2022-08-02

Gate is green

### master cs9
* Promoted - 1st aug
* https://bugs.launchpad.net/tripleo/+bug/1983467 - image mount

### wallaby cs9
* Promoted - 1st aug

### wallaby cs8
* Promoted - 2nd aug

### train cs8
* Promoted - 1st aug

### rhos16.2 on rhel8
* promoted on 1st Augest
* rerunning failing jobs: sc010, sc012 and fs020 - https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 [DNM] Test rhos-16.2 failing jobs
* sc012 and fs020 passed
* sc010 is failing on tempest test
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2&skip=0
* https://sf.hosted.upshift.rdu2.redhat.com/logs/00/422400/2/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/08ba8f6/logs/undercloud/var/log/tempest/stestr_results.html
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/d7dbc5e/logs/undercloud/var/log/tempest/stestr_results.html
* dooing regression onver sc010 job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 if it fails then will file a bug

### rhos17 on rhel8
* rerunning rhos17 on rhel8 failing jobs: fs001 , sc002 and multinode-ipa job failing rerunning jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422387
* sc002 and fs001 passed
* multinode-ipa is failing with some tempest test
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17&skip=0
* https://sf.hosted.upshift.rdu2.redhat.com/logs/87/422387/3/check/periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17/59f34d4/logs/undercloud/var/log/tempest/stestr_results.html
* https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17/cf9218e/logs/undercloud/var/log/tempest/stestr_results.html
* doing regression over multinode ipa job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422387 if it fails then will report a bug

### rhos17 on rhel9
* currently integration line is running waiting for result
* promoted aug 2

### rhos17.1 on rhel9

* https://bugzilla.redhat.com/show_bug.cgi?id=2113846

## Component

### master
* baremetal:
* last promotion 31 July
* transient error
* Rerun https://review.rdoproject.org/r/c/testproject/+/44365
* common:
* last promotion 23 July
* https://github.com/redhat-openstack/openstack-selinux/pull/96
* Rerun as ^ is merged https://review.rdoproject.org/r/c/testproject/+/44342 : Failed
* [ ] https://bugs.launchpad.net/tripleo/+bug/1983382
* Rerun
* https://review.rdoproject.org/r/c/testproject/+/44342
* https://review.rdoproject.org/r/c/testproject/+/44374 (all common jobs)
* https://review.rdoproject.org/r/c/testproject/+/44375 (all jobs without rm -rf)
* https://review.opendev.org/c/openstack/tripleo-common/+/851901 (dnm ^)
* Left notes in https://bugs.launchpad.net/tripleo/+bug/1983382/comments/3

* tempest:
* last promotion 30 July
* transient tempest failures
* Rerun https://review.rdoproject.org/r/c/testproject/+/44366 **is running now**

* tripleo:
* last promotion 18 July
* https://bugs.launchpad.net/tripleo/+bug/1983281
* takashi is working on this
* https://review.opendev.org/c/openstack/puppet-openstack-integration/+/851867 trying with this patch [wip]
* https://review.rdoproject.org/r/c/testproject/+/44373
* Chandan should be able to help you with the RDO pin

### wallaby c9
* common:
* last promotion 27 July
* https://github.com/redhat-openstack/openstack-selinux/pull/96
* Will rerun once the master common testproj passes
* https://bugs.launchpad.net/tripleo/+bug/1983382

### wallaby c8
* common:
* last promotion 26 July
* https://github.com/redhat-openstack/openstack-selinux/pull/96
* Will rerun once the master common testproj passes
* https://bugs.launchpad.net/tripleo/+bug/1983382

### train
* tripleo:
* last promotion 26 July
* Tempest failures
* Two job ran on the same hash with different tempest failures - so we can skip and promote https://logserver.rdoproject.org/65/31165/70/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/3e33929/job-output.txt and https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/dbe4ec3/job-output.txt
* Rerun https://review.rdoproject.org/r/c/testproject/+/44368 and https://review.rdoproject.org/r/c/testproject/+/44369 **is running now**

### Downstream component

* rekicked the failing triggerd jobs
* rerunning the failing component job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421971 [DNM] test failing component jobs
* fs001 bm job is failing with error DISK_FULL: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-bm_envE-3ctlr_1comp-featureset001-tripleo-rhos-17&skip=0
* common component jobs are failing on 16-2 and 17 on rhel8
* rhos17 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2112344
* rhos16.2 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2113948

---
## 2022-08-01

Gate is green

### master c9
* centos-9-scenario010-kvm-internal-standalone is failing in octavia tempest
* remove from criteria: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44347
* tests
* [ ] https://bugzilla.redhat.com/show_bug.cgi?id=2105408
* rerun: https://review.rdoproject.org/r/c/testproject/+/44334 (39)
* rerun: https://review.rdoproject.org/r/c/testproject/+/44337 (39 second testproj)
* rerun: https://code.engineering.redhat.com/gerrit/c/testproject/+/422321 (sc 10 kvm internal)
* **promoted - no skips required**

### wallaby c9
* centos-9-scenario010-kvm-internal-standalone is failing in octavia tempest tests
* [ ] https://bugzilla.redhat.com/show_bug.cgi?id=2105408
* remove from criteria: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44347
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby
* [ ] https://bugs.launchpad.net/tripleo/+bug/1982464
* holding node for amoralej: https://review.rdoproject.org/r/c/testproject/+/44338 - updated with remove_ovb_after_job: false, need to stop cleanup script
* added # to /etc/openstack/clouds.yaml in toolbox
* Looks like the mirrors are okay now as the job passed the ipa task
* Reverted cleanup script
* **TODO**: more reruns to verify if this is resolved
* **promoted**

### wallaby c8
* missing: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby
* rerun in https://review.rdoproject.org/r/c/testproject/+/31165
* **promoting**

### train c8
~~* fs001 failing - testproject rerun: https://review.rdoproject.org/r/c/testproject/+/36254~~
* **promoted**

### rhos17.1 on rhel9
* rerunning failed jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/418641
* failures on periodic-tripleo-ci-rhel-9-scenario001-standalone-rhos-17.1, periodic-tripleo-ci-rhel-9-scenario002-standalone-rhos-17.1, periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17.1 - needs investigation
* 2022-08-01 15:05:42.272269 | fa163e7d-167c-b5b9-83bf-0000000037cc | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-startup-config/step_5 | standalone | error={"changed": false, "msg": "Failed containers: ceilometer_gnocchi_upgrade"}
* **BZ required** https://sf.hosted.upshift.rdu2.redhat.com/logs/37/418637/18/check/periodic-tripleo-ci-rhel-9-scenario002-standalone-rhos-17.1/bdba6a3/logs/undercloud/var/log/extra/podman/containers/ceilometer_gnocchi_upgrade/stdout.log - pls bug and pass to #rhos-delivery

### rhos17 on rhel9
* rerunning failing rhos17 on rhel9 jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/421970 [DNM] Test failing jobs
* periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17 in rerun again

### rhos17 on rhel8
* https://code.engineering.redhat.com/gerrit/c/testproject/+/422387 [DNM] Test failing rhos17 on rhel8 jobs [NEW]
* whole line in rerun now
* rerun in https://code.engineering.redhat.com/gerrit/c/testproject/+/418637

### rhos16.2 on rhel8

* https://code.engineering.redhat.com/gerrit/c/testproject/+/422400 [DNM] Test rhos-16.2 failing jobs
* One job is failing - "pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv6-ge" currently it's running.
* **promoted**

## Components:

### common on master and wallaby c9
* [ ] https://bugs.launchpad.net/tripleo/+bug/1982744/ - workaround is to run sudo dnf reinstall openstack-selinux container-selinux
sudo rpm -V openstack-selinux
* releasefile workaround: https://review.opendev.org/c/openstack/tripleo-quickstart/+/851735
* testproj without container rebuild: https://review.rdoproject.org/r/c/testproject/+/44341
* Node is held
* **TODO**: run the workarounds on the node
* testproj with container rebuild: https://review.rdoproject.org/r/c/testproject/+/44342

### tripleo on master
* [ ] Bug: https://bugs.launchpad.net/tripleo/+bug/1983281
* pinged takashi

### network on master

* periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master fails tempest tests sometimes, but not consistent
* Rerun: It is running right now in the line
* hash e823c1f431ddf6303c45418660071394234fb233 failed these tempest tests - https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master/13984cc/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
* If https://review.rdoproject.org/r/c/testproject/+/44342 fails with different tests we can skip and promote
* ^^ looks like it's running comon wallaby so added a few component rerun jobs to: https://review.rdoproject.org/r/c/testproject/+/36254
* **promoted**

## Downstream

* component:
* hit all the failing triggered jobs....
* here running all the failing component jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/421971 [DNM] test failing component jobs
* skip the rhos17-rhel8-sc010-network component job to promote the newtwok component : https://code.engineering.redhat.com/gerrit/c/testproject/+/422385 [DNM] Promote network component by skiping rhel8-rhos17-sc010-network job

---
## 2022-07-29

* rdo upstream:
* master: promoted yesterday
* fs001 is passing now : https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0
* wallaby:
* cs9: currently integration line is running
* sc010-kvm-internal passed: https://code.engineering.redhat.com/gerrit/c/testproject/+/421335
* skipping fs039 to promote
* **promoted today**
* cs8:
* fs001 is failing rekicked the job: https://review.rdoproject.org/r/c/testproject/+/37916
* skipping fs001 to promote
* **promoted today**
* cs8 train promoted yesterday

* RDO components
* common is failing in master, wallaby c9 (and 16.2)
* related bug: https://bugs.launchpad.net/tripleo/+bug/1982744/
* tripleo component is failing master and wallaby

* downstream:
* **all integration lines promoted today**
* rekicked the failing integration periodic jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/421970 [DNM] Test failing jobs
* rekicked the failing component jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/421971 [DNM] test failing component jobs

---
## Handover
- Downstream:
- Bugs which are still hitting us
- https://bugzilla.redhat.com/show_bug.cgi?id=2105408
- https://bugzilla.redhat.com/show_bug.cgi?id=2110535 ( We have workaround for this and Ronelle have patches up, may be will get merged before rr sync.)

* **NOTE**: all check OVB failing on node provision: https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&skip=0. Ananya, Bhagyashri pls check in it
* https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby&skip=0 has not passed in a long time