# Ruck Rover - 22 Apr 2022 - 28 Apr 2022

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/kPLwaUsGQieevaU10lIf_A
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

## Bugs from previous hackmd
* NEW CS9 compose breaking the downstream trigger jobs to upstream and ended up with NODE_FAILURE
* https://bugzilla.redhat.com/2077514

* RHEL9 OSP17 OVB jobs failing overcloud deployment at NTP sync stage
* https://bugzilla.redhat.com/2076756

* test_device_tagging compute tempest tests failing in master network component line
* https://bugs.launchpad.net/bugs/1969669 [duplicate]
* https://bugs.launchpad.net/tripleo/+bug/1968732

* related CIX: https://trello.com/c/EZmictvR/2456-cixlp1968732tripleociproa-testupdaterouteradminstate-test-failed-with-unable-to-connect-to-port-22

* all osp17 rhel9 jobs are failed with retry_limit having error "nothing provides network-scripts-openvswitch2.17 needed by rhosp-network-scripts-openvswitch-2.17-1.el9osttrunk.noarch"
* https://bugzilla.redhat.com/2076450

* tripleo-ci-centos-9-standalone-on-multinode-ipa is failing ping test
* https://bugs.launchpad.net/bugs/1968615

* jss-5.2.0-0.2.beta1 breaks freeipa setup (centos9)
* https://bugs.launchpad.net/tripleo/+bug/1969613/

### Updates/Notes:

#### For all branches periodic rdo side :
* https://bugs.launchpad.net/tripleo/+bug/1970710
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839694
* https://review.rdoproject.org/r/c/testproject/+/39848
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/839724

#### Release updates:
* master:
* Bug: https://bugs.launchpad.net/tripleo/+bug/1970400
* https://review.rdoproject.org/r/c/rdo-jobs/+/42318
* https://review.rdoproject.org/r/c/rdo-jobs/+/42101

* c8 wallaby:
* https://bugs.launchpad.net/tripleo/+bug/1970484
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839526
* https://review.rdoproject.org/r/c/testproject/+/38646

* victoria:
* https://bugs.launchpad.net/tripleo/+bug/1970736
* skiplist patch: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839721

#### Downstream

* rhos17-rhel9:
* promoted on 28th

* rhos16.2:
* Promoted on 25th \o/

* rhos17-rhel8:
* Promoted on 27th

## 28th April 2022 :

NOTES:
* (dviroel): Missing Wallaby C8 - failing due to timeout and tempest tests. We have skiplist for that, @marios up to you - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/42352

**For all branches on periodic rdo side** :
* https://bugs.launchpad.net/tripleo/+bug/1970710**
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839694
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/839724
* https://review.rdoproject.org/r/c/testproject/+/39848
#### Upstream

* check/gate:
* No failures so far

* RDO:
* master cs9: **PROMOTED**
* CS9 - OVB FS001 master job is failing on overcloud_node_provisioning Failed to connect to the host via ssh: https://bugs.launchpad.net/tripleo/+bug/1970400
* some progress with https://review.rdoproject.org/r/c/rdo-jobs/+/42101 but still failing.

* wallaby - c9 **PROMOTED** with depends on
* Need to merge:
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/839724
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839694

* wallaby - c8
* bug report for node provision issues: https://bugs.launchpad.net/tripleo/+bug/1970484
* it seems that we have some progress with:
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839526

* victoria - 8: **PROMOTED** with skiplist
* promoting some components to see if clear some failures
* OVB jobs still need to pass - infra is not helping us today
* https://review.rdoproject.org/r/c/testproject/+/35235
* chasing **cdc557839058d2ffa384e294674bc09d**
* train - c8:
* train - c7:

###### Components
* master - c9:
* missing network
* wallaby c8:
*
* wallaby c9:
* missing network
* victoria c8:
*
* train c8:
*

#### Downstream

* rhos17-rhel9:
* promoted on 28th -

* rhos16.2:
* Promoted on 25th \o/
*
* rhos17-rhel8:
* Promoted on 27th

* component lines status:

## 27th April 2022 :

NOTES:
* (dviroel): we had lots of timeouts on OVB jobs, maybe infra had a slow down - need to check again anf ping infra folks if needed

#### Upstream

* check/gate:
* No failures so far

* RDO:
* master cs9:
* CS9 - OVB FS001 master job is failing on overcloud_node_provisioning Failed to connect to the host via ssh: https://bugs.launchpad.net/tripleo/+bug/1970400
* https://bugs.launchpad.net/tripleo/+bug/1970554
* https://review.opendev.org/c/openstack/tripleo-ci/+/839472
* some progress with https://review.rdoproject.org/r/c/rdo-jobs/+/42101 but still failing.

* wallaby - c9
* image build failure on last run - https://bugs.launchpad.net/tripleo/+bug/1970554
* https://review.opendev.org/c/openstack/tripleo-ci/+/839472
* **604a4294308f260acaefd0d3d9d6e02d** missing only *periodic-tripleo-ci-centos-9-undercloud-containers-wallaby*

* wallaby - c8
* bug report for node provision issues: https://bugs.launchpad.net/tripleo/+bug/1970484
* still needs investifation on fs001 failures
* rlandy saw baremetal component 20d old, we force promotion of the following components:
* tripleo, network, baremetal
* all affected by https://github.com/openstack/tripleo-validations/commit/33c50072982fd535680e27a514af583ef2bd3325 - which is already on int line
* manually triggered a new int line with components updates
* it seems that we have some progress with:
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839526
*
* victoria - 8:
* promoting some components to see if clear some failures
* OVB jobs still need to pass - infra is not helping us today
* https://review.rdoproject.org/r/c/testproject/+/35235
* chasing **cdc557839058d2ffa384e294674bc09d**
* train - c8:
* train - c7: **promoted**

###### Components
* master - c9:
* network is lagging
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master (node provision)
* chasing security
* wallaby c8:
*
* wallaby c9:
* added more jobs to skiplist https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839608
* victoria c8:
*
* train c8:
*

#### Downstream

* rhos17-rhel9:
* promoted on 25th - current run looks clean waiting for in progress jobs run result
* rhos16.2:
* Promoted on 25th \o/
* bm_envD-3ctlr_1comp-featureset035-rhos-16.2 is failing all other jobs are green re-running https://code.engineering.redhat.com/gerrit/c/testproject/+/315285
* rhos17-rhel8:
* Promoted on 27th

* component lines status:

## 26th April 2022 :

NOTES:
* (dviroel): bhagyashri can you check downstream components on your time? Thanks
* (dviroel): bhagyashri can you check master and wallaby c9 image builds? failed in last run: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-9-buildimage-overcloud-full-wallaby&job_name=periodic-tripleo-centos-9-buildimage-overcloud-full-master

#### Upstream

* check/gate:
* No failures so far

* RDO:
* master cs9: promoted
* CS9 - OVB FS001 master job is failing on overcloud_node_provisioning Failed to connect to the host via ssh: https://bugs.launchpad.net/tripleo/+bug/1970400
* Error: problem with installed package pki-java-11.2.0-0.2.beta1.el9.noarch requires jss >= 5.2.0, but none of the providers can be installed failing on Deploy FreeIPA (fs039 and multinode ipa): https://bugs.launchpad.net/tripleo/+bug/1970406
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/839333
* fs035,fs039 and fs064 failing recently re-runnong those jobs: https://review.rdoproject.org/r/c/testproject/+/39932
* same thing on new hash, still hard to file bugs, different errors on each run
* New bug reported against FS039 and FS064:
* https://bugs.launchpad.net/tripleo/+bug/1970262 - didn't see in newest hash yet
* **4e6773f57d1594b550fde23014953a7d promoted** \o/
* wallaby - c9
* image build failure on last run - issue?

* wallaby - c8
* bug report for node provision issues: https://bugs.launchpad.net/tripleo/+bug/1970484
* still needs investifation on fs001 failures
* rlandy saw baremetal component 20d old, we force promotion of the following components:
* tripleo, network, baremetal
* all affected by https://github.com/openstack/tripleo-validations/commit/33c50072982fd535680e27a514af583ef2bd3325 - which is already on int line
* manually triggered a new int line with components updates
* victoria - 8:
* promoting some components to see if clear some failures

* train - c8: promoted
* missing fs035 - https://review.rdoproject.org/r/c/testproject/+/36356
* promoted today

###### Components
* master - c9:
* missing security (jss issue needs to merge first)
* missing network - only fs001 failing - needs investigation
* wallaby c8:
* tripleo, network, baremetal promoted skipping fs001
* wallaby c9:
* rerunning: network, tripleo, manila
* victoria c8:
* ok - promoted few components, waiting new int line run
* train c8:
* ok

#### Downstream

* rhos17-rhel9:
* promoted on 25th
* rhos16.2:
* Promoted on 25th \o/
* rhos17-rhel8:
* Promoted on 26th

* component lines status:

## 25th April 2022 :

#### Upstream

* check/gate:
* No failures so far

* RDO:
* master cs9 :
* ovb jobs are failing inconsistent with inconsistant issue
* same thing on new hash, still hard to file bugs, different errors on each run
* New bug reported against FS039 and FS064:
* https://bugs.launchpad.net/tripleo/+bug/1970262 - didn't see in newest hash yet
* c8 train:
* ~~fs001 is failing due to tempest test failure and fs035 timed_out will re-run the job : https://review.rdoproject.org/r/c/testproject/+/32107~~
* missing fs035 - https://review.rdoproject.org/r/c/testproject/+/36356
* wallaby - c8
* fs001 and fs035 are failing a lot
* fs035 last failure: node provision

#### Downstream

* rhos17-rhel9:
* fs020 and standalone-on-multinode-ipa-rhos-17 failing - re-running job: https://code.engineering.redhat.com/gerrit/c/testproject/+/400570
* promoted on 25th
* rhos16.2: fs001 and fs035 failing : not consistent failure re-runnong: https://code.engineering.redhat.com/gerrit/c/testproject/+/315285
* Promoted on 25th \o/
* rhos17-rhel8:
* missing fs035 and full-tempest-scenario:
* https://code.engineering.redhat.com/gerrit/c/testproject/+/310670 - full-tempest-scenario

* component lines status:
* periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-tempest-rhos-17 - Randomly failing tempest test: -https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-full-tempest-scenario-tempest-rhos-17&skip=0
* periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-tempest-rhos-17 - Randomly failing tempest test: - https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-tempest-rhos-17&skip=0
* For rhos16.2 - Most of the network component jobs were failed but the histroy of all the failed jobs is good so re-running those jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/404474
* For rhso17-rhel9: No blockers

## 22nd April 2022 :

#### Upstream

* free ipa failure (fs064 and fs039 master and wallaby cs9 + multinode): https://bugs.launchpad.net/tripleo/+bug/1969613/ and fix for that is here ~~https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/838996~~ - workaround merged

* node provisioning failure in ovb third party jobs - https://logserver.rdoproject.org/90/838990/1/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001/5cad192/logs/baremetal_1_19542_0-console.log
https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001&project=openstack/tripleo-quickstart
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-undercloud-containers&job_name=tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates&job_name=tripleo-ci-centos-9-scenario007-multinode-oooq-container&skip=0
~~~
<chandankumar> ysandeep: https://logserver.rdoproject.org/96/838996/2/openstack-check/tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby/23c6774/logs/undercloud/home/zuul/undercloud_install.log.txt.gz
<chandankumar> ysandeep: i think https://review.opendev.org/c/openstack/python-tripleoclient/+/831425 caused it
<chandankumar> bhagyashris|ruck: pojadhav ^^ need a bug
<chandankumar> or may be we need a promotion https://review.opendev.org/c/openstack/tripleo-validations/+/831424/
<chandankumar> for tripleo-validations
<chandankumar> we need the promotion of tripleo-component https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L1192
<bhagyashris|ruck> chandankumar, ack
~~~

* master cs9 full tempest api and scenario jobs are failing with bunch of tempest test failure -
* good results on rerun: https://review.rdoproject.org/r/c/testproject/+/38646/25#message-8670c8e911fb08a1c0ce35797b82006b5d22c9db

##### Components
* *.featureset001-*-wallaby: will fail on until we promoted wallaby C9, since it needs https://review.opendev.org/c/openstack/tripleo-validations/+/831424/
* New bug [network]: https://bugs.launchpad.net/tripleo/+bug/1969985
* periodic-tripleo-ci-centos-9-scenario007-standalone-network-master
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master
* **We need master promotions to update containers with new pyroute2.**

#### Downstream

* rhos17 on rhel 9
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?pipeline=openstack-periodic-integration-rhos-17-rhel9&skip=0
* except baremetal fs001 job all are green
* testproject patch for bm-fs001 : https://code.engineering.redhat.com/gerrit/c/testproject/+/400570

* rhos 16.2 on rhel8
* few of the jobs failed due to node failure rerunning those here: https://code.engineering.redhat.com/gerrit/c/testproject/+/315285