# Ruck Rover 2022-11-18 - 2022-11-24
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Vn_zjslARMG5Crs4YwnSqw

##### ruck & rover: sandeep & ronelle

:::info

### Active Bugs

* ~~Reopened: https://bugs.launchpad.net/tripleo/+bug/1978997 tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis - [wallaby only]~~
* https://review.opendev.org/c/openstack/tripleo-common/+/865402

* ~~https://bugs.launchpad.net/tripleo/+bug/1997670 Master container build jobs failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_buildah_login.yml~~
* Fix: https://review.rdoproject.org/r/c/config/+/46197
* Rreunning container build here: https://review.rdoproject.org/r/c/testproject/+/42657

* https://bugs.launchpad.net/tripleo/+bug/1997664 Scenario 001 master failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_service
* Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/865503

* ~~https://bugs.launchpad.net/tripleo/+bug/1997202 Master/ Zed/Wallaby container build jobs are failing with error "Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org"~~
* Awaiting Wallaby cherry-pick to merge: https://review.opendev.org/c/openstack/tripleo-common/+/865397
* Need https://review.opendev.org/c/openstack/tripleo-common/+/865402 first

* Container-build job not building rpms for Depends-on patches. https://bugs.launchpad.net/tripleo/+bug/1997539

* ~~https://bugs.launchpad.net/tripleo/+bug/1996684 - FreeIPA failed to connect LDAP server - REOPENED as hitting fs039 and fs064 - both on master and zed. Ping out on #sec-interop. There is a diff ipa-server-install command on multinode-ipa that never fails. TO DO: ask Ade what the diff is. Raising this as CIX~~
Master fs039 is failing deploy - but at a diff place each time
- https://review.rdoproject.org/r/c/config/+/46173 - patch looks to help.

* ~~[Reopened]https://bugs.launchpad.net/tripleo/+bug/1996612 ansible-2.14: content-provider are broken~~
* https://review.opendev.org/c/openstack/tripleo-ci/+/865540/ - fix is up
* pinned with tripleo-quickstart/+/864498 (tripleo/+bug/1996482)
- https://bugs.launchpad.net/tripleo/+bug/1996482: Undercloud Install: 'Task' object has no attribute '_valid_attrs'
- ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/864392 - This needs to get merged ASAP to unblock gate.~~
- ~~[Merged] going with this for now https://review.opendev.org/c/openstack/tripleo-quickstart/+/864498 Pin ansible-core until we can fix compatibility with 2.14~~
-
- ~~https://bugs.launchpad.net/bugs/1996668 [NEW] openstack-tox-pep8 consistently fails because it fails to install flake8~~
- see fixes from Takashi

~~- https://bugs.launchpad.net/tripleo/+bug/1996264 Invalid argument: disk vda does not have an active block job~~

~~- https://bugs.launchpad.net/tripleo/+bug/1996684 : FreeIPA server failed to connect LDAP server~~
~~- https://bugs.launchpad.net/tripleo/+bug/1996686 : Mixed OS job failing with package dependency~~

- ~~https://bugzilla.redhat.com/show_bug.cgi?id=2143562 container build push jobs are failing with 'error while loading shared libraries: libc.so.6: cannot change memory protections'~~
* https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435653

- ~~https://bugzilla.redhat.com/show_bug.cgi?id=2143572 16.2/17.1/17 rhel9/rhel8 components are failing with error 'failed to detect a valid IP address from '- 10.11.5.160 - 10.2.70.215''~~

- ~~https://bugs.launchpad.net/tripleo/+bug/1996874 Buld-containers-quay-push: libselinux-python aren't installed~~
DNM test patch: https://review.opendev.org/c/openstack/python-tripleoclient/+/864952

Fixes are up:-
* https://review.rdoproject.org/r/c/config/+/46139
* https://review.rdoproject.org/r/c/rdo-jobs/+/46141

- https://bugs.launchpad.net/tripleo/+bug/1996917 openstack-tox: ERROR: Cannot install cryptography<37.0.0

https://review.opendev.org/c/openstack/tripleo-ansible/+/864999
https://review.opendev.org/c/openstack/tripleo-ansible/+/864994
https://review.opendev.org/c/openstack/tripleo-ansible/+/864906

- ~~https://bugs.launchpad.net/tripleo/+bug/1996933
tripleo-ci-centos-9-undercloud-upgrade is failing undercloud upgrade - No such file or directory: '/home/zuul/tripleo-deploy/undercloud/tripleo-heat-installer-templates/environments/deployed-server-noop-ctlplane.yaml'~~
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/864700 (revert)
* Issue solved itself - revert not needed anymore.

:::

### Nov 24

Gate:

No new failure for content provider for master/zed
wallaby: https://review.opendev.org/c/openstack/tripleo-common/+/865397

### New bugs:
* https://bugs.launchpad.net/tripleo/+bug/1997664
Scenario 001 master failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_service
* https://bugs.launchpad.net/tripleo/+bug/1997670 Master container build jobs failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_buildah_login.ym
* Reopened: https://bugs.launchpad.net/tripleo/+bug/1978997 tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis - [wallaby only]
* https://review.opendev.org/c/openstack/tripleo-common/+/865402

#### Upstream promotion pipelines
- Master - 23rd Nov
- fs064
- fs39
- fs002
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
- Zed - 23rd Nov
- fs001
- full-tempest-api
- Running here: https://review.rdoproject.org/r/c/testproject/+/38348
- C9 Wallaby - 22nd Nov (No new hash)
- C8 wallaby - 23rd Nov (No new hash)
- C8 Train - 22nd Nov (Line running right now - will check in few hours)
#### Upstream Component pipelines
- Master
- baremetal - ovb
- network - ovb/sc07
- tempest - tempest-scenario
- tripleo - ovb/multinode-ipa
- ui - standalone
- validation ovb

- Zed
- tripleo - ovb/standalone
- validation - ovb
- C9 wallaby
- tripleo
- ovb
- validation
- ovb
- C8 wallaby - All green
- C8 train - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537
#### Downstream promotion pipelines
- 17/9 - 22nd Nov (No new hash)
- fs035 (Rerunning)
- tempest: https://sf.hosted.upshift.rdu2.redhat.com/logs/99/430999/25/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17/1045ca6/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- 17.1/9 - 22nd Nov (No new hash)
- fs035 (Rerunning)
- tempest: https://sf.hosted.upshift.rdu2.redhat.com/logs/99/430999/25/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1/abb4c93/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- 17.1/8 - 22nd Nov (No new hash)
- 16.2/8 - 23rd Nov (No new hash)
#### Downstream component promotion pipelines
17/9 - All green
17.1/9 - All green
17.1/8 - No job in criteria
16.2/8 - All green

### Nov 23

#### New Bug
* Container-build job not building rpms for Depends-on patches. https://bugs.launchpad.net/tripleo/+bug/1997539

#### Gate
* https://review.opendev.org/c/openstack/tripleo-validations/+/865170/ - content-provider failed with dns issue
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/862973/ - content-provider failed with dns issue
* https://review.opendev.org/c/openstack/tripleo-ansible/+/864468/ - ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/containers/podman/versions/1.9.0/': The read operation timed out
* rekicked looks transient
#### Upstream promotion pipelines
- Master - 23nd Nov
- Latest hash missing fs039
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420

- Zed - 23nd Nov
- Tracking hash: 9ecd6d8c6b2caff2be435d87b506888d
- fs039 missing
- Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/38348

- C9 Wallaby - 23nd
- C8 Wallaby - 22nd - (All green - No new Hash)
- C8 Train - 22nd
- Rechecking fs020 - missing

#### Upstream Component pipelines
- Master
- network - ovb(tempest)
- https://logserver.rdoproject.org/37/28537/59/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/e35a106/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- ~~tempest scenario000-multinode-oooq-container-updates~~
- ~~validation - ovb~~

- Zed
- ~~network - ovb(tempest)~~
- https://logserver.rdoproject.org/37/28537/59/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-zed/a197150/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- ~~tripleo - ovb~~

- C9 wallaby - All green
- C8 wallaby - All green
- Train - All green

Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/28537

#### Downstream promotion pipelines
- 17/9 - 22nd - (All green - No new Hash)
- 17.1/9 - 22nd - (All green - No new Hash)
- 17.1/8 - 22nd - (All green - No new Hash)
- 16.2/8 - 23rd
- Line running right now
- ~~full-tempest-api failed on tempest~~
- rerunning: ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/211643~~

#### Downstream component pipelines
* 17/9 - All green
* 17.1/9 - All green
* 17.1/8 - No jobs in criteria
* 16.2/8 - All green

### Nov 22

#### Gate
- Some failures but no pattern
- content-provider - still some failures, 13 so far
- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-content-provider&job_name=tripleo-ci-centos-9-content-provider-zed&job_name=tripleo-ci-centos-9-content-provider-wallaby&result=FAILURE&skip=0
#### Upstream promotion pipelines

- Master - 22/11
- Tracking below hash
- rr --release master --aggregate_hash tripleo-ci-testing/7f/83/7f83ad06cca0225723adb4f189cf05ee
- fs039 https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/920b882/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420

**Please revert below Patches once master promotes**

https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45892
https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45891

- Zed - 22/11
- latest hash on fs064
- https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-zed/7648fa0/logs/supplemental/home/cloud-user/deploy_freeipa.log.txt.gz
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/25325
- Wallaby C9 - 22/11
- fs039
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/25325
- Wallaby C8 - 22/11 (No new Hash)
- Train - 22/11
- fs020(Rerunning): https://review.rdoproject.org/r/c/testproject/+/42657
#### Upstream Component pipelines
* Master
* ~~common - sc01~~
* ~~compute - tempest-api/scenario~~
* tripleo - most of jobs failed(mirror/random issues)
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/38348
* Zed
* network - ovb
* tripleo - ovb
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/38348
* C9 wallaby - Green
* C8 wallaby - Green
* C8 Train - Green

#### Downstream promotion pipelines
- RHEL9 17 - promoted 22/11
~~- full-tempest-api [Re-running]~~(https://code.engineering.redhat.com/gerrit/c/testproject/+/211643)
- RHEL9 17.1 - promoted on 22/11
- ~~Jenkins jobs failed - Couldn't fetch dlrn repo~~
- ~~Retriggered via jenkins~~
- RHEL8 16.2 - promoted on 20/11 (No new Hash - Green)
- RHEL8 17.1 - promoted on 21/11 (No new Hash - Green)

#### Downstream component pipelines

### Nov 21
#### Upstream
- Gate:
- Content-provider failing due to mirror issue. (Intermittent) - https://bugs.launchpad.net/tripleo/+bug/1997202
- https://review.opendev.org/c/openstack/tripleo-ansible/+/864994/
- master : Promoted on 15/11
- Container build failing with a new bug
- https://bugs.launchpad.net/tripleo/+bug/1997202
- Tracking good hash from 18th Nov, only fs064 left
- rr --release master --aggregate_hash tripleo-ci-testing/1d/8e/1d8e37b0de13dabf2789047ad17e4cea
- Rerun failed with https://logserver.rdoproject.org/20/39420/12/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/23f2a17/logs/supplemental/var/log/ipaserver-install.log.txt.gz
- python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release master --distro centos-9 --aggregate_hash tripleo-ci-testing/1d/8e/1d8e37b0de13dabf2789047ad17e4cea

- Zed : Promoted on 18/11
- Container build failing with a new bug
- https://bugs.launchpad.net/tripleo/+bug/1997202
- Wallaby-c9: Promoted on 12/11
- Container build failing with a new bug
- https://bugs.launchpad.net/tripleo/+bug/1997202
- Tracking good hash from 19th Nov, only fs064 left
- rr --release wallaby --aggregate_hash tripleo-ci-testing/86/9e/869e5864e17d9d54ca8cbc34666ca813
- python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release wallaby --distro centos-9 --aggregate_hash tripleo-ci-testing/da/0d/da0d01de4b96e54689cd18f509306f11

- Wallaby-c8: Promoted on 21/11
- Train: Promoted on 19/11 (No new hash)

#### Upstream components:
- Master component
- client - ovb(Rerunning)
- ~~network - ovb(Rerunning)~~
- tripleo - ovb(Rerunning)
- Zed components
- baremetal - ovb (Rerunning)
- network - ovb(Rerunning)
- tripleo - ovb (Rerunning)
- wallaby c9 component - All green
- wallaby c8 component - All green
- train c8 component
- network - ovb (Rerunning)

### Testprojects
* ~~17/9 sc001 https://code.engineering.redhat.com/gerrit/c/testproject/+/398540~~
* ~~17/9 sc001 https://code.engineering.redhat.com/gerrit/c/testproject/+/398540~~
* osp-trunk.hosted.upshift.rdu2.redhat.com was inaccesible in last run
* ~~17.1/9 tripleo component https://code.engineering.redhat.com/gerrit/c/testproject/+/211643~~
* Component jobs ovb rerun
* https://review.rdoproject.org/r/c/testproject/+/38348
* fs064 master hash from 18th
* https://review.rdoproject.org/r/c/testproject/+/39420
* fs064 wallaby hash from 19th
* https://review.rdoproject.org/r/c/testproject/+/25325

#### Downstream
- RHEL9 17 - promoted 21/11
- ~~sc001 (In rerun)~~
- RHEL9 17.1 - promoted on 21/11
- ~~sc007 (In rerun)~~
- RHEL8 16.2 - promoted on 20/11 (No new Hash - Green)
- RHEL8 17.1 - promoted on 20/11 (No new Hash - Green)

#### Downstream components:
- RHEL8 16.2 component - All green
- RHEL8 17.1 component - No jobs in criteria
- RHEL9 17.1 component
- ~~tripleo - ovb(In rerun)~~
- ~~Two nodes didn't provision~~
- RHEL9 17 - All green

---
### Nov 18
#### Upstream
- master : Promoted on 15/11
- Zed : Promoted on 18/11
- Wallaby-c9: Promoted on 15/11
- Wallaby-c8: Promoted on 18/11
- Train: Promoted on 18/11

### Testprojects
* fs064 is failing - will try change to deploy IPA to remove --auto-reverse

#### Downstream
- RHEL9 17 - promoted on 16/11 (no new hash)
- RHEL9 17.1 - promoted on 15/11 (RBAC job in rerun)
- RHEL8 16.2 - promoted on 18/11
- RHEL8 17.1 - promoted on 16/11 (will be shutdown)

##### Downstream components:

---

### Nov 17
#### Upstream
- master : Promoted on 15/11
- Zed : Promoted on 16/11
- Wallaby-c9: Promoted on 15/11
- Wallaby-c8: Promoted on 17/11
- Train: Promoted on 15/11

### Testprojects
* https://review.rdoproject.org/r/c/testproject/+/36254 - wallaby, train reruns

#### Downstream
- RHEL9 17 - promoted on 16/11
- RHEL9 17.1 - promoted on 15/11
- RHEL8 16.2 - promoted on 14/11
- RHEL8 17.1 - promoted on 16/11

##### Downstream components:
- RHEL 9 17/17.1: baremetal/clients/tripleo/network
- blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2143572
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
- RHEL 8 16.2: https://code.engineering.redhat.com/gerrit/c/testproject/+/430517