Ruck Rover 2022-11-18 - 2022-11-24

RDO Cockpit / RHOS Cockpit RDO Promoter / RHOS Promoter

Zuul Status:

Review List

tags: `ruck_rover`

Previous RR notes: https://hackmd.io/Vn_zjslARMG5Crs4YwnSqw

ruck & rover: sandeep & ronelle

Active Bugs

~~Reopened: https://bugs.launchpad.net/tripleo/+bug/1978997 tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis - [wallaby only]~~
- https://review.opendev.org/c/openstack/tripleo-common/+/865402
~~https://bugs.launchpad.net/tripleo/+bug/1997670 Master container build jobs failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_buildah_login.yml~~
- Fix: https://review.rdoproject.org/r/c/config/+/46197
- Rreunning container build here: https://review.rdoproject.org/r/c/testproject/+/42657
https://bugs.launchpad.net/tripleo/+bug/1997664 Scenario 001 master failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_service
- Fix: https://review.opendev.org/c/openstack/tripleo-ansible/+/865503
~~https://bugs.launchpad.net/tripleo/+bug/1997202 Master/ Zed/Wallaby container build jobs are failing with error "Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org"~~
- Awaiting Wallaby cherry-pick to merge: https://review.opendev.org/c/openstack/tripleo-common/+/865397
- Need https://review.opendev.org/c/openstack/tripleo-common/+/865402 first
Container-build job not building rpms for Depends-on patches. https://bugs.launchpad.net/tripleo/+bug/1997539
https://bugs.launchpad.net/tripleo/+bug/1996684 - FreeIPA failed to connect LDAP server - REOPENED as hitting fs039 and fs064 - both on master and zed. Ping out on #sec-interop. There is a diff ipa-server-install command on multinode-ipa that never fails. TO DO: ask Ade what the diff is. Raising this as CIX Master fs039 is failing deploy - but at a diff place each time
- https://review.rdoproject.org/r/c/config/+/46173 - patch looks to help.
~~[Reopened]https://bugs.launchpad.net/tripleo/+bug/1996612 ansible-2.14: content-provider are broken~~
- https://review.opendev.org/c/openstack/tripleo-ci/+/865540/ - fix is up
- pinned with tripleo-quickstart/+/864498 (tripleo/+bug/1996482)

https://bugs.launchpad.net/tripleo/+bug/1996482: Undercloud Install: 'Task' object has no attribute '_valid_attrs'
- ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/864392 - This needs to get merged ASAP to unblock gate.~~
- ~~[Merged] going with this for now https://review.opendev.org/c/openstack/tripleo-quickstart/+/864498 Pin ansible-core until we can fix compatibility with 2.14~~
~~https://bugs.launchpad.net/bugs/1996668 [NEW] openstack-tox-pep8 consistently fails because it fails to install flake8~~
- see fixes from Takashi

~~- https://bugs.launchpad.net/tripleo/+bug/1996264 Invalid argument: disk vda does not have an active block job~~

~~- https://bugs.launchpad.net/tripleo/+bug/1996684 : FreeIPA server failed to connect LDAP server~~ ~~- https://bugs.launchpad.net/tripleo/+bug/1996686 : Mixed OS job failing with package dependency~~

~~https://bugzilla.redhat.com/show_bug.cgi?id=2143562 container build push jobs are failing with 'error while loading shared libraries: libc.so.6: cannot change memory protections'~~
- https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/435653
~~https://bugzilla.redhat.com/show_bug.cgi?id=2143572 16.2/17.1/17 rhel9/rhel8 components are failing with error 'failed to detect a valid IP address from '- 10.11.5.160 - 10.2.70.215''~~
~~https://bugs.launchpad.net/tripleo/+bug/1996874 Buld-containers-quay-push: libselinux-python aren't installed~~ DNM test patch: https://review.opendev.org/c/openstack/python-tripleoclient/+/864952

Fixes are up:-
- https://review.rdoproject.org/r/c/config/+/46139
- https://review.rdoproject.org/r/c/rdo-jobs/+/46141
https://bugs.launchpad.net/tripleo/+bug/1996917 openstack-tox: ERROR: Cannot install cryptography<37.0.0

https://review.opendev.org/c/openstack/tripleo-ansible/+/864999 https://review.opendev.org/c/openstack/tripleo-ansible/+/864994 https://review.opendev.org/c/openstack/tripleo-ansible/+/864906
https://bugs.launchpad.net/tripleo/+bug/1996933 tripleo-ci-centos-9-undercloud-upgrade is failing undercloud upgrade - No such file or directory: '/home/zuul/tripleo-deploy/undercloud/tripleo-heat-installer-templates/environments/deployed-server-noop-ctlplane.yaml'
- https://review.opendev.org/c/openstack/tripleo-heat-templates/+/864700 (revert)
- Issue solved itself - revert not needed anymore.

Nov 24

Gate:

No new failure for content provider for master/zed wallaby: https://review.opendev.org/c/openstack/tripleo-common/+/865397

New bugs:

https://bugs.launchpad.net/tripleo/+bug/1997664 Scenario 001 master failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_service
https://bugs.launchpad.net/tripleo/+bug/1997670 Master container build jobs failing with ERROR! Could not find specified file in role: tasks/tripleo_podman_buildah_login.ym
Reopened: https://bugs.launchpad.net/tripleo/+bug/1978997 tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis - [wallaby only]
- https://review.opendev.org/c/openstack/tripleo-common/+/865402

Upstream promotion pipelines

Master - 23rd Nov
- fs064
- fs39
- fs002
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
Zed - 23rd Nov
- fs001
- full-tempest-api
- Running here: https://review.rdoproject.org/r/c/testproject/+/38348
C9 Wallaby - 22nd Nov (No new hash)
C8 wallaby - 23rd Nov (No new hash)
C8 Train - 22nd Nov (Line running right now - will check in few hours)

Upstream Component pipelines

Master
- baremetal - ovb
- network - ovb/sc07
- tempest - tempest-scenario
- tripleo - ovb/multinode-ipa
- ui - standalone
- validation ovb
Zed
- tripleo - ovb/standalone
- validation - ovb
C9 wallaby
- tripleo
  - ovb
- validation
  - ovb
C8 wallaby - All green
C8 train - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537

Downstream promotion pipelines

17/9 - 22nd Nov (No new hash)
- fs035 (Rerunning)
  - tempest: https://sf.hosted.upshift.rdu2.redhat.com/logs/99/430999/25/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17/1045ca6/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
17.1/9 - 22nd Nov (No new hash)
- fs035 (Rerunning)
  - tempest: https://sf.hosted.upshift.rdu2.redhat.com/logs/99/430999/25/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1/abb4c93/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
17.1/8 - 22nd Nov (No new hash)
16.2/8 - 23rd Nov (No new hash)

Downstream component promotion pipelines

17/9 - All green 17.1/9 - All green 17.1/8 - No job in criteria 16.2/8 - All green

Nov 23

New Bug

Container-build job not building rpms for Depends-on patches. https://bugs.launchpad.net/tripleo/+bug/1997539

Gate

https://review.opendev.org/c/openstack/tripleo-validations/+/865170/ - content-provider failed with dns issue
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/862973/ - content-provider failed with dns issue
https://review.opendev.org/c/openstack/tripleo-ansible/+/864468/ - ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/containers/podman/versions/1.9.0/': The read operation timed out
- rekicked looks transient

Upstream promotion pipelines

Master - 23nd Nov
- Latest hash missing fs039
  - Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
Zed - 23nd Nov
- Tracking hash: 9ecd6d8c6b2caff2be435d87b506888d
- fs039 missing
- Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/38348
C9 Wallaby - 23nd
C8 Wallaby - 22nd - (All green - No new Hash)
C8 Train - 22nd
- Rechecking fs020 - missing

Upstream Component pipelines

Master
- network - ovb(tempest)
  - https://logserver.rdoproject.org/37/28537/59/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/e35a106/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- ~~tempest scenario000-multinode-oooq-container-updates~~
- ~~validation - ovb~~
Zed
- ~~network - ovb(tempest)~~
  - https://logserver.rdoproject.org/37/28537/59/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-zed/a197150/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- ~~tripleo - ovb~~
C9 wallaby - All green
C8 wallaby - All green
Train - All green

Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/28537

Downstream promotion pipelines

17/9 - 22nd - (All green - No new Hash)
17.1/9 - 22nd - (All green - No new Hash)
17.1/8 - 22nd - (All green - No new Hash)
16.2/8 - 23rd
- Line running right now
- ~~full-tempest-api failed on tempest~~
  - rerunning: ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/211643~~

Downstream component pipelines

17/9 - All green
17.1/9 - All green
17.1/8 - No jobs in criteria
16.2/8 - All green

Nov 22

Gate

Some failures but no pattern
content-provider - still some failures, 13 so far
- https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-content-provider&job_name=tripleo-ci-centos-9-content-provider-zed&job_name=tripleo-ci-centos-9-content-provider-wallaby&result=FAILURE&skip=0

Upstream promotion pipelines

Master - 22/11
- Tracking below hash
  - rr –release master –aggregate_hash tripleo-ci-testing/7f/83/7f83ad06cca0225723adb4f189cf05ee
  - fs039 https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/920b882/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
  - Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420

Please revert below Patches once master promotes

https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45892 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45891

Zed - 22/11
Wallaby C9 - 22/11
- fs039
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/25325
Wallaby C8 - 22/11 (No new Hash)
Train - 22/11
- fs020(Rerunning): https://review.rdoproject.org/r/c/testproject/+/42657

Upstream Component pipelines

Master
- ~~common - sc01~~
- ~~compute - tempest-api/scenario~~
- tripleo - most of jobs failed(mirror/random issues)
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/38348
Zed
- network - ovb
- tripleo - ovb
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/38348
C9 wallaby - Green
C8 wallaby - Green
C8 Train - Green

Downstream promotion pipelines

RHEL9 17 - promoted 22/11 ~~- full-tempest-api [Re-running]~~(https://code.engineering.redhat.com/gerrit/c/testproject/+/211643)
RHEL9 17.1 - promoted on 22/11
- ~~Jenkins jobs failed - Couldn't fetch dlrn repo~~
- ~~Retriggered via jenkins~~
RHEL8 16.2 - promoted on 20/11 (No new Hash - Green)
RHEL8 17.1 - promoted on 21/11 (No new Hash - Green)

Downstream component pipelines

Nov 21

Upstream

Gate:
- Content-provider failing due to mirror issue. (Intermittent) - https://bugs.launchpad.net/tripleo/+bug/1997202
  - https://review.opendev.org/c/openstack/tripleo-ansible/+/864994/
master : Promoted on 15/11
- Container build failing with a new bug
  - https://bugs.launchpad.net/tripleo/+bug/1997202
- Tracking good hash from 18th Nov, only fs064 left
  - rr –release master –aggregate_hash tripleo-ci-testing/1d/8e/1d8e37b0de13dabf2789047ad17e4cea
  - Rerun failed with https://logserver.rdoproject.org/20/39420/12/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master/23f2a17/logs/supplemental/var/log/ipaserver-install.log.txt.gz
  - python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py –release master –distro centos-9 –aggregate_hash tripleo-ci-testing/1d/8e/1d8e37b0de13dabf2789047ad17e4cea
Zed : Promoted on 18/11
- Container build failing with a new bug
  - https://bugs.launchpad.net/tripleo/+bug/1997202
Wallaby-c9: Promoted on 12/11
- Container build failing with a new bug
  - https://bugs.launchpad.net/tripleo/+bug/1997202
- Tracking good hash from 19th Nov, only fs064 left
  - rr –release wallaby –aggregate_hash tripleo-ci-testing/86/9e/869e5864e17d9d54ca8cbc34666ca813
- python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py –release wallaby –distro centos-9 –aggregate_hash tripleo-ci-testing/da/0d/da0d01de4b96e54689cd18f509306f11
Wallaby-c8: Promoted on 21/11
Train: Promoted on 19/11 (No new hash)

Upstream components:

Master component
- client - ovb(Rerunning)
- ~~network - ovb(Rerunning)~~
- tripleo - ovb(Rerunning)
Zed components
- baremetal - ovb (Rerunning)
- network - ovb(Rerunning)
- tripleo - ovb (Rerunning)
wallaby c9 component - All green
wallaby c8 component - All green
train c8 component
- network - ovb (Rerunning)

Testprojects

~~17/9 sc001 https://code.engineering.redhat.com/gerrit/c/testproject/+/398540~~
~~17/9 sc001 https://code.engineering.redhat.com/gerrit/c/testproject/+/398540~~
- osp-trunk.hosted.upshift.rdu2.redhat.com was inaccesible in last run
~~17.1/9 tripleo component https://code.engineering.redhat.com/gerrit/c/testproject/+/211643~~
Component jobs ovb rerun
- https://review.rdoproject.org/r/c/testproject/+/38348
fs064 master hash from 18th
- https://review.rdoproject.org/r/c/testproject/+/39420
fs064 wallaby hash from 19th
- https://review.rdoproject.org/r/c/testproject/+/25325

Downstream

RHEL9 17 - promoted 21/11
- ~~sc001 (In rerun)~~
RHEL9 17.1 - promoted on 21/11
- ~~sc007 (In rerun)~~
RHEL8 16.2 - promoted on 20/11 (No new Hash - Green)
RHEL8 17.1 - promoted on 20/11 (No new Hash - Green)

Downstream components:

RHEL8 16.2 component - All green
RHEL8 17.1 component - No jobs in criteria
RHEL9 17.1 component
- ~~tripleo - ovb(In rerun)~~
  - ~~Two nodes didn't provision~~
RHEL9 17 - All green

Nov 18

Upstream

master : Promoted on 15/11
Zed : Promoted on 18/11
Wallaby-c9: Promoted on 15/11
Wallaby-c8: Promoted on 18/11
Train: Promoted on 18/11

Testprojects

fs064 is failing - will try change to deploy IPA to remove –auto-reverse

Downstream

RHEL9 17 - promoted on 16/11 (no new hash)
RHEL9 17.1 - promoted on 15/11 (RBAC job in rerun)
RHEL8 16.2 - promoted on 18/11
RHEL8 17.1 - promoted on 16/11 (will be shutdown)

Downstream components:

Nov 17

Upstream

master : Promoted on 15/11
Zed : Promoted on 16/11
Wallaby-c9: Promoted on 15/11
Wallaby-c8: Promoted on 17/11
Train: Promoted on 15/11

Testprojects

https://review.rdoproject.org/r/c/testproject/+/36254 - wallaby, train reruns

Downstream

RHEL9 17 - promoted on 16/11
RHEL9 17.1 - promoted on 15/11
RHEL8 16.2 - promoted on 14/11
RHEL8 17.1 - promoted on 16/11

Downstream components:

RHEL 9 17/17.1: baremetal/clients/tripleo/network
- blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2143572
- https://code.engineering.redhat.com/gerrit/c/testproject/+/430523
RHEL 8 16.2: https://code.engineering.redhat.com/gerrit/c/testproject/+/430517

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2022-11-18 - 2022-11-24

tags: ruck_rover

Previous RR notes: https://hackmd.io/Vn_zjslARMG5Crs4YwnSqw

ruck & rover: sandeep & ronelle

Active Bugs

Nov 24

New bugs:

Upstream promotion pipelines

Upstream Component pipelines

Downstream promotion pipelines

Downstream component promotion pipelines

Nov 23

New Bug

Gate

Upstream promotion pipelines

Upstream Component pipelines

Downstream promotion pipelines

Downstream component pipelines

Nov 22

Gate

Upstream promotion pipelines

Upstream Component pipelines

Downstream promotion pipelines

Downstream component pipelines

Nov 21

Upstream

Upstream components:

Testprojects

Downstream

Downstream components:

Nov 18

Upstream

Testprojects

Downstream

Downstream components:

Nov 17

Upstream

Testprojects

Downstream

Downstream components:

tags: `ruck_rover`