# Ruck Rover 2023-02-03 - 2023-02-09
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/sacADiWwRO64_CzBHUFwiA

##### ruck & rover: ysandeep, dasm

:::warning
### Ongoing bugs

#### Upstream blockers
CS9 Master promo is blocked by:
* https://bugs.launchpad.net/tripleo/+bug/2006442 Master Undercloud upgrade job is failing with Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[neutron]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer
* **Order of merge**
* https://review.opendev.org/c/openstack/releases/+/872858
* https://review.opendev.org/c/openstack/puppet-tripleo/+/872859
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213
* https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/873121
* https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/870966

#### Downstream blockers
* https://bugzilla.redhat.com/show_bug.cgi?id=2167724 RHOS18 on RHEL9.2 - Error running container image prepare
* The downstream bug is fixed, we're still waiting for a merge of below patch.
* https://review.opendev.org/c/openstack/ansible-role-tripleo-modify-image/+/870966
* ~~https://code.engineering.redhat.com/gerrit/c/ansible-role-tripleo-modify-image/+/439594~~
* https://bugzilla.redhat.com/show_bug.cgi?id=2167302 17.1/9 - Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy:
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/439570
* https://code.engineering.redhat.com/gerrit/c/testproject/+/398540

### QE Bug
It is blocked by **Upstream blocker** https://bugs.launchpad.net/tripleo/+bug/2006442
* https://bugs.launchpad.net/tripleo/+bug/2000226
* unpin systemd*-252-*: tripleo-ci/CentOS-9/master.yaml
* revert: https://review.opendev.org/c/openstack/tripleo-quickstart/+/872672
* testproject: https://review.rdoproject.org/r/c/testproject/+/45851
:::

:::success
### Closed bugs:
:::spoiler
* ~~https://bugs.launchpad.net/tripleo/+bug/2004541~~ Hitting mirror issue everywhere
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2166325~~ periodic-tripleo-ci-rhel-9-standalone-full-tempest-api-network-rhos-17 failing consistently with sporadic tempest tests failures
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2166892~~ [rhos 16.2][rhos17.1]Container builds are failing on 'Tag images' step after
2.13.7 upgrade
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2166456~~ periodic-tripleo-build-containers-ubi-9-internal-rhel-9-build-push-upload-rhos-18 failing with “nothing provides network-scripts-openvswitch3.0”
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2154179~~ periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy
* ~~https://bugs.launchpad.net/tripleo/+bug/2004633~~ Ansible-lint in pep8 jobs on tht master is failing
* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2166265~~ periodic-tripleo-ci-rhel-9-standalone-on-multinode-ipa-tripleo-rhos-17.1 consistently failing on task “install FreeIPA packages”
* ~~https://bugs.launchpad.net/tripleo/+bug/2002478~~ centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops
* ~~https://bugs.launchpad.net/tripleo/+bug/2006479~~ openstack-tox-py27 job against stable/train is failing - bug fixed

:::

## Thu, Feb 09
:::danger
Due to upgrades, ZUUL was down for the entire day.
:::

## Wed, Feb 08
#### New Bugs
No new bug today

### Promotions
:::info
#### downstream
* RHEL 9 RHOS 18.0 - Red:
* https://bugzilla.redhat.com/show_bug.cgi?id=2162050
* RHEL 9 RHOS 17.1 2023-02-07 - Green
* RHEL 9 RHOS 17.0 2023-02-07 - Green - Line running right now
* RHEL 8 RHOS 17.1 2023-02-07 - Green - no new content
* RHEL 8 RHOS 16.2 - 2023-02-05 - Green - no new content

#### upstream
* CS 9 Master:
* upgrade job is blocking promotion
* CS 9 Zed: 2023-02-08 - Green - no new content
* CS 9 Wallaby: 2023-02-08 - Green - no new content
* CS 8 Wallaby: 2023-02-08 - Green - no new content
* CS 8 Train: 2023-02-01 - Green - no new content
:::
## Tuesday, Feb 07

#### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/2006479 openstack-tox-py27 job against stable/train is failing. - Need debug

* https://bugzilla.redhat.com/show_bug.cgi?id=2167724 RHOS18 on RHEL9.2 - Error running container image prepare

* https://bugs.launchpad.net/tripleo/+bug/2006442 Master Undercloud upgrade job is failing with Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[neutron]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer
* Takashi proposed fixes

* libvirt/issue
* Read comments on rhos-delivery that nova team need a reproducer of libvirt issue
* Created a revert of tripleo-environment workaround and offered it to kashyap but he already identified the issue by then.
* https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/439496
* https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
* We also had to pin python-libvirt

#### Upstream integration line Promotion status:

* Master - 2023-02-06 - yellow
* Upgrade job failed - bug reported
* https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-master/06b0525/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz
* 2023-02-07 04:38:33 | <13>Feb 7 04:38:22 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Haproxy::Balancermember[neutron]: parameter 'ports' expects a value of type Undef, Array, or String, got Integer (file: /etc/puppet/modules/tripleo/manifests/haproxy/endpoint.pp, line: 313) on node undercloud.localdomain
* Zed - 2023-02-~~06~~07 - ~~Yellow~~ Green
* ~~sc000: https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario000-multinode-oooq-container-updates-zed/d8b98cf/job-output.txt~~
* kex_exchange_identification
* Rerunning here:
* Wallaby C9 2023-02-07 - Green
* t~~empest-api - https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby/0af5c53 - tempest failure~~
* rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537
* Wallaby c8 - 2023-02-07 - Green (No new content)
* Train - 2023-02-01 - Green (No new content)

#### Upstream component line Promotion status:
* master
* https://review.rdoproject.org/r/c/testproject/+/45781
* zed - all green
* Wallaby C9 - all green
* Wallaby C8 - all green
* train - all green

#### Downstream integration line Promotion status:
* RHOS-18 RH9 - Standalone failing - https://bugzilla.redhat.com/show_bug.cgi?id=2167724
* RHOS-17.1 RH9 - ~~2023-01-30~~ 2023-02-07 - Green
* RHOS-17 RH9 - 2023-02-05
* https://code.engineering.redhat.com/gerrit/c/testproject/+/432677
* RHOS-17.1 RH8
* RHOS-17 RH9 - 2023-02-05 - Yellow
* Line running right now
* RHOS-16-2 RH8 - 2023-02-05 - Green (no new content)

## Monday, Feb 06

Check/gate: Currently all changes proposed to tripleo-heat-template master branch are failing on pep8 failure.
https://zuul.opendev.org//t/openstack/builds?job_name=openstack-tox-pep8&project=openstack/tripleo-heat-templates&branch=master

## New Issue

* https://bugs.launchpad.net/tripleo/+bug/2006125 periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation fails with validation errors on keystone token request

* Ansible-lint in pep8 jobs on tht master is failing https://bugs.launchpad.net/tripleo/+bug/2004633
* pinning ansible-lint: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/872749

* All jobs in downstream failing on Post failures.
https://trello.com/c/rdZWvnCi/2880-cixosp162osp171-all-zuul-jobs-in-downstream-failing-on-post-failure
* rhos-ops purged the log server

* 17.1/9 - Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy: https://bugzilla.redhat.com/show_bug.cgi?id=2167302
* Waiting for compute folks to take a look on bug
* We can try pinning libvirt/qemu but the nightly appstream repo we use don't have previous version anymore
* Pinged delivery to know if there is some repo from where we can still pull older versions.
* https://redhat-internal.slack.com/archives/C04HN4L5XHV/p1675689206995939
* trying pinning: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/439476
* logs for libvirt provided. PSA: container is named `nova_virtqemud`

#### Upstream integration line Promotion status:
* Master - 2023-02-~~04~~06 - Green
* ~~[fs039 failed](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/735af4b/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)~~
* promoted yesterday's hash
* ~~sc01 failed on today's run~~
* Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420

* Zed - 2023-02-06 - Green
* ~~[sc010 ovn provider failed ](https://logserver.rdoproject.org/openstack-periodic-integration-zed-centos9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario010-ovn-provider-standalone-zed/de7ceb8/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)~~
* ~~Rerunning here: https://review.rdoproject.org/r/c/testproject/+/36478~~

* Wallaby c9 - 2023-02-05 - Green (no new content)

* Wallaby C8 - 2023-02-06 - Green - (no new content)
**check cs8 wallaby criteria for swift**
* Train C8 - 2023-02-01 - Green - (no new content)

#### Downstream integration line Promotion status:
* RHOS-18 RH9
* RHOS-17.1 RH9
* RHOS-17.1 RH8
* RHOS-17 RH9 - 2023-02-05 - Green (no new content)
* RHOS-16-2 RH8 - 2023-02-05 - Green (no new content)

#### Upstream component line Promotion status:
* master
* Validation have real issue - Discussing with Jiri Podvin - he will report a bug soon.
* network line running right now
* Zed - Green
* network line running right now
* wallaby - Green
* network line running right now

## 3 Feb

#### Mirror issue is affecting downstream/upstream lines: https://bugs.launchpad.net/tripleo/+bug/2004541
Was debuging/following up with #rhos-ops

mirror issue got solved in downstream, this patch was missing in downstream/vexx: https://opendev.org/opendev/system-config/commit/0f1063963ea5be59b08e45db58a1b2d82258bda0 , theory for rca, something changed on pypi server side and length exceed 5M which was not comptaible with pip >22 , waiting for infra to apply same fix on vexx mirrors.

vexx mirror fix patch: https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/27312

#### retry/ retry_limit failure
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-9-content-provider - pinged infra, chatting with fungi

https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2023-02-03.log.html#t2023-02-03T12:15:04

#### Downstream pipelines: rhos-* were failing on Friday with `RETRY_LIMIT`
> Failure downloading http://download.devel.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm, Request failed: <urlopen error timed out>

Possible issue connected to outage: https://groups.google.com/a/redhat.com/g/outage-list/c/EpnXS0kuySA
> PSI RHOS-D RDU2 - Incident - Degraded State - 20220203-1810