owned this note
owned this note
Published
Linked with GitHub
# Ruck Rover - Mar 25th - 31st Mar 2022
###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/wjL3Tpu8RhqiFWdKoGTOSw?view
## Mar 31
### New/transient/no bug yet:
* 09:38 < ysandeep> marios|ruck soniya29|rover: Seeing some NODE_FAILURE issues today: https://zuul.opendev.org/t/openstack/builds?result=NODE_FAILURE
* https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-03-31.log.html#t2022-03-31T06:48:54
* 11:42 < Tengu> ModuleNotFoundError: No module named 'ansible.module_utils.openstack'
* (local env not seen in ci yet?)
* 11:50 < Tengu> jm1: https://bugs.launchpad.net/tripleo/+bug/1967252 11:50 < Tengu> not added the ci/alert tags yet, but at least, it's ready.
* ~~promoter not pulling? merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41216 (+ revert @ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41233 )~~
* ~~had to login and git pull~~
* promotions master testproject there https://review.rdoproject.org/r/c/testproject/+/41219
#### BUGS
*
## Mar 30 (& earlier ongoing things tracked here)
#### BUGS
* https://bugs.launchpad.net/tripleo/+bug/1967430 Upstream jobs broken by removal of setup.py from openstack-ansible-os_tempest (waiting on fix from os_tempest team - revert was not enough - patch 2 under test)
* https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) **RAISED THIS TO CRITICAL - BLOCKING tripleo component line**
* https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"
* https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
* https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
* https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
* https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
* https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
* https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
### New/transient/no bug yet:
* https://bugs.launchpad.net/tripleo/+bug/1966382/comments/27 - we'll need the fix on stable branches too
* PSI outage continues (rhos-ops)
* train new issue/? https://review.rdoproject.org/zuul/buildset/11c238968964492e9b78295eecf4e1c1 fs1/fs2 heat stack create fail #TODO
* https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/fd51416/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
* https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/480755a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
* testprojects
* 8/wallaby https://review.rdoproject.org/r/c/testproject/+/41088/1#message-94bae9e6124006472aa2368805f14feccf712523
* 9/wallaby https://review.rdoproject.org/r/c/testproject/+/41145/1#message-d722c28c0852321c9d549946b35b5d83a6f80158
## Mar 29
#### BUGS
* ~~https://bugs.launchpad.net/tripleo/+bug/1967012
tripleo-ci-centos-8-content-provider is failing DLRN build with "Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist" - probably need Alfredo for this one - only build-test-packages brings in that repo~~
* https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) **RAISED THIS TO CRITICAL - BLOCKING tripleo component line**
* ~~https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"~~
* https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"
* https://bugs.launchpad.net/tripleo/+bug/1966797
fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue
* https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted)
* **https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"**
* https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)
* https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
* https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
* https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
* https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
* https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
* https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
* ~~https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried~~
* ~~https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted)~~
* ~~https://bugs.launchpad.net/tripleo/+bug/1966797
fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue~~ (duplicate)
~~* https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)~~
* ~~https://bugs.launchpad.net/tripleo/+bug/1966972
cs9standalone-network wallaby fails with error running container image prepare~~
* ~~https://bugs.launchpad.net/tripleo/+bug/1966879 openstack-tox-linters fails (tripleo-ci) ImportError: cannot import name '_unicodefun' from 'click'~~
* ~~https://bugs.launchpad.net/tripleo/+bug/1966914 Centos-8 wallaby buildimage jobs fail dependency issue for centos-gpg-keys-1:8-2.el8.noarch~~
### New/transient/no bug yet:
* ~~< ysandeep> marios|ruck: thanks we have a bug already: https://bugs.launchpad.net/tripleo/+bug/1966625 thanks lmiccini to file that.. may be we can add our details in same bug and mark it romotion blocker for tripleo component. 13:04 < ysandeep> bogdando, same issue in tripleo component pipeline https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/639f586/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz~~
* PSI Network Outage (see outage list) downstream was unavailable all of this afternoon
* https://review.rdoproject.org/r/c/testproject/+/39960 fs035 failing for wallaby c8 and c9
* https://bugs.launchpad.net/tripleo/+bug/1967073 ~~cs9 scenario000 multinode-oooq-container-updates-cloudops-master fails with "Error running container image prepare: Not found image"~~ - we are no more seeing this
## Mar 28
#### BUGS
* **https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"**
* **https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"**
* **https://bugs.launchpad.net/tripleo/+bug/1966797
fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue**
* **https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master**
* **https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"**
* https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)
* https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
* https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
* https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
* https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
* https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
* https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
### New/transient/no bug yet:
* upstream zuul 10:17 -opendevstatus:#opendev- NOTICE: zuul isn't executing check jobs at the moment, investigation is ongoing, please be patient
* downstream zuul 11:00 < fbo> In the process of restarting Zuul and Nodepool ... It seems to me that
nodepool lost status of some nodes
* Note that there was an issue with opendev zuul (around 10 pm UTC): all jobs were cleared from check and gate. Infra is working on it.
https://review.opendev.org/c/openstack/tripleo-ci/+/835101 needs a rerun if not back in check/gate by tomorrow
##### Downstream rhel-9 line:
(click to expand)
<details> <summary> periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 fails while overcloud deployment with following issue </summary>
* https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17
* chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/211643
* failed logs:- https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/129/check/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/1094b43/logs/undercloud/home/zuul/overcloud_deploy.log
```
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1362, in take_action
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise()
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 407, in config_download
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 732, in run_ansible_playbook
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh
```
</details>
<details><summary>periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17>periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17 </summary>is failing on overcloud deployment with tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError
* chasing here: https://code.engineering.redhat.com/gerrit/c/testproject/+/397219
```
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1216, in take_action
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.setup_ephemeral_heat(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 679, in setup_ephemeral_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.orchestration_client = utils.launch_heat(self.heat_launcher)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2746, in launch_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud launcher.launch_heat()
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/heat_launcher.py", line 507, in launch_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.check_call([
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise CalledProcessError(retcode, cmd)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'play', 'kube', '/home/zuul/overcloud-deploy/overcloud/heat-launcher/heat-pod.yaml']' returned non-zero exit status 125.
```
</details>
#### LAGGING COMPONENTS
##### RDO
<details>
<summary>rdo components</summary>
<p>
c8victoria c8train c9wallaby * all clear
ussuri * tempest 18 days ago - 14 days ago
wallaby c8 * network 11 days ago 3 days ago
master * 11 days ago compute 3 days * 6 days ago tripleo 2 days ago * 5 days ago network 2 days ago * 4 days ago tempest 2 days ago * 4 days ago clients 3 days ago
</p>
* </details>
<details>
<summary> OSP </summary>
* 16.2
* 17 on rhel 8
* 17 on rhel 9
</details>
## March 27
* https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried
* https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
* https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot
* https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
* https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
* https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
* https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
* https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
* https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
#### Notes:
* openstack-periodic-integration-stable4-centos7 is broken now ... change to container build parenting?
* ussuri missing fs035 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41024
#### Notes:
* wallaby c8 - https://review.rdoproject.org/r/c/testproject/+/35663 in rerun - missing only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby. Ask amol to skip on promoter and promote this hash
* https://code.engineering.redhat.com/gerrit/c/testproject/+/190672/ rerun missing two jobs to promote 17
## Bugs
* https://bugs.launchpad.net/tripleo/+bug/1965426
* https://bugs.launchpad.net/bugs/1964940
* https://bugs.launchpad.net/tripleo/+bug/1965540
* https://bugzilla.redhat.com/2066264 - Downstream bug
* https://bugs.launchpad.net/tripleo/+bug/1966165
## March 25
* new gate blocker centos-9 content provider https://bugs.launchpad.net/tripleo/+bug/1966384 (takashi also filed https://bugs.launchpad.net/tripleo/+bug/1966382)
* RHOSP17 on RHEL9 container build failing:
* Sandeep proposed a patch: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059
* Upstream
* centos 9 master jobs are failing over compute. Following is the common failures seen among all jobs.
```
Preparing metadata (pyproject.toml) did not run successfully.
2022-03-25 03:41:26.265925 | primary | error: Multiple top-level packages discovered in a flat-layout: ['roles', 'plugins', 'infrared_plugin'].
2022-03-25 03:41:26.265933 | primary |
2022-03-25 03:41:26.265940 | primary | To avoid accidental inclusion of unwanted files or directories,
2022-03-25 03:41:26.265948 | primary | setuptools will not proceed with this build.
2022-03-25 03:41:26.265955 | primary |
2022-03-25 03:41:26.265962 | primary | If you are trying to create a single distribution with multiple packages
2022-03-25 03:41:26.265970 | primary | on purpose, you should not rely on automatic discovery.
2022-03-25 03:41:26.265977 | primary | Instead, consider the following options:
2022-03-25 03:41:26.265991 | primary |
2022-03-25 03:41:26.265999 | primary | 1. set up custom discovery (`find` directive with `include` or `exclude`)
2022-03-25 03:41:26.266006 | primary | 2. use a `src-layout`
2022-03-25 03:41:26.266014 | primary | 3. explicitly set `py_modules` or `packages` with a list of names
2022-03-25 03:41:26.266021 | primary |
2022-03-25 03:41:26.266028 | primary | To find more information, look for "package discovery" on setuptools docs.
2022-03-25 03:41:26.266036 | primary | [end of output]
2022-03-25 03:41:26.266043 | primary |
2022-03-25 03:41:26.266068 | primary | note: This error originates from a subprocess, and is likely not a problem with pip.
2022-03-25 03:41:26.267683 | primary | error: metadata-generation-failed
2022-03-25 03:41:26.267713 | primary |
2022-03-25 03:41:26.267721 | primary | × Encountered error while generating package metadata.
2022-03-25 03:41:26.267729 | primary | ╰─> See above for output.
2022-03-25 03:41:26.267736 | primary |
2022-03-25 03:41:26.267744 | primary | note: This is an issue with the package mentioned above, not pip.
```
- https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-compute-master/1d9317b/job-output.txt
- rekicking these jobs here:- https://review.rdoproject.org/r/c/testproject/+/40932
## March 24:
* Upstream
* Status of component lines as follows :
* **periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master**
sporadic tempest test failures (different tests everytime based on latest two failed run)
* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master**
"tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state" failed once in latest run
* **periodic-tripleo-ci-centos-9-containers-multinode-network-master**
OSError : OSError: [Errno 16] Device or resource busy: '/home/zuul/tripleo-deploy/undercloud/heat_launcher/tripleo_deploy-_oygmwuk' failed on latest run only
* **periodic-tripleo-ci-centos-9-scenario007-standalone-network-wallaby**
failed in latest run with Error running container image prepare
* Running above failed jobs again due to not having consistent failures on this patch : https://review.rdoproject.org/r/c/testproject/+/33582
* Testing of failed component jobs for all stable branches : https://review.rdoproject.org/r/q/topic:testing+status:open
* Still having vexx host issues
* https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/2bc20b1/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz
* Downstream
* osp16.2
* Promotions happened on march 22. Rekicked failed jenkins job :
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/(Latest update : we can see one job green and other is running well)
* Components :
* periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2 : consistent failures with bunch of tempest test in latest two runs.
(https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2/6e615ec/logs/undercloud/var/log/tempest/stestr_results.html)
* periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2 : consistent failures with different tempest tests. sporadic failures. (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2/d95bf66/logs/undercloud/var/log/tempest/stestr_results.html)
* osp17
* Promotions happend on march 23. Retesting failed job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/210197/91
* Components : Not at all blockers on component blockers
* osp17 rhel9
* Promotions happend on march 23. No blockers.
* Components :
* periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17 failed once (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17/542913d/job-output.txt)