Ruck Rover - Mar 25th - 31st Mar 2022

# Ruck Rover - Mar 25th - 31st Mar 2022 ###### tags: `ruck_rover` ###### Previous RR notes: https://hackmd.io/wjL3Tpu8RhqiFWdKoGTOSw?view ## Mar 31 ### New/transient/no bug yet: * 09:38 < ysandeep> marios|ruck soniya29|rover: Seeing some NODE_FAILURE issues today: https://zuul.opendev.org/t/openstack/builds?result=NODE_FAILURE * https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-03-31.log.html#t2022-03-31T06:48:54 * 11:42 < Tengu> ModuleNotFoundError: No module named 'ansible.module_utils.openstack' * (local env not seen in ci yet?) * 11:50 < Tengu> jm1: https://bugs.launchpad.net/tripleo/+bug/1967252 11:50 < Tengu> not added the ci/alert tags yet, but at least, it's ready. * ~~promoter not pulling? merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41216 (+ revert @ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41233 )~~ * ~~had to login and git pull~~ * promotions master testproject there https://review.rdoproject.org/r/c/testproject/+/41219 #### BUGS * ## Mar 30 (& earlier ongoing things tracked here) #### BUGS * https://bugs.launchpad.net/tripleo/+bug/1967430 Upstream jobs broken by removal of setup.py from openstack-ansible-os_tempest (waiting on fix from os_tempest team - revert was not enough - patch 2 under test) * https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) **RAISED THIS TO CRITICAL - BLOCKING tripleo component line** * https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over" * https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue" * https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time. * https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout" * https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering' * https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria * https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON * https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) * https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train ### New/transient/no bug yet: * https://bugs.launchpad.net/tripleo/+bug/1966382/comments/27 - we'll need the fix on stable branches too * PSI outage continues (rhos-ops) * train new issue/? https://review.rdoproject.org/zuul/buildset/11c238968964492e9b78295eecf4e1c1 fs1/fs2 heat stack create fail #TODO * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/fd51416/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/480755a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz * testprojects * 8/wallaby https://review.rdoproject.org/r/c/testproject/+/41088/1#message-94bae9e6124006472aa2368805f14feccf712523 * 9/wallaby https://review.rdoproject.org/r/c/testproject/+/41145/1#message-d722c28c0852321c9d549946b35b5d83a6f80158 ## Mar 29 #### BUGS * ~~https://bugs.launchpad.net/tripleo/+bug/1967012 tripleo-ci-centos-8-content-provider is failing DLRN build with "Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist" - probably need Alfredo for this one - only build-test-packages brings in that repo~~ * https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) **RAISED THIS TO CRITICAL - BLOCKING tripleo component line** * ~~https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"~~ * https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over" * https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue * https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted) * **https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"** * https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried * https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time. * https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released) * https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout" * https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering' * https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria * https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON * https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) * https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train * ~~https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried~~ * ~~https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted)~~ * ~~https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue~~ (duplicate) ~~* https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)~~ * ~~https://bugs.launchpad.net/tripleo/+bug/1966972 cs9standalone-network wallaby fails with error running container image prepare~~ * ~~https://bugs.launchpad.net/tripleo/+bug/1966879 openstack-tox-linters fails (tripleo-ci) ImportError: cannot import name '_unicodefun' from 'click'~~ * ~~https://bugs.launchpad.net/tripleo/+bug/1966914 Centos-8 wallaby buildimage jobs fail dependency issue for centos-gpg-keys-1:8-2.el8.noarch~~ ### New/transient/no bug yet: * ~~< ysandeep> marios|ruck: thanks we have a bug already: https://bugs.launchpad.net/tripleo/+bug/1966625 thanks lmiccini to file that.. may be we can add our details in same bug and mark it romotion blocker for tripleo component. 13:04 < ysandeep> bogdando, same issue in tripleo component pipeline https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/639f586/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz~~ * PSI Network Outage (see outage list) downstream was unavailable all of this afternoon * https://review.rdoproject.org/r/c/testproject/+/39960 fs035 failing for wallaby c8 and c9 * https://bugs.launchpad.net/tripleo/+bug/1967073 ~~cs9 scenario000 multinode-oooq-container-updates-cloudops-master fails with "Error running container image prepare: Not found image"~~ - we are no more seeing this ## Mar 28 #### BUGS * **https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"** * **https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"** * **https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue** * **https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master** * **https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"** * https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried * https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time. * https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released) * https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout" * https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering' * https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria * https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON * https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) * https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train ### New/transient/no bug yet: * upstream zuul 10:17 -opendevstatus:#opendev- NOTICE: zuul isn't executing check jobs at the moment, investigation is ongoing, please be patient * downstream zuul 11:00 < fbo> In the process of restarting Zuul and Nodepool ... It seems to me that nodepool lost status of some nodes * Note that there was an issue with opendev zuul (around 10 pm UTC): all jobs were cleared from check and gate. Infra is working on it. https://review.opendev.org/c/openstack/tripleo-ci/+/835101 needs a rerun if not back in check/gate by tomorrow ##### Downstream rhel-9 line: (click to expand) <details> <summary> periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 fails while overcloud deployment with following issue </summary> * https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 * chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/211643 * failed logs:- https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/129/check/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/1094b43/logs/undercloud/home/zuul/overcloud_deploy.log ``` 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1362, in take_action 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.set_deployment_status( 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud deployment.config_download( 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 407, in config_download 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud utils.run_ansible_playbook( 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 732, in run_ansible_playbook 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh ``` </details> <details><summary>periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17>periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17 </summary>is failing on overcloud deployment with tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError * chasing here: https://code.engineering.redhat.com/gerrit/c/testproject/+/397219 ``` 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1216, in take_action 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.setup_ephemeral_heat(parsed_args) 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 679, in setup_ephemeral_heat 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.orchestration_client = utils.launch_heat(self.heat_launcher) 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2746, in launch_heat 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud launcher.launch_heat() 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.9/site-packages/tripleoclient/heat_launcher.py", line 507, in launch_heat 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.check_call([ 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise CalledProcessError(retcode, cmd) 2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'play', 'kube', '/home/zuul/overcloud-deploy/overcloud/heat-launcher/heat-pod.yaml']' returned non-zero exit status 125. ``` </details> #### LAGGING COMPONENTS ##### RDO <details> <summary>rdo components</summary> <p> c8victoria c8train c9wallaby * all clear ussuri * tempest 18 days ago - 14 days ago wallaby c8 * network 11 days ago 3 days ago master * 11 days ago compute 3 days * 6 days ago tripleo 2 days ago * 5 days ago network 2 days ago * 4 days ago tempest 2 days ago * 4 days ago clients 3 days ago </p> * </details> <details> <summary> OSP </summary> * 16.2 * 17 on rhel 8 * 17 on rhel 9 </details> ## March 27 * https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried * https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time. * https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot * https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout" * https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering' * https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria * https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON * https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby) * https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train #### Notes: * openstack-periodic-integration-stable4-centos7 is broken now ... change to container build parenting? * ussuri missing fs035 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41024 #### Notes: * wallaby c8 - https://review.rdoproject.org/r/c/testproject/+/35663 in rerun - missing only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby. Ask amol to skip on promoter and promote this hash * https://code.engineering.redhat.com/gerrit/c/testproject/+/190672/ rerun missing two jobs to promote 17 ## Bugs * https://bugs.launchpad.net/tripleo/+bug/1965426 * https://bugs.launchpad.net/bugs/1964940 * https://bugs.launchpad.net/tripleo/+bug/1965540 * https://bugzilla.redhat.com/2066264 - Downstream bug * https://bugs.launchpad.net/tripleo/+bug/1966165 ## March 25 * new gate blocker centos-9 content provider https://bugs.launchpad.net/tripleo/+bug/1966384 (takashi also filed https://bugs.launchpad.net/tripleo/+bug/1966382) * RHOSP17 on RHEL9 container build failing: * Sandeep proposed a patch: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059 * Upstream * centos 9 master jobs are failing over compute. Following is the common failures seen among all jobs. ``` Preparing metadata (pyproject.toml) did not run successfully. 2022-03-25 03:41:26.265925 | primary | error: Multiple top-level packages discovered in a flat-layout: ['roles', 'plugins', 'infrared_plugin']. 2022-03-25 03:41:26.265933 | primary | 2022-03-25 03:41:26.265940 | primary | To avoid accidental inclusion of unwanted files or directories, 2022-03-25 03:41:26.265948 | primary | setuptools will not proceed with this build. 2022-03-25 03:41:26.265955 | primary | 2022-03-25 03:41:26.265962 | primary | If you are trying to create a single distribution with multiple packages 2022-03-25 03:41:26.265970 | primary | on purpose, you should not rely on automatic discovery. 2022-03-25 03:41:26.265977 | primary | Instead, consider the following options: 2022-03-25 03:41:26.265991 | primary | 2022-03-25 03:41:26.265999 | primary | 1. set up custom discovery (`find` directive with `include` or `exclude`) 2022-03-25 03:41:26.266006 | primary | 2. use a `src-layout` 2022-03-25 03:41:26.266014 | primary | 3. explicitly set `py_modules` or `packages` with a list of names 2022-03-25 03:41:26.266021 | primary | 2022-03-25 03:41:26.266028 | primary | To find more information, look for "package discovery" on setuptools docs. 2022-03-25 03:41:26.266036 | primary | [end of output] 2022-03-25 03:41:26.266043 | primary | 2022-03-25 03:41:26.266068 | primary | note: This error originates from a subprocess, and is likely not a problem with pip. 2022-03-25 03:41:26.267683 | primary | error: metadata-generation-failed 2022-03-25 03:41:26.267713 | primary | 2022-03-25 03:41:26.267721 | primary | × Encountered error while generating package metadata. 2022-03-25 03:41:26.267729 | primary | ╰─> See above for output. 2022-03-25 03:41:26.267736 | primary | 2022-03-25 03:41:26.267744 | primary | note: This is an issue with the package mentioned above, not pip. ``` - https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-compute-master/1d9317b/job-output.txt - rekicking these jobs here:- https://review.rdoproject.org/r/c/testproject/+/40932 ## March 24: * Upstream * Status of component lines as follows : * **periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master** sporadic tempest test failures (different tests everytime based on latest two failed run) * **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master** "tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state" failed once in latest run * **periodic-tripleo-ci-centos-9-containers-multinode-network-master** OSError : OSError: [Errno 16] Device or resource busy: '/home/zuul/tripleo-deploy/undercloud/heat_launcher/tripleo_deploy-_oygmwuk' failed on latest run only * **periodic-tripleo-ci-centos-9-scenario007-standalone-network-wallaby** failed in latest run with Error running container image prepare * Running above failed jobs again due to not having consistent failures on this patch : https://review.rdoproject.org/r/c/testproject/+/33582 * Testing of failed component jobs for all stable branches : https://review.rdoproject.org/r/q/topic:testing+status:open * Still having vexx host issues * https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/2bc20b1/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz * Downstream * osp16.2 * Promotions happened on march 22. Rekicked failed jenkins job : https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/(Latest update : we can see one job green and other is running well) * Components : * periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2 : consistent failures with bunch of tempest test in latest two runs. (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2/6e615ec/logs/undercloud/var/log/tempest/stestr_results.html) * periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2 : consistent failures with different tempest tests. sporadic failures. (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2/d95bf66/logs/undercloud/var/log/tempest/stestr_results.html) * osp17 * Promotions happend on march 23. Retesting failed job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/210197/91 * Components : Not at all blockers on component blockers * osp17 rhel9 * Promotions happend on march 23. No blockers. * Components : * periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17 failed once (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17/542913d/job-output.txt)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.