Ruck Rover - Mar 25th - 31st Mar 2022

tags: ruck_rover

Previous RR notes: https://hackmd.io/wjL3Tpu8RhqiFWdKoGTOSw?view

Mar 31

New/transient/no bug yet:

• 09:38 < ysandeep> marios|ruck soniya29|rover: Seeing some NODE_FAILURE issues today: https://zuul.opendev.org/t/openstack/builds?result=NODE_FAILURE
  • https://meetings.opendev.org/irclogs/%23opendev/%23opendev.2022-03-31.log.html#t2022-03-31T06:48:54
• 11:42 < Tengu> ModuleNotFoundError: No module named 'ansible.module_utils.openstack'
  • (local env not seen in ci yet?)
  • 11:50 < Tengu> jm1: https://bugs.launchpad.net/tripleo/+bug/1967252 11:50 < Tengu> not added the ci/alert tags yet, but at least, it's ready.
• promoter not pulling? merged https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41216 (+ revert @ https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41233 )
  • had to login and git pull
• promotions master testproject there https://review.rdoproject.org/r/c/testproject/+/41219

BUGS

Mar 30 (& earlier ongoing things tracked here)

BUGS

• https://bugs.launchpad.net/tripleo/+bug/1967430 Upstream jobs broken by removal of setup.py from openstack-ansible-os_tempest (waiting on fix from os_tempest team - revert was not enough - patch 2 under test)

• https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) RAISED THIS TO CRITICAL - BLOCKING tripleo component line

• https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"

• https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"

• https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

• https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"

• https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'

• https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria

• https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON

• https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

• https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train

New/transient/no bug yet:

• https://bugs.launchpad.net/tripleo/+bug/1966382/comments/27 - we'll need the fix on stable branches too
• PSI outage continues (rhos-ops)
• train new issue/? https://review.rdoproject.org/zuul/buildset/11c238968964492e9b78295eecf4e1c1 fs1/fs2 heat stack create fail #TODO
  • https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/fd51416/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
  • https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-train/480755a/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
• testprojects
  • 8/wallaby https://review.rdoproject.org/r/c/testproject/+/41088/1#message-94bae9e6124006472aa2368805f14feccf712523
  • 9/wallaby https://review.rdoproject.org/r/c/testproject/+/41145/1#message-d722c28c0852321c9d549946b35b5d83a6f80158

Mar 29

BUGS

• https://bugs.launchpad.net/tripleo/+bug/1967012 tripleo-ci-centos-8-content-provider is failing DLRN build with "Error: Failed to download metadata for repo 'extras-common': Cannot prepare internal mirrorlist: No URLs in mirrorlist" - probably need Alfredo for this one - only build-test-packages brings in that repo

• https://bugs.launchpad.net/tripleo/+bug/1966625 deployment failures with latest puppet-ssh (from current-tripleo) RAISED THIS TO CRITICAL - BLOCKING tripleo component line

• https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"

• https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"

• https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue

• https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted)

• https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"

• https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried

• https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

• https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)

• https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"

• https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'

• https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria

• https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON

• https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

• https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train

• https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried

• https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (Master compute line just promoted)

• https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue (duplicate) * https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)

• https://bugs.launchpad.net/tripleo/+bug/1966972 cs9standalone-network wallaby fails with error running container image prepare

• https://bugs.launchpad.net/tripleo/+bug/1966879 openstack-tox-linters fails (tripleo-ci) ImportError: cannot import name '_unicodefun' from 'click'

• https://bugs.launchpad.net/tripleo/+bug/1966914 Centos-8 wallaby buildimage jobs fail dependency issue for centos-gpg-keys-1:8-2.el8.noarch

New/transient/no bug yet:

• < ysandeep> marios|ruck: thanks we have a bug already: https://bugs.launchpad.net/tripleo/+bug/1966625 thanks lmiccini to file that.. may be we can add our details in same bug and mark it romotion blocker for tripleo component. 13:04 < ysandeep> bogdando, same issue in tripleo component pipeline https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-tripleo-master/639f586/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

• PSI Network Outage (see outage list) downstream was unavailable all of this afternoon

• https://review.rdoproject.org/r/c/testproject/+/39960 fs035 failing for wallaby c8 and c9

• https://bugs.launchpad.net/tripleo/+bug/1967073 cs9 scenario000 multinode-oooq-container-updates-cloudops-master fails with "Error running container image prepare: Not found image" - we are no more seeing this

Mar 28

BUGS

• https://bugs.launchpad.net/tripleo/+bug/1966661 centos-9-standalone-on-multinode-ipa-master fails tempest "standalone-0.ooo.test' is not mapped to any cell"
• https://bugs.launchpad.net/tripleo/+bug/1966663 centos-8 featureset 002 wallaby fails node provision "Operation was aborted due to conductor take over"
• https://bugs.launchpad.net/tripleo/+bug/1966797 fs001 network wallaby fails with Server failed to reach ACTIVE status and task state "None" within the required time issue
• https://bugs.launchpad.net/tripleo/+bug/1966799 periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master
• https://bugzilla.redhat.com/show_bug.cgi?id=2069171 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing on tempest tests with "SSHTimeout issue"
• https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried
• https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
• https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot (This is marked fix released)
• https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
• https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
• https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
• https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
• https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
• https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train

New/transient/no bug yet:

• upstream zuul 10:17 -opendevstatus:#opendev- NOTICE: zuul isn't executing check jobs at the moment, investigation is ongoing, please be patient
• downstream zuul 11:00 < fbo> In the process of restarting Zuul and Nodepool … It seems to me that nodepool lost status of some nodes
• Note that there was an issue with opendev zuul (around 10 pm UTC): all jobs were cleared from check and gate. Infra is working on it. https://review.opendev.org/c/openstack/tripleo-ci/+/835101 needs a rerun if not back in check/gate by tomorrow

Downstream rhel-9 line:

(click to expand)

2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1362, in take_action
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.set_deployment_status(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.force_reraise()
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise self.value
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     deployment.config_download(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/deployment.py", line 407, in config_download
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     utils.run_ansible_playbook(
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 732, in run_ansible_playbook
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise RuntimeError(err_msg)
2022-03-25 14:41:05 | 2022-03-25 14:41:05.858 157532 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/zuul/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/zuul/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh

2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last):
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 34, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     super(Command, self).run(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return super(Command, self).run(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     return_code = self.take_action(parsed_args) or 0
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1216, in take_action
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.setup_ephemeral_heat(parsed_args)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/v1/overcloud_deploy.py", line 679, in setup_ephemeral_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     self.orchestration_client = utils.launch_heat(self.heat_launcher)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/utils.py", line 2746, in launch_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     launcher.launch_heat()
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib/python3.9/site-packages/tripleoclient/heat_launcher.py", line 507, in launch_heat
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     subprocess.check_call([
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud   File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud     raise CalledProcessError(retcode, cmd)
2022-03-27 23:32:55 | 2022-03-27 23:32:55.049 150380 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud subprocess.CalledProcessError: Command '['sudo', 'podman', 'play', 'kube', '/home/zuul/overcloud-deploy/overcloud/heat-launcher/heat-pod.yaml']' returned non-zero exit status 125.

LAGGING COMPONENTS

RDO

March 27

• https://bugs.launchpad.net/tripleo/+bug/1965124 various jobs failing repo setup Cannot download repodata/repomd.xml: All mirrors were tried
• https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
• https://bugs.launchpad.net/tripleo/+bug/1958230 [CS9/master] unable to provision OC node: timeout on second boot
• https://bugs.launchpad.net/tripleo/+bug/1964129 centos 9 wallaby periodic jobs failing container image prepare "read |0: i/o timeout"
• https://bugs.launchpad.net/tripleo/+bug/1965540 Tempest test 'test_image_web_download_import' failing with 'URI for web-download does not pass filtering'
• https://bugs.launchpad.net/tripleo/+bug/1959014 tempest.api.compute.images.test_list_image_filters failing on victoria
• https://bugs.launchpad.net/tripleo/+bug/1960661 Ussuri jobs failing on tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON
• https://bugs.launchpad.net/tripleo/+bug/1960355 Tempest tests "test_oauth1_tokens.OAUTH1TokensTest" are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
• https://bugs.launchpad.net/tripleo/+bug/1959328 tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops failing on periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train

Notes:

• openstack-periodic-integration-stable4-centos7 is broken now … change to container build parenting?
• ussuri missing fs035 https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/41024

Notes:

• wallaby c8 - https://review.rdoproject.org/r/c/testproject/+/35663 in rerun - missing only periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby. Ask amol to skip on promoter and promote this hash
• https://code.engineering.redhat.com/gerrit/c/testproject/+/190672/ rerun missing two jobs to promote 17

Bugs

• https://bugs.launchpad.net/tripleo/+bug/1965426
• https://bugs.launchpad.net/bugs/1964940
• https://bugs.launchpad.net/tripleo/+bug/1965540
  • https://bugzilla.redhat.com/2066264 - Downstream bug
• https://bugs.launchpad.net/tripleo/+bug/1966165

March 25

• new gate blocker centos-9 content provider https://bugs.launchpad.net/tripleo/+bug/1966384 (takashi also filed https://bugs.launchpad.net/tripleo/+bug/1966382)

• RHOSP17 on RHEL9 container build failing:

  • Sandeep proposed a patch: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/346059

• Upstream

  • centos 9 master jobs are failing over compute. Following is the common failures seen among all jobs.

  ​​​​Preparing metadata (pyproject.toml) did not run successfully.
  ​​​​2022-03-25 03:41:26.265925 | primary |       error: Multiple top-level packages discovered in a flat-layout: ['roles', 'plugins', 'infrared_plugin'].
  ​​​​2022-03-25 03:41:26.265933 | primary |
  ​​​​2022-03-25 03:41:26.265940 | primary |       To avoid accidental inclusion of unwanted files or directories,
  ​​​​2022-03-25 03:41:26.265948 | primary |       setuptools will not proceed     with this build.
  ​​​​2022-03-25 03:41:26.265955 | primary |
  ​​​​2022-03-25 03:41:26.265962 | primary |       If you are trying to create a single distribution with multiple packages
  ​​​​2022-03-25 03:41:26.265970 | primary |       on purpose, you should not rely on automatic discovery.
  ​​​​2022-03-25 03:41:26.265977 | primary |       Instead, consider the following options:
  ​​​​2022-03-25 03:41:26.265991 | primary |
  ​​​​2022-03-25 03:41:26.265999 | primary |       1. set up custom discovery (`find` directive with `include` or `exclude`)
  ​​​​2022-03-25 03:41:26.266006 | primary |       2. use a `src-layout`
  ​​​​2022-03-25 03:41:26.266014 | primary |       3. explicitly set `py_modules` or `packages` with a list of names
  ​​​​2022-03-25 03:41:26.266021 | primary |
  ​​​​2022-03-25 03:41:26.266028 | primary |       To find more information, look for "package discovery" on setuptools docs.
  ​​​​2022-03-25 03:41:26.266036 | primary |       [end of output]
  ​​​​2022-03-25 03:41:26.266043 | primary |
  ​​​​2022-03-25 03:41:26.266068 | primary |   note: This error originates from a subprocess, and is likely not a problem with pip.
  ​​​​2022-03-25 03:41:26.267683 | primary | error: metadata-generation-failed
  ​​​​2022-03-25 03:41:26.267713 | primary |
  ​​​​2022-03-25 03:41:26.267721 | primary | × Encountered error while generating package metadata.
  ​​​​2022-03-25 03:41:26.267729 | primary | ╰─> See above for output.
  ​​​​2022-03-25 03:41:26.267736 | primary |
  ​​​​2022-03-25 03:41:26.267744 | primary | note: This is an issue with the package mentioned above, not pip.
  

• https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-compute-master/1d9317b/job-output.txt

• rekicking these jobs here:- https://review.rdoproject.org/r/c/testproject/+/40932

March 24:

• Upstream

  • Status of component lines as follows :

    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master sporadic tempest test failures (different tests everytime based on latest two failed run)
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master "tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state" failed once in latest run
    • periodic-tripleo-ci-centos-9-containers-multinode-network-master OSError : OSError: [Errno 16] Device or resource busy: '/home/zuul/tripleo-deploy/undercloud/heat_launcher/tripleo_deploy-_oygmwuk' failed on latest run only
    • periodic-tripleo-ci-centos-9-scenario007-standalone-network-wallaby failed in latest run with Error running container image prepare

  • Running above failed jobs again due to not having consistent failures on this patch : https://review.rdoproject.org/r/c/testproject/+/33582

  • Testing of failed component jobs for all stable branches : https://review.rdoproject.org/r/q/topic:testing+status:open

  • Still having vexx host issues

    • https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario001-standalone-train/2bc20b1/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz

• Downstream

  • osp16.2

    • Promotions happened on march 22. Rekicked failed jenkins job : https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/(Latest update : we can see one job green and other is running well)
    • Components :
      • periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2 : consistent failures with bunch of tempest test in latest two runs. (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-containers-multinode-network-rhos-16.2/6e615ec/logs/undercloud/var/log/tempest/stestr_results.html)
      • periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2 : consistent failures with different tempest tests. sporadic failures. (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2/d95bf66/logs/undercloud/var/log/tempest/stestr_results.html)

  • osp17

    • Promotions happend on march 23. Retesting failed job here : https://code.engineering.redhat.com/gerrit/c/testproject/+/210197/91
    • Components : Not at all blockers on component blockers

  • osp17 rhel9

    • Promotions happend on march 23. No blockers.
    • Components :
      • periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17 failed once (https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-17/542913d/job-output.txt)