# RR 7 March 2022:

## RHOS 17 failures - soniya29

1. periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-compute-rhos-17
- failing on compute tests

traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 312, in wait_for_volume_resource_status
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 1850310b-4cc4-40f6-a328-b7d7890599e1 failed to reach available status (current in-use) within the required time (300 s).
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 380, in wait_for_volume_attachment_remove_from_server
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: Volume 1850310b-4cc4-40f6-a328-b7d7890599e1 failed to detach from server 014313f8-8d24-4b76-a614-3bc43a8c643b within the required time (300 s) from the compute API perspective

- https://code.engineering.redhat.com/gerrit/c/testproject/+/315939

2. integration line seems good for rhos-17
3. all jenkins job are failing on rhos-17 line
- https://bugzilla.redhat.com/show_bug.cgi?id=2059157
- not in criteria

## RHOS 16.2 failures - soniay29

1. periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2
- failing on neutron tempest test
- https://code.engineering.redhat.com/gerrit/c/testproject/+/316224

AssertionError: Timed out waiting for 10.100.0.17 to become reachable from 10.9.122.72

2. periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2
- undercloud installation failed
- testproject:- https://code.engineering.redhat.com/gerrit/c/testproject/+/316211

2022-03-07 01:03:26.393807 |  The following node(s) had failures: undercloud
2022-03-07 01:03:26.393902 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Exception: Deployment failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1357, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed
None
** Found ansible errors for undercloud deployment! **
[
 [
  "Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log",
  {
   "msg": "non-zero return code",
   "cmd": "/usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.ugxqp8ei-role-data --environment-file /tmp/ansible.kj4rj9m5-prepare-param --cleanup partial --log-file /var/log/tripleo-container-image-prepare.log  --debug  >/dev/null",
   "stdout": "",

3. 16.2 seems to be in good shape
4. jenkins

## wallaby centos-8 failures(soniya29):

1. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby -
- Situation remains same..all tempest tests are failing
- [logs](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5aa2d3d/logs/undercloud/var/log/tempest/stestr_results.html.gz)
- [testproject](https://review.rdoproject.org/r/c/testproject/+/40101)
- https://bugs.launchpad.net/tripleo/+bug/1963874

2. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby
- timed_out
- [testproject](https://review.rdoproject.org/r/c/testproject/+/40102)
- https://bugs.launchpad.net/tripleo/+bug/1963884

3. periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby
- passing
- [job_health](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby)

4. periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby
- passing
- [job_health](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby)

## wallaby centos-8 failures:

1. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby - soniya29
- All tempest tests are failing here
- testproject:- https://review.rdoproject.org/r/c/testproject/+/40101
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5f880b7/logs/undercloud/var/log/tempest/stestr_results.html.gz
3. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby - soniya29
- Neutron tempest tests are failing here with 'Max tries exceeded''
- Testing here:- https://review.rdoproject.org/r/c/testproject/+/40102
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby/7d13c5b/logs/undercloud/var/log/tempest/stestr_results.html.gz
5. periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby - soniya29
- passing
- https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby
7. periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby - soniya29
- passing
- https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby

Testproject topic for wallaby failures:-
https://review.rdoproject.org/r/q/topic:chasing-wallaby-failures
---

## Rhos-17 failures:

* periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-compute-rhos-17 - fails on compute tempest tests

    - tempest.lib.exceptions.ServerFault: Got server fault

    

- Testing here :- https://code.engineering.redhat.com/gerrit/c/testproject/+/315939
---

Testproj topic: https://review.rdoproject.org/r/q/topic:chasing_promotions_040322

---
## Integration master failures

* **periodic-tripleo-ci-centos-9-scenario001-standalone-master - inconsistent failures**

- Reason: inconsistent - Executing python-tempestconf,
Failed tempest tests

* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master (AB)- multiple network issues**
- ~~Bug: https://bugs.launchpad.net/tripleo/+bug/1963873~~

* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master** (AB) - fails tripleo-ipa preparation

- Bug: https://bugs.launchpad.net/tripleo/+bug/1963876

- Log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/3efe875/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz

2022-03-06 21:42:17 | fatal: [localhost]: FAILED! => {"changed": false, "msg": "login: HTTP Error 500: Internal Server Error"}

* **periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master**, **periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master** (AB) -
- Reason: tempest multiple failures
- Log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/233a029/logs/undercloud/var/log/tempest/stestr_results.html.gz

  Response - Headers: {'date': 'Mon, 07 Mar 2022 03:08:44 GMT', 'server': 'Apache', 'content-length': '111', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-9db28f47-a191-40d2-97c7-618901830126', 'x-compute-request-id': 'req-9db28f47-a191-40d2-97c7-618901830126', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'status': '404', 'content-location': 'http://10.0.0.5:8774/v2.1/servers/b7bbe195-ea0a-4e97-bab0-87b14895b4e2'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance b7bbe195-ea0a-4e97-bab0-87b14895b4e2 could not be found."}}'

* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master** (AB)
* Reason: inconsistent errors

---

## Integration wallaby c8 failures

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby,
Bug: https://bugs.launchpad.net/tripleo/+bug/1962727

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby
Inconsistent failures (overcloud deploy Message queue for ephemeral heat not created in time., or tempest tests)

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby
inconsistent failures (tempest, undercloud, overcloud)

## Integration ussuri failures

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-ussuri (AB)

FATAL | Install - IPA client test | undercloud | error={"changed": false, "msg": "Unable to discover domain, not provided on command line"}
2022-03-04 08:55:41.688103 | fa163e8f-fef8-2748-cb2e-0000000004b9 |     TIMING | ipaclient : Install - IPA client test | undercloud | 0:00:54.408165 | 0.81s

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri (AB)

nova_db_sync.log: 2022-03-04T09:18:29.920031985+00:00 stdout F Skipping execution since this is not the bootstrap node for this service.

SK | Check containers status
2022-03-04 09:51:28 |  [ERROR]: Container(s) which failed to be created by podman_container module:
2022-03-04 09:51:28 | ['nova_db_sync']
2022-03-04 09:51:28 |  [ERROR]: Container(s) which did not finish after 300 minutes: ['nova_db_sync']
2022-03-04 09:51:28 | 2022-03-04 09:51:28.999785 | fa163e1f-30ca-4089-242a-00000000a05b |      FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['nova_db_sync'], check logs in /var/log/containers/stdouts/"}
2022-03-04 09:51:28 | 2022-03-04 09:51:29.000873 | fa163e1f-30ca-4089-242a-00000000a05b |     TIMING | tripleo_container_manage : Check containers status | overcloud-controller-0 | 1:09:05.953041 | 461.26s

* periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri (AB) - inconsistent failures, looks green

---

## component job failures

* periodic-tripleo-ci-centos-9-standalone-baremetal-master (AB) - Tempest fail

tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state[compute,id-f5dfcc22-45fd-409f-954c-5bd500d7890b,network,slow]
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state[compute,id-04b9fe4e-85e8-4aea-b937-ea93885ac59f,network,slow]
tearDownClass (tempest.scenario.test_network_basic_ops.TestNetworkBasicOps)
setUpClass (tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern)
setUpClass (keystone_tempest_plugin.tests.api.identity.v3.test_service_providers.ServiceProvidersTest)

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master (AB) - Node provisioning for overcloud failed

* periodic-tripleo-ci-centos-9-scenario003-standalone-common-master, periodic-tripleo-ci-centos-9-standalone-tempest-master (AB)

Error running playbook in directory: /tmp/tripleo-modify-image-playbook-1nir90xk
Playbook log information can be reviewed here: /var/log/tripleo-container-image-prepare-ansible.log
2022-03-04 10:29:39,817 61756 ERROR tripleo_common.image.image_uploader [  ] [192.168.24.1:8787/tripleomastercentos9/openstack-cron:afeb49ceb7917185b2d930aeb7a0785c-updated-20220304045521] Failed processing the target image
2022-03-04 10:29:39,849 61756 INFO tripleo_common.image.image_uploader [  ] [trunk.registry.rdoproject.org/tripleomastercentos9/openstack-memcached:afeb49ceb7917185b2d930aeb7a0785c] Starting upload image process
2022-03-04 10:29:39,850 61756 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2022-03-04 10:29:39,853 61756 DEBUG urllib3.connectionpool [  ] Starting new HTTP connection (1): 192.168.24.1:8787
2022-03-04 10:29:39,855 61756 DEBUG urllib3.connectionpool [  ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2022-03-04 10:29:39,856 61756 DEBUG tripleo_common.image.image_uploader [  ] http://192.168.24.1:8787/v2/ status code 200
2022-03-04 10:29:39,859 61756 DEBUG urllib3.connectionpool [  ] http://192.168.24.1:8787 "GET /v2/tripleomastercentos9/openstack-memcached/tags/list HTTP/1.1" 404 196
2022-03-04 10:29:39,859 61756 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id 7e20929a3dcd948cc4cdbf250f7540fca1fcc895, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

`FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-designate-mdns:afeb49ceb7917185b2d930aeb7a0785c failed", "msg": "Error running container image prepare: Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-designate-mdns:afeb49ceb7917185b2d930aeb7a0785c failed", "params": {}, "success": false}`

* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (AB)

`neutron_tempest_plugin.scenario.test_basic.NetworkBasicTest.test_basic_instance`

and others which are not same every time

* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-master (AB) - inconsistent