---
# System prepended metadata

title: 'RR 7 March 2022:'

---

# RR 7 March 2022:


## RHOS 17 failures - soniya29

1. periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-compute-rhos-17
    - failing on compute tests
```
traceback-1: {{{
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 312, in wait_for_volume_resource_status
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: volume 1850310b-4cc4-40f6-a328-b7d7890599e1 failed to reach available status (current in-use) within the required time (300 s).
}}}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 380, in wait_for_volume_attachment_remove_from_server
    raise lib_exc.TimeoutException(message)
tempest.lib.exceptions.TimeoutException: Request timed out
Details: Volume 1850310b-4cc4-40f6-a328-b7d7890599e1 failed to detach from server 014313f8-8d24-4b76-a614-3bc43a8c643b within the required time (300 s) from the compute API perspective

```
    - https://code.engineering.redhat.com/gerrit/c/testproject/+/315939

2. integration line seems good for rhos-17
3. all jenkins job are failing on rhos-17 line
    - https://bugzilla.redhat.com/show_bug.cgi?id=2059157
    - not in criteria

## RHOS 16.2 failures - soniay29

1. periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2
    - failing on neutron tempest test
    - https://code.engineering.redhat.com/gerrit/c/testproject/+/316224
```
AssertionError: Timed out waiting for 10.100.0.17 to become reachable from 10.9.122.72

```
2. periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2
    - undercloud installation failed
    - testproject:- https://code.engineering.redhat.com/gerrit/c/testproject/+/316211
```
2022-03-07 01:03:26.393807 |  The following node(s) had failures: undercloud
2022-03-07 01:03:26.393902 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Exception: Deployment failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_deploy.py", line 1357, in _standalone_deploy
    raise exceptions.DeploymentError('Deployment failed')
tripleoclient.exceptions.DeploymentError: Deployment failed
None
** Found ansible errors for undercloud deployment! **
[
 [
  "Run tripleo-container-image-prepare logged to: /var/log/tripleo-container-image-prepare.log",
  {
   "msg": "non-zero return code",
   "cmd": "/usr/bin/tripleo-container-image-prepare --roles-file /tmp/ansible.ugxqp8ei-role-data --environment-file /tmp/ansible.kj4rj9m5-prepare-param --cleanup partial --log-file /var/log/tripleo-container-image-prepare.log  --debug  >/dev/null",
   "stdout": "",
```
3. 16.2 seems to be in good shape
4. jenkins 

## wallaby centos-8 failures(soniya29):

1. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby - 
    -  Situation remains same..all tempest tests are failing
    -  [logs](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5aa2d3d/logs/undercloud/var/log/tempest/stestr_results.html.gz)
    -  [testproject](https://review.rdoproject.org/r/c/testproject/+/40101)
    -  https://bugs.launchpad.net/tripleo/+bug/1963874

2. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby
    - timed_out
    - [testproject](https://review.rdoproject.org/r/c/testproject/+/40102)
    - https://bugs.launchpad.net/tripleo/+bug/1963884

3. periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby
    - passing
    - [job_health](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby)

4. periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby
    - passing
    - [job_health](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby)





## wallaby centos-8 failures:

1. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby - soniya29
    - All tempest tests are failing here
    - testproject:- https://review.rdoproject.org/r/c/testproject/+/40101
    - https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-wallaby/5f880b7/logs/undercloud/var/log/tempest/stestr_results.html.gz
3. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby - soniya29
    - Neutron tempest tests are failing here with 'Max tries exceeded''
    - Testing here:- https://review.rdoproject.org/r/c/testproject/+/40102
    - https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-wallaby/7d13c5b/logs/undercloud/var/log/tempest/stestr_results.html.gz
5. periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby - soniya29
    - passing
    - https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-wallaby
7. periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby - soniya29
    - passing
    - https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario007-multinode-oooq-container-wallaby


Testproject topic for wallaby failures:-
https://review.rdoproject.org/r/q/topic:chasing-wallaby-failures
---

## Rhos-17 failures:

* periodic-tripleo-ci-rhel-8-standalone-full-tempest-api-compute-rhos-17 - fails on compute tempest tests
```
    - tempest.lib.exceptions.ServerFault: Got server fault

    
```
- Testing here :- https://code.engineering.redhat.com/gerrit/c/testproject/+/315939
---

Testproj topic: https://review.rdoproject.org/r/q/topic:chasing_promotions_040322

---
## Integration master failures

* **periodic-tripleo-ci-centos-9-scenario001-standalone-master - inconsistent failures**

    - Reason: inconsistent -  Executing python-tempestconf, 
        Failed tempest tests

* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master (AB)- multiple network issues**
    - ~~Bug: https://bugs.launchpad.net/tripleo/+bug/1963873~~


* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master** (AB) - fails tripleo-ipa preparation
 
 
    - Bug: https://bugs.launchpad.net/tripleo/+bug/1963876

    - Log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master/3efe875/logs/supplemental/home/cloud-user/ipa_prep.sh.log.txt.gz
```TASK [add nova host manager role] **********************************************
2022-03-06 21:42:17 | fatal: [localhost]: FAILED! => {"changed": false, "msg": "login: HTTP Error 500: Internal Server Error"}
```

* **periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master**, **periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master** (AB) - 
    - Reason: tempest multiple failures
    - Log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master/233a029/logs/undercloud/var/log/tempest/stestr_results.html.gz
```
  Response - Headers: {'date': 'Mon, 07 Mar 2022 03:08:44 GMT', 'server': 'Apache', 'content-length': '111', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-9db28f47-a191-40d2-97c7-618901830126', 'x-compute-request-id': 'req-9db28f47-a191-40d2-97c7-618901830126', 'content-type': 'application/json; charset=UTF-8', 'connection': 'close', 'status': '404', 'content-location': 'http://10.0.0.5:8774/v2.1/servers/b7bbe195-ea0a-4e97-bab0-87b14895b4e2'}
        Body: b'{"itemNotFound": {"code": 404, "message": "Instance b7bbe195-ea0a-4e97-bab0-87b14895b4e2 could not be found."}}'
```
* **periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master** (AB) 
    * Reason: inconsistent errors


---

## Integration wallaby c8 failures

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-wallaby, 
 Bug: https://bugs.launchpad.net/tripleo/+bug/1962727

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby
 Inconsistent failures (overcloud deploy Message queue for ephemeral heat not created in time., or tempest tests)
 
 * periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-wallaby
  inconsistent failures (tempest, undercloud, overcloud)
  
## Integration ussuri failures

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-ussuri (AB)
```
FATAL | Install - IPA client test | undercloud | error={"changed": false, "msg": "Unable to discover domain, not provided on command line"}
2022-03-04 08:55:41.688103 | fa163e8f-fef8-2748-cb2e-0000000004b9 |     TIMING | ipaclient : Install - IPA client test | undercloud | 0:00:54.408165 | 0.81s
```


* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri, periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri (AB)

nova_db_sync.log: 2022-03-04T09:18:29.920031985+00:00 stdout F Skipping execution since this is not the bootstrap node for this service.
```
SK | Check containers status
2022-03-04 09:51:28 |  [ERROR]: Container(s) which failed to be created by podman_container module:
2022-03-04 09:51:28 | ['nova_db_sync']
2022-03-04 09:51:28 |  [ERROR]: Container(s) which did not finish after 300 minutes: ['nova_db_sync']
2022-03-04 09:51:28 | 2022-03-04 09:51:28.999785 | fa163e1f-30ca-4089-242a-00000000a05b |      FATAL | Check containers status | overcloud-controller-0 | error={"changed": false, "msg": "Failed container(s): ['nova_db_sync'], check logs in /var/log/containers/stdouts/"}
2022-03-04 09:51:28 | 2022-03-04 09:51:29.000873 | fa163e1f-30ca-4089-242a-00000000a05b |     TIMING | tripleo_container_manage : Check containers status | overcloud-controller-0 | 1:09:05.953041 | 461.26s
```

* periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri (AB) - inconsistent failures, looks green

---

## component job failures

* periodic-tripleo-ci-centos-9-standalone-baremetal-master (AB) - Tempest fail

```
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_instance_port_admin_state[compute,id-f5dfcc22-45fd-409f-954c-5bd500d7890b,network,slow]
tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_update_router_admin_state[compute,id-04b9fe4e-85e8-4aea-b937-ea93885ac59f,network,slow]
tearDownClass (tempest.scenario.test_network_basic_ops.TestNetworkBasicOps)
setUpClass (tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern)
setUpClass (keystone_tempest_plugin.tests.api.identity.v3.test_service_providers.ServiceProvidersTest)
```

* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master (AB) - Node provisioning for overcloud failed

* periodic-tripleo-ci-centos-9-scenario003-standalone-common-master, periodic-tripleo-ci-centos-9-standalone-tempest-master (AB)

```
Error running playbook in directory: /tmp/tripleo-modify-image-playbook-1nir90xk
Playbook log information can be reviewed here: /var/log/tripleo-container-image-prepare-ansible.log
2022-03-04 10:29:39,817 61756 ERROR tripleo_common.image.image_uploader [  ] [192.168.24.1:8787/tripleomastercentos9/openstack-cron:afeb49ceb7917185b2d930aeb7a0785c-updated-20220304045521] Failed processing the target image
2022-03-04 10:29:39,849 61756 INFO tripleo_common.image.image_uploader [  ] [trunk.registry.rdoproject.org/tripleomastercentos9/openstack-memcached:afeb49ceb7917185b2d930aeb7a0785c] Starting upload image process
2022-03-04 10:29:39,850 61756 DEBUG urllib3.util.retry [  ] Converted retries value: 8 -> Retry(total=8, connect=None, read=None, redirect=None, status=None)
2022-03-04 10:29:39,853 61756 DEBUG urllib3.connectionpool [  ] Starting new HTTP connection (1): 192.168.24.1:8787
2022-03-04 10:29:39,855 61756 DEBUG urllib3.connectionpool [  ] http://192.168.24.1:8787 "GET /v2/ HTTP/1.1" 200 2
2022-03-04 10:29:39,856 61756 DEBUG tripleo_common.image.image_uploader [  ] http://192.168.24.1:8787/v2/ status code 200
2022-03-04 10:29:39,859 61756 DEBUG urllib3.connectionpool [  ] http://192.168.24.1:8787 "GET /v2/tripleomastercentos9/openstack-memcached/tags/list HTTP/1.1" 404 196
2022-03-04 10:29:39,859 61756 INFO tripleo_common.image.image_uploader [  ] Non-2xx: id 7e20929a3dcd948cc4cdbf250f7540fca1fcc895, status 404, reason Not Found, text <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
```

  `FATAL | Container image prepare | undercloud | error={"changed": false, "error": "Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-designate-mdns:afeb49ceb7917185b2d930aeb7a0785c failed", "msg": "Error running container image prepare: Modifying image 192.168.24.1:8787/tripleomastercentos9/openstack-designate-mdns:afeb49ceb7917185b2d930aeb7a0785c failed", "params": {}, "success": false}`

* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-compute-master (AB)

`neutron_tempest_plugin.scenario.test_basic.NetworkBasicTest.test_basic_instance`

and others which are not same every time

* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-master (AB) - inconsistent