# Ruck and Rover notes #32
###### tags: `ruck_rover`

:::info
Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ)
**Ruck Rover - Unified Sprint #32
Dates: July 30th - August 19

Tripleo CI team ruck|rover: Arx Cruz (arxcruz) Amol Kahat (akahat)
Downstream CI team ruck|rover: TBD

Vex/rdocloud introspection: vexx: https://bit.ly/3kfhIOb rdo-cloud: https://bit.ly/2Xu4jIG

Previous notes(sprint #31): https://hackmd.io/QnprH9-yRTi6uWlEfaahoQ?both
Next notes(sprint #32): none - this sprint 32 is current one
:::

[TOC]

---

## on-going issues
:::danger
## TripleO

* Stein and Rocky tempest container failures.

### gate

### periodic / 3rd party

## OSP

:::

### Reviews / Fixes
::: spoiler PATCHES

- https://review.opendev.org/748160

:::

### Bugs reported
::: spoiler PATCHES
- https://bugs.launchpad.net/tripleo/+bug/1893173
- https://bugs.launchpad.net/tripleo/+bug/1893192
- https://bugs.launchpad.net/tripleo/+bug/1892837
-
-
:::
---
## 09/09/2020
- triplo-ci-centos-7-standalone-upgrade-train job is failing [LP: 1895005](https://bugs.launchpad.net/tripleo/+bug/1895005)
- https://bugs.launchpad.net/tripleo/+bug/1894953 - tempest failure, seems to be nova bug
- https://review.opendev.org/#/c/750586/ Added on skiplist

## 09/08/2020
- Python tripleoclient -tox-py27 stable/train broken [1894841](https://bugs.launchpad.net/tripleo/+bug/1894841)
## 09/07/2020
- ~~https://bugs.launchpad.net/tripleo/+bug/1894664 - Train containers job failing~~
- CentOS 8 train jobs are failing. [Log URL](https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-common-train/ccb13fb/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz)

    2020-09-04 07:24:26 | Host 192.168.24.16 not found in /home/zuul/.ssh/known_hosts
    2020-09-04 07:24:26 | Overcloud configuration failed.

## 09/04/2020
- Master Promotion
- Extra openstack- prefix.. caused master promotion failure.
- http://paste.openstack.org/show/797455/
- AI: Wes

## 09/03/2020
- Upstream is red, https://bugs.launchpad.net/tripleo/+bug/1893339
- https://review.opendev.org/#/c/748609/
- Error:

    error={"changed": false, "error": "Not found image: https://registry-1.docker.io/v2/tripleomaster/centos-binary-swift-object/manifests/df855c291ed5531d780dbba4a768e1f3", "msg": "Error running container image prepare: Not found image: https://registry-1.docker.io/v2/tripleomaster/centos-binary-swift-object/manifests/df855c291ed5531d780dbba4a768e1f3", "params": {}, "success": false}

## 09/02/2020
- Nothing news so far

## 09/01/2020
- Same errors still happening

## 08/31/2020

- Same errors as last week

### Tripleo
- Promotion Blocker
- https://bugs.launchpad.net/tripleo/+bug/1893618 - Tempest test_shelve_unshelve_server failing constantly
- https://bugs.launchpad.net/tripleo/+bug/1893621 - Could not establish a connection to the Zaqar websocket
- https://review.opendev.org/#/c/748673/
- Ussuri had condition race in podman, Sagi already backport his patch
- https://review.opendev.org/#/q/72f0e09019ee13ad306287a8f60faba30d4fb60e
- Rocky and Stein tempest container issue
- https://review.opendev.org/#/c/748955/ - Thanks to Yatin
---
## 08/28/2020

### Tripleo
- Promotion Blocker:
- https://bugs.launchpad.net/tripleo/+bug/1893339
- https://review.opendev.org/#/c/748974/ - Added test on skiplist

## 08/27/2020

### Tripleo
- https://bugs.launchpad.net/tripleo/+bug/1893173 -
Error while evaluating a Resource Statement
- https://review.opendev.org/#/c/748360/

- https://bugs.launchpad.net/tripleo/+bug/1893192 - octavia_tempest_plugin.tests.scenario.v2.test_listener.ListenerScenarioTest failed in scen010 ovn

### Others

---
## 08/26/2020

Same issues from yesterday happening again, that's the reason there's nothing here today

### Tripleo

- Patches to review:
- https://review.opendev.org/748160
-

### Other

---
## 08/25/2020

### TripleO
- Timeout on [periodic-tripleo-ci-centos-8-scenario010-standalone-master](https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-standalone-master/ae4abe7/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz)
- Happened only today, should a bug be oppened? (check with wes)

- Error in tempest-api job https://bugs.launchpad.net/tripleo/+bug/1892837
- https://review.opendev.org/#/c/747868/ Added test on skiplist

### Other

---
## 08/24/2020

### TripleO
- Centos8-scenario012 failing on stable/train: Tempest failure
- Test is added in skiplist for [master](https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-scenario012-standalone&branch=stable%2Ftrain/), [train and usuri](https://review.opendev.org/#/c/747619/)
- [LP](https://bugs.launchpad.net/tripleo/+bug/1890424)
- Periodic Scenario 010 ovn provider job failing with ovn provider is not enabled https://bugs.launchpad.net/tripleo/+bug/1892682

- Job [tripleo-ci-centos-7-scenario009-multinode-oooq-container](https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario009-multinode-oooq-container) continuously failing.

- Race condition in podman https://bugs.launchpad.net/tripleo/+bug/1892701
- https://review.opendev.org/747685 might help to avoid race condition

- Upgrade jobs failing in create container https://bugs.launchpad.net/tripleo/+bug/1892702
### Others

- ~~Pre-commit bug https://github.com/pre-commit/pre-commit/issues/1575~~
- ~~Did not fix the problem, opening another one~~
- ~~https://github.com/pre-commit/pre-commit/issues/1578~~
- ~~https://review.opendev.org/#/c/747649/ pin pre-commit to < 2.7.0~~

- Jobs getting timed out: This issue is seen in few jobs.
- ERROR: RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
- https://643169a2915789dbd696-0cff35b3d101f5c13616cb96b9f1415f.ssl.cf5.rackcdn.com/746219/6/gate/tripleo-ci-centos-8-undercloud-containers/48539e2/job-output.txt
- [periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri](https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-ussuri/59424a6/job-output.txt)
- [tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates](https://8442fc48a58b3d85474c-27618e467ab9b983fdfb7a89cc8b48ca.ssl.cf1.rackcdn.com/745327/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/e702a56/job-output.txt)

### OSP
---
# 08/21/2020

### tripleo

handoff -> wes

### osp

## Bugs from sprint 31
### Patches in check/gate

* https://bugs.launchpad.net/tripleo/+bug/1892008 -
periodic OVB train centos 7 failing overcloud configuration (config download) Invalid cross-device link

* https://bugs.launchpad.net/tripleo/+bug/1891372 -
rocky periodic jobs are failing with " Error: image tripleorocky/centos-binary-tempest:9801dc7461cbd6cbd73868e72e74d21d586c6708_fbb4de96-updated-20200812131025 not found"

### Bugs still seen

* https://bugs.launchpad.net/tripleo/+bug/1892326 -
Jobs failing with RETRY_LIMIT with primary | /bin/sh: line 1: git: command not found at prepare-workspace-git : Clone cached repo to workspace

### Bugs no activity yet

* https://bugs.launchpad.net/tripleo/+bug/1892169 -
periodic centos 8 train FS20 tempest fails test_server_connectivity_resize (in fs020 not part of train promotion)

### Keep an eye on
* https://review.opendev.org/#/c/747129

:::info
add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections.
:::