# Ruck and Rover notes #NovDec2020
###### tags: `ruck_rover`

:::info
Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ)
**Ruck Rover - Unified Sprint #36 (NovDec 2020 timeframe)**
Dates: Nov 11 - Jan 11

Tripleo CI team ruck|rover: marios, rlandy, wes
OSP CI team ruck|rover:

Previous notes: https://hackmd.io/a_PhAlijTr-KTWHzGuf3qQ
:::

:::info
RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

Please try this out instead of cutting pasting bugs.
Give feedback to wes please
:::

[TOC]

---

## on-going issues
:::danger
## TripleO

### gate

### periodic / 3rd party

http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1

## OSP

:::

### Reviews / Fixes
::: spoiler PATCHES

:::

### Bugs reported
::: spoiler PATCHES

:::

---
:::info
add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections.
:::

## Jan 25 - Jan 29
- Tempest volume tests filures
- https://bugs.launchpad.net/tripleo/+bug/1913383

- Ara installation is failing
- https://bugs.launchpad.net/tripleo/+bug/1913046
- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/772325

- Destination directory /home/zuul/workspace/logs does not exists
- https://bugs.launchpad.net/tripleo/+bug/1913077
- https://review.opendev.org/c/openstack/tripleo-ci/+/772372

## Jan 18 - Jan 22
- Mirros are down
- https://bugs.launchpad.net/tripleo/+bug/1912177

- Tempest test [Victoria] FloatingIpTestCasesAdmin.test_two_vms_fips
- https://bugs.launchpad.net/tripleo/+bug/1912171

- Python deps error in content provider
- https://bugs.launchpad.net/tripleo/+bug/191247
- Cause: https://review.rdoproject.org/r/#/c/31573/
-

## Jan 7 - Jan 15
- Tempest tried to run on overcloud [1911696](https://bugs.launchpad.net/tripleo/+bug/1911696)

- Revert OS Tempest
- [770359](https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359)
- [770837](https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837)
## Jan 6th

- FS037-updates stein job fails consistenly.
https://bugs.launchpad.net/tripleo/+bug/1910399

Fix : https://review.opendev.org/c/openstack/tripleo-ci/+/769536

Testing : https://review.rdoproject.org/r/#/c/31487/

## Jan 5th

- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master (recent two failures)

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master

test project passed - https://review.rdoproject.org/r/#/c/31424/

## Dec 31st

- https://bugs.launchpad.net/tripleo/+bug/1909750
**C7 train**
Overcloud deployment is failing in C7 train with "stderr": "Get https://192.168.24.1:8787/v1/_ping: http: server gave HTTP response to HTTPS client"

## Dec 30th

- **ussuri**

https://bugs.launchpad.net/tripleo/+bug/1909658

periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri is failing during undercloud install with message - error checking if pulling from registry is blocked: unable to parse the registries configuration (/etc/containers/registries.conf): mixing sysregistry v1/v2 is not supported

- **stein**

Fixed https://bugs.launchpad.net/tripleo/+bug/1909671

Centos 7 based jobs are failing on TASK [Enable container-tools and disable rhel-modules] with error: /bin/sh: line 1: dnf: command not found

## Dec 29th

- **Component pipeline**
https://bugs.launchpad.net/tripleo/+bug/1909574
"periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-octavia-master/victoria/ussuri failing on tempest tests with SSHTimeout issue with error message "socket.timeout: timed out""

- **train promotion blocker**

Fixed https://bugs.launchpad.net/tripleo/+bug/1909582
"Introspection failing in train branch with message: mistral.exceptions.InvalidActionException: Failed to find action [action_name=baremetal_introspection.get_status]\n', 'introspection_attempt': 0}" [Critical,Triaged]
[High,Triaged]
## Dec 28th

- **master:**
- Fixed - https://bugs.launchpad.net/tripleo/+bug/1909445
Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n" - fixed by https://review.rdoproject.org/r/#/c/31439/

## Dec 24th

- **master:**
- FIXED https://bugs.launchpad.net/tripleo/+bug/1909191 (
Error: Unable to find a match: cephadm failing on ovb-1ctlr_1comp-featureset002-master)- Patch is up https://review.opendev.org/c/openstack/tripleo-quickstart/+/768599

- **victoria:**

---

## Dec 23rd

- **Gate:**
- https://bugs.launchpad.net/tripleo/+bug/1909118 (
[train] b"TypeError: a bytes-like object is required, not 'MagicMock'" failing openstack-tox-py36 & openstack-tox-py37)

- **master:**
- https://bugs.launchpad.net/tripleo/+bug/1909105 (
ERROR openstack Stderr: 'level=debug msg="Pull Policy for pull [PullIfNewer]"\nerror building at STEP "RUN ln -s /usr/share/openstack-tripleo-common/healthcheck/swift-account failing on build-containers-ubi-8-push master )
- Notes:
Because of change https://review.opendev.org/c/openstack/tripleo-common/+/747731 this issue started hitting periodic-tripleo-ci-build-containers-ubi-8-push job. So after reverting back the change https://review.opendev.org/c/openstack/tripleo-common/+/768264 it started passing the build-containers-ubi-8-push at https://review.rdoproject.org/r/#/c/31424/.

- https://bugs.launchpad.net/tripleo/+bug/1908976 (
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-<uuid> is failing on featureset001 and featureset035 master)
- https://review.opendev.org/c/openstack/neutron/+/753847 (Avoid race condition when processing RowEvents)
- https://review.rdoproject.org/r/31417 [DNM] Test bug/1908976
- Result: Still failing with same issue.

- **victoria:**
- **train:**
- fs020 failed in the last run of periodic integration pipeline but it passed in testproject patch : https://review.rdoproject.org/r/#/c/31425/ hope this will help for promotion.

---

## Dec 22nd

- **master:**
- https://bugs.launchpad.net/tripleo/+bug/1908976 (
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-<uuid> is failing on featureset001 and featureset035 master)
- https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/768163 (DNM: Revert "Fix lower constraints")
- This revert didn't work so Abandoned patch.
- https://review.opendev.org/c/openstack/neutron/+/753847 checked with this fix as well but still facing same issue.
- https://review.rdoproject.org/r/31417 [DNM] Test bug/1908976

- https://bugs.launchpad.net/tripleo/+bug/1907193 (
package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering) - failing featureset050-upgrades-master, multinode-oooq-container-updates-master and featureset037-updates-master

- https://bugs.launchpad.net/tripleo/+bug/1909008 (
tempest.lib.exceptions.PreconditionFailed: Precondition Failed on standalone-full-tempest-api-master)
- https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768204

- **victoria:**
- https://bugs.launchpad.net/tripleo/+bug/1908982 (
[victoria] tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.115 via SSH timed out. failing on standalone-full-tempest-scenario-victoria)
- https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768152

## Dec 21st

- **master:**
- periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master and scenario000-multinode-oooq-container-updates-master is failing = https://bugs.launchpad.net/tripleo/+bug/1907193
- Error log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master/6d3f5b1/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz
- Error

      Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n  - package podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64 requires podman = 2.0.5-5.module_el8.3.0+512+b3b58dca, but none of the providers can be installed 

- ovb-1ctlr_1comp-featureset002-master is failing
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/495856c/logs/undercloud/home/zuul/build-err.log.gz

- featureset035-master is failing because of some tempest test failure
- https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/8f42da5/logs/undercloud/var/log/tempest/stestr_results.html.gz

- master tempest failures .. start debugging from https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/43f38e4/logs/overcloud-controller-0/var/log/extra/errors.txt.txt.gz

- **Victoria:**
- standalone-full-tempest-scenario-victoria is failing = tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.105 via SSH timed out.
- https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/2f22bf0/logs/undercloud/var/log/tempest/stestr_results.html.gz

## Dec 18th
#### AMOL's notes
- Logs:
- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8f4/767679/1/check/tripleo-ci-centos-8-containers-multinode/8f43a97/job-output.txt
- ssh key change
- http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22REMOTE%20HOST%20IDENTIFICATION%20HAS%20CHANGED!%5C%22
- Job tripleo-ci-centos-7-standalone-upgrade-train
- Do we need to keep it around?
- https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-train

## Dec 17th

## Dec 16th

## Dec 15th
- Bugs
- https://bugs.launchpad.net/tripleo/+bug/1908246 Victoria upgrade jobs aren't using content provider passed dlrn hash tag. [Patch](https://review.opendev.org/c/openstack/tripleo-quickstart/+/767112)
- https://bugs.launchpad.net/tripleo/+bug/1908274 tox-py36-ci-config job is broken.

## Dec 14th

- Bugs:
- https://bugs.launchpad.net/tripleo/+bug/1908054 Ansible collect logs failing.
-

### summary of status.. shitty week last week..
pypy depsolv, ubi-container in check, centos 8.2 - 8.3, content-provider patches all contributing to failures upstream atm. We're through most issues atm. OVB in check is full red, but as I said looks to be coming back to green via periodic promotions.

* pypy - lower-constraints https://review.opendev.org/q/topic:%2522tripleo-lower%2522+(status:open+OR+status:merged)
* ubi-8 - container-builds e.g. https://review.opendev.org/c/openstack/tripleo-ci/+/766839
* c8-train -> ussuri upgrade jobs failing on pacemaker mismatch - requires promotion to fix or method to inject change to content-providers.

* content-provider ip failures https://bugs.launchpad.net/tripleo/+bug/1907657
* overcloud-full-hardened moved to nv until we promote all c8 branches https://bugs.launchpad.net/tripleo/+bug/1907457

* master appears to be back and healthy enough for a promotion. fs01/035 starting to pass again. wes to make sure master is promoted soon.

* c8-train is the most import and a problem. container-builds are failing and tripleo-common needs patches to help us diagnose. See https://review.rdoproject.org/r/#/c/31336/

* need to land to fix ovb 3rd party https://review.opendev.org/c/openstack/paunch/+/766779 https://bugs.launchpad.net/tripleo/+bug/1907833

## Dec 10th
Need help landing https://review.opendev.org/c/openstack/tripleo-common/+/766516

Please review: https://review.opendev.org/c/openstack/tripleo-ci/+/766621

## Dec 8th
Please focus on unblocking: https://review.opendev.org/#/q/owner:%22amolkahat+%253Camolkahat%2540gmail.com%253E%22+branch:stable/ussuri

gate - release issue
https://bugs.launchpad.net/tripleo/+bug/1907122
https://review.opendev.org/q/topic:%22release-aware-gate%22+(status:open%20OR%20status:merged)

Not sure if https://bugs.launchpad.net/tripleo/+bug/1907006 is related to the above issue or not

## Dec 7th
new bugs in https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

---
## Dec 4th & earlier ongoing things

#### https://bugs.launchpad.net/tripleo/+bug/1906769 periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master failing intermitently with paramiko.ssh_exception.AuthenticationException: Authentication failed

#### https://bugs.launchpad.net/tripleo/+bug/1883843 tripleo-ci-centos-8-scenario007-multinode-oooq-container is having intermittently timeouts

#### https://bugs.launchpad.net/tripleo/+bug/1906486 FS01 ovb job fails with failed to stat() /home/zuul/overcloud-full.raw

#### https://bugs.launchpad.net/tripleo/+bug/1905386 Centos 7 train jobs failing with pymod2pkg requiring python 3.6

#### https://bugs.launchpad.net/tripleo/+bug/1903508 periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler
* just waiting on train cherrypick https://review.opendev.org/c/openstack/tripleo-common/+/763721
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903882 tripleo-ci-centos-8-content-provider failed on TASK [container-build : Pull failed containers from RDO registry] with error was: 'dict object' has no attribute 'split'
:::spoiler
* waiting on fix to merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/762400/
:::

---
# stop here eveything still open is duplicated above this line
---

## ~~Dec 3rd~~
* inconsistent timeout issue for updates job : (https://bugs.launchpad.net/tripleo/+bug/1883843)
https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates&branch=stable%2Ftrain
## Dec 2nd
* Scenario 01/04 : Fix was merged (https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764140/)
* New issue reported consistent failure : https://bugs.launchpad.net/tripleo/+bug/1906486

## Dec 1st
* Random timed_out on this patch : https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764140

## Nov 30th
* wes - to check gate failures.. https://zuul.openstack.org/status
* FAK, all upstream jobs failing https://bugs.launchpad.net/tripleo/+bug/1906265, patch up

## Nov 27th
- https://bugs.launchpad.net/tripleo/+bug/1905942 Ussuri jobs failing to lookup hiera
- 13 gate failures UTC 20:00
- bug spreadsheet updated

## Nov 26th
- Nothing much, a lot of stuff is broken right now
- 90+ gate failures

## Nov 25th
- https://bugs.launchpad.net/tripleo/+bug/1905386
- potential fix: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232
- https://bugs.launchpad.net/tripleo/+bug/1905536 = Ceph wrong role name
- https://bugs.launchpad.net/tripleo/+bug/1905592 = quay.io issue
- https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764217
- https://review.opendev.org/c/openstack/tripleo-ci/+/764218

## Nov 24th

- https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/764055 Fix stackviz for failed tempest runs [WIP] [NEW]
- https://bugs.launchpad.net/tripleo/+bug/1905460

- periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master
- random failures: TestVolumeBootPattern, NetworkSecGroupTest, TestShelveInstance (dstat looks good, no indication of resource consumption problems)
- testing here https://review.rdoproject.org/r/31193

- https://bugs.launchpad.net/tripleo/+bug/1905348 has a fix https://review.opendev.org/c/openstack/python-tripleoclient/+/763919

- https://bugs.launchpad.net/tripleo/+bug/1905421 - Tempest error
- https://bugs.launchpad.net/tripleo/+bug/1905418 - Tempest error
- Error in train jobs due python 2.7 and pymod2pkg
- https://bugs.launchpad.net/tripleo/+bug/1905386
- attempt to bump venv to py3: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232 Switch to python3 venv by default

- ~~introspection bug 1904936 fix~~
- ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/763972~~

## Nov 23th

- ussuri gate jobs still failing
- ussuri repo-setup
- introspection bug 1904936 is still a thing
- certificate tempest failure on victoria - 1903508 (waiting to merge)
- cold migration bug 1903033 - waiting to merge
- centos-8-scenario002-standalone ssh time out bug 1904223 (unresolved)
- affects victoria promotion?
- ~~ensure private network exists bug 1902048 >> FIXED~~
- ~~fs037-stein >> FIXED~~
- ~~bug 1905036 (scen10 ovn) >> FIXED~~
- ~~pulling containers from docker.io (stein) >> FIXED~~

(details below...)

## Nov 20th

#### ~~new/transient/not filed yet~~
:::spoiler ~~periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently: >> fixed fs037-stein is green: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein~~
* https://logserver.rdoproject.org/25/25325/67/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/d358af8/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz
* Should have been fixed by: https://review.opendev.org/#/c/762871/
* marios: added https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18 i think we need stein cherrypick @ https://review.opendev.org/#/c/763546/ & testing with rlandy testproject review https://review.rdoproject.org/r/#/c/25325/
:::

:::spoiler ussuri gate is blocked on our content provider stuff merging STILL!
* https://review.opendev.org/#/c/757544/
* now blocked on https://bugs.launchpad.net/tripleo/+bug/1905034
* 16:29 <@mwhahaha> marios|ruck, weshay|ruck: sooo what are we doing about the fact that ussuri CI is completely blocked on tripleo-common and THT?
:::

:::spoiler ~~scen10-ovn~~
https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect
15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/
:::

:::spoiler ~~pulling containers from docker.io~~
https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io
* bug is fixed but the fix needs to go back to stein https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18
* waiting on stein cherrypick https://review.opendev.org/#/c/763546/
:::

:::spoiler ussuri repo-setup
https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0
waiting to merge: https://review.opendev.org/c/openstack/tripleo-quickstart/+/763633
:::

#### https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect
:::spoiler info
15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io
:::spoiler info
* bug is fixed but the fix needs to go back to stein https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18
* waiting on stein cherrypick https://review.opendev.org/#/c/763546/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0

---

## Nov 19th

#### https://bugs.launchpad.net/tripleo/+bug/1904936 openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node

#### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler info
* ~~https://review.opendev.org/#/c/762497/~~
* https://review.opendev.org/#/c/763185/ - victoria cherry-pick
* blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs)
* victoria promotion blocker
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'
:::spoiler info
* currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini
:::

#### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

#### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]
:::spoiler info
* temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec
* waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

#### new/transient/not filed yet
:::spoiler chkumar working on rdo->vexx remaining jobs
09:32 < chandankumar> marios: sshnaidm I am tracking rdocloud nodeset removal here https://etherpad.opendev.org/p/remove-rdocloud-jobs
:::
:::spoiler still chasing a queens promotion
* latest buildset https://review.rdoproject.org/zuul/buildset/14a3df4f2f4544a3b6928e1326f0b19b
* rlandy posted yesterday https://review.rdoproject.org/r/#/c/25325/ - some green from that
* still missing fs18/20 http://38.102.83.109/promoter_logs/centos7_queens.log
2020-11-19 09:10:53,087 22394 INFO promoter Skipping promotion of centos7-queens from tripleo-ci-testing to current-tripleo, missing successful jobs: [u'periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens', u'periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens']
* posted https://review.rdoproject.org/r/31145 for missing ^^^ let's see
* it is a good hash - from 18th https://trunk.rdoproject.org/centos7-queens/83/37/83378d81927e5f1e0c1b7e85cf7c4258c4e32356_7268d992/
:::
:::spoiler Merged Update rdo-openvswitch in ussuri to support ovs/ovn2.13 NFV SIG https://review.rdoproject.org/r/30774
12:07 < ykarel> marios|ruck, fyi ^ in case u see something wron gin ussuri jobs, let us know
12:07 < ykarel> we tested multiple jobs and it went good, but just in case if there is some issue
:::

:::spoiler https://review.rdoproject.org/r/#/c/30776/ - network component still very red

:::

:::spoiler periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently:
https://logserver.rdoproject.org/25/25325/67/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/d358af8/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz

Should have been fixed by: https://review.opendev.org/#/c/762871/
:::

---

## ~~Nov 18th~~

:::info
RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

Please try this out instead of cutting pasting bugs.
Give feedback to wes please
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler info
* ~~https://review.opendev.org/#/c/762497/~~
* https://review.opendev.org/#/c/763185/ - victoria cherry-pick
* blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs)
* victoria promotion blocker
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'
:::spoiler info
* currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini
:::

#### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

#### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]
:::spoiler info
* temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec
* waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE~~
:::spoiler info
* Moved fs19 and fs18 to a single playbook. May be fixed
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source~~
:::spoiler info
blocks victoria gate
fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112
:::

#### new/transient/not filed yet
:::spoiler ussuri *just* missed promotion
* ussuri good buildset https://review.rdoproject.org/zuul/buildset/87a37f60460c4ffd9e2aee3f49256efb
* missing periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri so
* posted https://review.rdoproject.org/r/31132
* job is generally green https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri
:::
:::spoiler Vexx mirrors outdated
* http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos7-stein/tripleo-ci-testing/
* baseurl=https://trunk.rdoproject.org/centos7-stein/d7/87/d787d250cd2a93bf6368cd433bcb2c8b306c8edd_8d9f1861 (old hash) causes failures when pulling containers
* https://review.opendev.org/#/c/763233/
* https://review.opendev.org/763401
* Wes requests python script to pull from trunk and retry
:::

---

## ~~Nov 17th~~

#### https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source
:::spoiler info
blocks victoria gate
fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler info
* ~~https://review.opendev.org/#/c/762497/~~
* https://review.opendev.org/#/c/763185/ - victoria cherry-pick
* blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs)
* victoria promotion blocker
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'
:::spoiler info
* currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini
:::

#### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

#### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]
:::spoiler info
* temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec
* waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE
:::spoiler info
* Moved fs19 and fs18 to a single playbook. May be fixed
:::

#### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]~~
:::spoiler info
reopened bug
~~fix at https://review.rdoproject.org/r/#/c/31114/~~
tested at https://review.rdoproject.org/r/#/c/25325/
~~@marios pls confirm that the upgrades/updates jobs are ok with the one playbook for multinode.~~
:::

#### new/transient/not filed yet

:::spoiler ~~undercloud-upgrade & standalone-upgrade victoria problem~~ filed https://bugs.launchpad.net/tripleo/+bug/1904565
*

~~https://7085ed736f9157d6d701-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf1.rackcdn.com/762658/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/ad84735/logs/undercloud/home/zuul/undercloud_upgrade.log tripleo_common.image.exception.ImageNotFoundException: Not found image: http://23.253.57.32:5001/v2/tripleovictoria/openstack-keystone/manifests/7f974e10d7184d5fc45445a3073333bd"], "stdout": "", "stdout_lines": []}~~

:::
:::spoiler vexx infra/introspection issues
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/835c3e6/logs/undercloud/var/log/extra/baremetal_list.txt.gz
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/b99e7f1/job-output.txt
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/2726291/job-output.txt
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/970a3ea/job-output.txt
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/72f523b/job-output.txt
:::

#### fs050 victoria jobs are failing:
:::spoiler
Related fixes:
https://review.opendev.org/#/c/763097/ Make victoria upgrades job non-voting until fixed
~~https://review.rdoproject.org/r/31120 Add undercloud fs050 upgrades to master/v/u periodic ~~
@marios : your thoughts here?
where though? are you referring to upstream undercloud upgrade job fixed by https://review.opendev.org/#/c/763005/ ? or periodic job? can't be periodic since you're adding it with ~~https://review.rdoproject.org/r/31120~~
:::

---

## ~~Nov 16th~~

#### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler info
* https://review.opendev.org/#/c/762497/
* victoria promotion blocker
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'
:::spoiler info
* currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini
:::

#### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

#### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]
:::spoiler info
* temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec
* waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

#### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

#### ~~https://bugs.launchpad.net/tripleo/+bug/1904418 ( IN PROGRESS) buildimage centos7 jobs are failing to fetch mirror info 404~~
:::spoiler info
* should be fixed, watch https://static.opendev.org/mirror/logs/rsync-mirrors/
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1904483 (Require attention)centos-7 overcloud image build jobs are failing on pre-install.d/03-centos-rt - rt repo no longer exists~~
:::spoiler info
shoudl be fixed by https://review.opendev.org/#/c/762935 (will need cherry-pick to train, stein, rocky)
Related fix on removing rt-repo: https://review.opendev.org/#/c/762926/
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'~~
:::spoiler info
https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498)
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests.~~
:::spoiler info
* job is just taking too long... removing tempest after deployment https://review.opendev.org/#/c/762674/
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762871/~~

---

## ~~Nov 13th~~

#### https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'
:::spoiler info
https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498)
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."
:::spoiler info
* https://review.opendev.org/#/c/762497/
* victoria promotion blocker
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'
:::spoiler info
* currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini
:::

#### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

#### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]
:::spoiler info
* temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec
* waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

#### https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests.
:::spoiler info
* job is just taking too long... removing tempest after deployment https://review.opendev.org/#/c/762674/
:::

#### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

#### https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762547/
Additional fix: https://review.opendev.org/#/c/762871/

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903498 tripleo-ci-centos-8-undercloud-upgrade-ussuri is failing RuntimeError: Update extra packages failed: b'sudo: yum: command not found\n~~
:::spoiler info
* waiting for merge https://review.opendev.org/#/c/761892/
:::

#### ~~https://bugs.launchpad.net/tripleo/+bug/1904214 centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7 * centos7 promotion blocker~~

#### ~~https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]~~

.

:::spoiler ~~promotion status~~
#### victoria promotion:
Last promotion 10th Nov
Last buildset - https://review.rdoproject.org/zuul/buildset/f8e8812c49c14d2c8915af307d85731c
* victoria promotions blocked on https://bugs.launchpad.net/tripleo/+bug/1903508/comments/6

* periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria(In criteria) & periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria, Bug: https://bugs.launchpad.net/tripleo/+bug/1903508 Cixed to security team, Provided reproducer to martin today, Patch is up - https://review.opendev.org/#/c/762497

* ~~Master/Ussuri/Victoria promotion blocker
https://bugs.launchpad.net/tripleo/+bug/1900949
Ovb jobs are failing on master branch, imported nodes are not transitioning to manageable state with ‘Error: Unable to establish IPMI v2 / RMCP+ session\n’~~
* rdocloud infra issue, we moved jobs to vexx host

#### ussuri promotion:
Last promotion 11/13

#### train promotion:

* new/transient/not filed yet:
* ~~master scenario2 standalone failures https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario002-standalone&branch=master * blocks https://review.opendev.org/#/c/762497/4~~
* filed @ https://bugs.launchpad.net/tripleo/+bug/1904223

* c8 should promote again today https://review.rdoproject.org/zuul/buildset/1f410357edb34516ab3ce4cadffa491f
* c7 is in trouble: (rlandy investigating)
https://bugs.launchpad.net/tripleo/+bug/1904214 -
centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7

<amoralej> yes, 7.9.20009 was pushed yesterday to mirrors

<rlandy|rover> that's probably it

<amoralej> let me dig into it

<jpena> amoralej: that server mounts opendev's AFS, just like the rest of the mirrors we have in opendev
<jpena> looking at https://grafana.opendev.org/d/HxrNXt2Gk/afs?orgId=1, it looks like the mirror was synced 1 hour ago
<jpena> we might check where does it sync from, maybe that one seed does not have 7.9 yet
<jpena> so, what I see in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L51 is that the opendev AFS mirrors sync from mirror.lstn.net/centos
<jpena> looking at http://mirror.lstn.net/centos/7/os/x86_64/Packages/?C=M;O=D, it looks like they don't have the 7.9 content yet

Blocking Sagi's patch will solve the issue https://review.opendev.org/#/c/761892/ for https://bugs.launchpad.net/tripleo/+bug/1903498

2020-11-13 12:22:22.839674 | primary | Error: Package: glibc-2.17-307.el7.1.i686 (base)
2020-11-13 12:22:22.839814 | primary |            Requires: glibc-common = 2.17-307.el7.1
2020-11-13 12:22:22.839848 | primary |            Installed: glibc-common-2.17-317.el7.x86_64 (@base)
2020-11-13 12:22:22.839870 | primary |                glibc-common = 2.17-317.el7
2020-11-13 12:22:22.839898 | primary |            Available: glibc-common-2.17-307.el7.1.x86_64 (base)
2020-11-13 12:22:22.839920 | primary |                glibc-common = 2.17-307.el7.1
2020-11-13 12:22:22.839943 | primary | Error: Package: python3-devel-3.6.8-13.el7.x86_64 (base)
2020-11-13 12:22:22.839989 | primary |            Requires: python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 12:22:22.840015 | primary |            Installed: python3-libs-3.6.8-17.el7.x86_64 (@base)
2020-11-13 12:22:22.840037 | primary |                python3-libs(x86-64) = 3.6.8-17.el7
2020-11-13 12:22:22.840062 | primary |            Available: python3-libs-3.6.8-13.el7.x86_64 (base)
2020-11-13 12:22:22.840083 | primary |                python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 12:22:22.840125 | primary |  You could try using --skip-broken to work around the problem
2020-11-13 12:22:23.195322 | primary |  You could try running: rpm -Va --nofiles --nodigest

#### rocky promotion:
* rekicked on vexx https://review.rdoproject.org/r/#/c/29969/16/.zuul.yaml
* blocked on https://bugs.launchpad.net/tripleo/+bug/1904214

#### stein promotion:
* promoted yesterday - leaving for weekend run

#### queens promotion:
* testproject there (rdo) https://review.rdoproject.org/r/30906 per https://bugs.launchpad.net/tripleo/+bug/1903996/comments/1 but failing so try vexx
* testproject there (vexx) https://review.rdoproject.org/r/#/c/25325/
* blocked on https://bugs.launchpad.net/tripleo/+bug/1904214
:::
---
## ~~Nov 12th~~
### TripleO

#### Headsup

<amoralej> we are updating ovs/ovn 2.13 for master and victoria today. We've gated it with different jobs so i expect to go smooth but let us know if you find any issue

* [Blocking check jobs]tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'
https://bugs.launchpad.net/tripleo/+bug/1903980
* Sagi patch will solve the issue https://review.opendev.org/#/c/761892/

* [Promotion blocker for c7 train/stein/rocky]Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]
https://bugs.launchpad.net/tripleo/+bug/1903961
* Hit on rerun via testproject too - Need to investigate

* "Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests."
https://bugs.launchpad.net/tripleo/+bug/1903993
* Need to investigate

* Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE
https://bugs.launchpad.net/tripleo/+bug/1903996
* Need to investigate
* testproject running - https://review.rdoproject.org/r/#/c/25325/

* Ran rocky /stein promotion ...
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-5

stein should promote.
rocky failed container builds again. need to look at increasing the timeout there