# Ruck Rover - 13th May 2022 - 19th May 2022

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/zjCkCnVeQ06bnBq836iF8g
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

---

## Thursday 19 May

rlandy to check below failures:-
~~~
periodic-tripleo-ci-centos-9-scenario004-standalone-master TIMED_OUT 3h 09m 09s
periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master FAILURE 2h 28m 48s
periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master
~~~

* periodic-tripleo-ci-centos-9-scenario004-standalone-master openstack/tripleo-ci SUCCESS in last two runs
* periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master openstack/tripleo-ci master openstack-periodic-integration-main 2 hrs 16 mins 44 secs 2022-05-19 09:25:35 SUCCESS
* periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master - failed - will rerun
* reopened https://bugs.launchpad.net/tripleo/+bug/1973223 for scenario010 kvm - job now passed in rerun

#### Check/gate
* https://review.opendev.org/c/openstack/tripleo-ci/+/835460/

#### Master integration line - last_promotion=2022-05-16

Rerunning failed jobs: https://review.rdoproject.org/r/c/testproject/+/39357
https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/42965

##### New bug:

[Master overcloud deployment failing with 'Node' object has no attribute 'uuid'](https://bugs.launchpad.net/tripleo/+bug/1974134)
* reverting patch that caused this issue: https://review.opendev.org/c/openstack/tripleo-ansible/+/842399 (-W)
* Holding above for proper fix:
* https://review.opendev.org/c/openstack/tripleo-ansible/+/842473
* https://bugs.launchpad.net/tripleo/+bug/1974244 openstack-tox-py36 is failing on tripleo-heat-templates master tests - oslo-log (possible mirror issue)

#### Master Component line
* tripleo (last_promotion=2022-05-16)- ovb fs001 failed - new [failure](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master/da3e7d7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz) - rerunning to confirm if failure is consitent
* security (last_promotion=2022-05-13)- ovb fs039 failed - failing with known [bug](https://bugs.launchpad.net/tripleo/+bug/1973863) - should be resolved now
* network (last_promotion=2022-05-18)- ovb fs001 failed -
tempest [failure](https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master/a77865e/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz)

Rerunning failed jobs [here](https://review.rdoproject.org/r/c/testproject/+/39420)

#### Wallaby c9 integration line - last_promotion=2022-05-17
* Only fs001 - failing - need to check

Rerunning here https://review.rdoproject.org/r/c/testproject/+/26273
#### wallaby c9 component line - tripleo
* tripleo(last_promotion=2022-05-18) - container-multinode failing - tempest [failure](https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-containers-multinode-tripleo-wallaby/edb08a9/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz) - rechecking to confirm if conistent
* https://trunk.rdoproject.org/api-centos9-wallaby/api/civotes_agg_detail.html?ref_hash=c736812dd676ac4f0b15f70f9bc668cb
Promotable - internal fs001 in tempest now - rest passed in rerun

Rerunning failed jobs [here](https://review.rdoproject.org/r/c/testproject/+/39420)

#### Wallaby c8 integration line - last_promotion=2022-05-16
* fs001/fs035 - failing - need to check
* promoted
#### wallaby c8 component line - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/26273

#### Victoria integration line - last_promotion=2022-05-16

* [Rerun 1](https://review.rdoproject.org/r/c/testproject/+/26273)
* [Rerun 2+3](https://review.rdoproject.org/r/c/testproject/+/42978)

Bunch of jobs failed:
* ~~[periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria&skip=0) fails on different tempest errors all the time,
[sometimes](https://logserver.rdoproject.org/73/26273/21/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria/b79f4af/logs/undercloud/var/log/tempest/stestr_results.html.gz) it fails on [LP bug #1960310](https://bugs.launchpad.net/tripleo/+bug/1960310)~~
* ~~[periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria&skip=0) fails on different tempest and non-tempest errors all the time~~
* ~~[periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria&skip=0) dito~~
* ~~[periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria&skip=0) dito~~

* ~~Jobs keep failing because of MySQL errors in various/random services,
[example](https://logserver.rdoproject.org/78/42978/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-victoria/5aca676/logs/undercloud/var/log/containers/cinder/cinder-volume.log.txt.gz)~~

* ~~Job `periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria` might be failing because of an real issue:~~

  2022-05-18T05:59:27.276336835+00:00 stderr F 2022-05-18T05:59:27Z|00021|patch|WARN|Bridge 'br-tenant' not found for network 'tenant'

~~[log example 1](https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/986574c/logs/undercloud/var/log/containers/stdouts/ovn_controller.log.txt.gz),
[log example 2](https://logserver.rdoproject.org/78/42978/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/b31d491/logs/undercloud/var/log/containers/stdouts/ovn_controller.log.txt.gz).
But maybe it is also related to MySQL issues mentioned above because errors are showing up here as well:
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/986574c/logs/undercloud/var/log/containers/neutron/server.log.txt.gz~~

~~Same for [periodic-tripleo-ci-centos-8-standalone-upgrade-victoria](https://logserver.rdoproject.org/78/42978/1/check/periodic-tripleo-ci-centos-8-standalone-upgrade-victoria/dc8ddd6/logs/undercloud/var/log/containers/neutron/server.log.txt.gz).~~

* Jobs `periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria` and
`periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria` failed twice with:

  2022-05-21 08:44:18.734207 | primary | ERROR: Cannot install oslo.utils because these package versions have conflicting dependencies.
  2022-05-21 08:44:18.734262 | primary |
  2022-05-21 08:44:18.734277 | primary | The conflict is caused by:
  2022-05-21 08:44:18.734291 | primary |     The user requested oslo.utils
  2022-05-21 08:44:18.734303 | primary |     The user requested (constraint) oslo-utils===4.6.1
  2022-05-21 08:44:18.734314 | primary |
  2022-05-21 08:44:18.734324 | primary | To fix this you could try to:
  2022-05-21 08:44:18.734335 | primary | 1. loosen the range of package versions you've specified
  2022-05-21 08:44:18.734347 | primary | 2. remove package versions to allow pip attempt to solve the dependency conflict
  2022-05-21 08:44:18.734361 | primary |
  2022-05-21 08:44:18.734373 | primary | ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

https://logserver.rdoproject.org/78/42978/7/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria/8539a90/job-output.txt

#### victoria c8 component line - All green

Rerunning here: https://review.rdoproject.org/r/c/testproject/+/26273

#### Train integration line - last_promotion=2022-05-16 - Green (No new hash when I checked this morning)
#### Train component line - Only tripleo lagging
* ~~tripleo~~ - passed in rerun

Analysis, 18th and 16th failed on same tempest test - 17th had different tempest failure - rerunning to confirm if issue is consistent

~~~
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/319b011/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
https://logserver.rdoproject.org/73/26273/20/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/1de3186/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
https://logserver.rdoproject.org/openstack-component-tripleo/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train/58fce8a/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
~~~

Rerunning failed jobs [here](https://review.rdoproject.org/r/c/testproject/+/39420)

#### 16.2 integration line - last_promotion==2022-05-16 - Green (No new hash when I checked this morning
#### 16.2 component line - All green

#### 17/8 integration line - last_promotion=2022-05-18
* fs035 failed in last run - rerunning [here](https://code.engineering.redhat.com/gerrit/c/testproject/+/201382)
#### 17/8 component line - All green

#### 17/9 integration line - last_promotion=2022-05-18 - All green
* ~~fs035 failed in last run~~ passed in rerun
#### 17/9 component line - All green

## Wednesday 18 May (and earlier ongoing things tracked here)

* Check/gate
[Master fs035 and fs064 failing on overcloud deploy on task "Render keystone-db-sync container definitions" with error - can only concatenate list (not "AnsibleUnicode") to list. Unexpected templating type error occurred on (image: "{{ tripleo_keystone_image }}"](https://bugs.launchpad.net/tripleo/+bug/1973863 )

Master integration line
* [Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503](https://bugs.launchpad.net/tripleo/+bug/1973568)
* Need to revert tempest

* [Master Sc010-kvm job is failing on octavia related tempest test: octavia_tempest_plugin.tests.scenario.v2.test_traffic_ops.TrafficOperationsScenarioTest](https://bugs.launchpad.net/tripleo/+bug/1973223)
* Latest run passed twice under observation

* https://bugs.launchpad.net/tripleo/+bug/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors
* Under observation

Master Component line
* https://bugs.launchpad.net/tripleo/+bug/1973038 Master Node provisioning failing because nodes failing to boot and entering Emergency shell
* We are waiting for latest diskimage-builder to promote till current-tripleo to fix component lines.

* https://bugs.launchpad.net/tripleo/+bug/1972163 Master cs9 standalone full tempest api failing for volume tempest tests and with error - 'Volume failed to detach from server'

Wallaby

https://bugs.launchpad.net/tripleo/+bug/1964940 Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.

---
## ***STOP (all tracked bugs duplicated above stop scrolling) STOP***

## Wednesday 18 May

## Promotion status

#### Master integration line - last_promotion=2022-05-16
* New bug:
* [Master fs035 and fs064 failing on overcloud deploy on task "Render keystone-db-sync container definitions" with error - can only concatenate list (not "AnsibleUnicode") to list. Unexpected templating type error occurred on (image: "{{ tripleo_keystone_image }}"](https://bugs.launchpad.net/tripleo/+bug/1973863 )
* Cedric proposed a patch.
* https://review.opendev.org/c/openstack/tripleo-ansible/+/842303
* rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537

* ovb jobs failed in rerun [here](https://review.rdoproject.org/r/c/testproject/+/39357)
* Need to investigate other failures

#### Master Component line - last_promotion=2022-05-16
Component which have failing jobs:-
* ~~Baremetal~~
* ~~Compute~~
* ~~Network~~
* security
* ~~tempest~~
* tripleo

Security and tripleo failing because of above known bug.

security failure looks wrong that mean we don't have proper isolication because we include delorean-current.

#### Wallaby c9 integration line - last_promotion=2022-05-17
* random tempest failures - rerunning jobs [here](https://review.rdoproject.org/r/c/testproject/+/28446)

### wallaby c9 component line
* tripleo - ovb failed - rerunning [here](https://review.rdoproject.org/r/c/testproject/+/39420)

#### Wallaby c8 integration line - last_promotion=2022-05-16
* fs001/fs035 failed - need to rerun

### wallaby c8 component line
* tripleo - ovb failed - rerunning [here](https://review.rdoproject.org/r/c/testproject/+/39420)

#### Victoria integration line - last_promotion=2022-05-16
* Line running right now - will recheck in few hours

### victoria c8 component line - All green

#### Train integration line - last_promotion=2022-05-16 - All green no new hash to promote

#### Train component line
* tripleo - ovb and sc04 - reruning [here](https://review.rdoproject.org/r/c/testproject/+/26273)

#### 16.2 integration line - 2022-05-16 - All green no new hash

#### 17/8 integration line - last_promotion=2022-05-17
* full-tempest-api/fs001 failed, rerunning [here](https://code.engineering.redhat.com/gerrit/c/testproject/+/201382)

#### 17/9 integration line - last_promotion=2022-05-17
* Line just triggered, will recheck in few hours

---

---
# Tuesday 17 May

* Random tempest issue is back
* Running wallaby/master on psi to see if we see same tempest failures.
* https://code.engineering.redhat.com/gerrit/c/testproject/+/209874

* Master integration line:
* fs001/020/35/39/~~64~~/~~ipa~~/~~container-updates~~ failed, rerunning [here](https://review.rdoproject.org/r/c/testproject/+/39357)
* afternoon hash - periodic-tripleo-ci-build-containers-centos-9-quay-master failed - rerunning.
* Master component line - rerunning [here](https://review.rdoproject.org/r/c/testproject/+/28537)
* ~~baremetal~~ - ovb fs001 - passed in testproject rerun
* security - ovb fs039
* ~~validation~~ - container multinode - passed in testproject rerun

* C9 wallaby integration line - 2022-05-17 - All green
* fs001 rbac failed - not in promotion criteria - do we need to report?

* c9 wallaby component line - rerunning [here](https://review.rdoproject.org/r/c/testproject/+/28537)
* tripleo

* C8 wallaby integration line - last_promotion=2022-05-16 - All green - No new hash when I checked this morning.

* c8 wallaby component line - All green

* Victoria integration line - last_promotion=2022-05-16 - All green - No new hash when I checked this morning.

* Victoria component line - ALl greeen

* Train integration line- last_promotion=2022-05-16 - All green - No new hash when I checked this morning.
* Train component line - All green

* OSP16.2 integration line - last_promotion=2022-05-16 - All green
* OSP16.2 Component line - All green

* OSP17/9 Integration line - In rerun
* full-tempest/fs020/bm failed
* testproject
* https://code.engineering.redhat.com/gerrit/c/testproject/+/190672
* https://code.engineering.redhat.com/gerrit/c/testproject/+/211643

* OSP17/9 Component line - All green
* ~~network~~ - full-tempest-api - cleared in rerun
* ~~tripleo~~ - ipa/container-multinode - cleared in rerun

* OSP17/8 Integration line - last_promotion=2022-05-17 - All green
* ~~full-tempest-scenario~~ - cleared in rerun
* ~~fs001~~ - cleared in rerun
* OSP17/8 Component line - All green
* tripleo
* ~~ipa~~ - cleared in rerun
* ~~container-multinode~~ - cleared in rerun

---
# Monday May 16th

* Check/Gate
* rechecked - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841284

### Status
#### Master integration line - last promotion - 2022-05-16 - Green
* only 1 failing job - sc02 and blocking master promotion
* [Master Scenario002 is failing on Barbican related tempest test - tempest.lib.exceptions.UnexpectedResponseCode: Unexpected response code received , Details: 503](https://bugs.launchpad.net/tripleo/+bug/1973568)
* Added in skiplist:
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/841834
* reruning in testproject: https://review.rdoproject.org/r/c/testproject/+/39357
* Fix: https://review.rdoproject.org/r/c/openstack/barbican-distgit/+/42876
* Promoting fix via component line: https://review.rdoproject.org/zuul/status#28537

#### Master component line - All green

#### Wallaby c9 integration line - 2022-05-15
* Ovb and full-tempest-api failing - need to dbeug

#### Wallaby c9 component line - only tripleo lagging
* tripleo

#### Wallaby c8 integration line - 2022-05-11
* fs01 and fs035 failing
* all fs passed independently today, promotion in progress.

#### Wallaby c8 component line - 2022-05-12
* ~~[last tripleo promotion on thursday](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby)~~
passed after rerun
* [last network promotion on friday](https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby)
[lets try to reproduce this re-popping error](https://review.rdoproject.org/r/c/testproject/+/42902)
* [Reruns](https://review.rdoproject.org/r/c/testproject/+/42890)

#### Victoria integration line - last_promotion=2022-05-16 - Green
* Promoted today
#### Victoria component line - baremetal
* baremetal - ovb - rlandy force promoted.

#### train integration line , last_promotion=2022-05-16 - All green
#### train component lines - All green

#### 16.2 integration line, last_promotion=2022-05-13
* Bm job failed - rerunning [here](https://code.engineering.redhat.com/gerrit/c/testproject/+/201382)

#### 16.2 component lines - All Green

#### 17/rhel-9 integration line, last_promotion=2022-05-15 (Green)
* Scheduled run is running, lets see in few hours
#### 17/rhel-9 component lines - All green
* ~~common~~ - passed in rerun

#### 17/rhel-8 integration line - last_promotion=2022-05-16 (green)

#### 17/rhel-8 component lines - All green
* ~~tempest~~ - passed in testproject
* ~~tripleo~~ - passed in testproject

---
## Friday 13 May
## Status:-

## Check
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841284/ (Under observation)
* Validation rpm build [failed](https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_7bc/841284/1/gate/tripleo-ci-centos-8-content-provider-train/7bc468f/logs/undercloud/home/zuul/DLRN/data/repos/component/validation/36/48/3648dcc24a06e493daa1715c0dd94615ec8da3e1_dev/rpmbuild.log) - invesitgate why we are build validation rpm in tripleo job
* https://review.opendev.org/c/openstack/tripleo-validations/+/841378/
* https://0fc95984b662eca8c7af-61a732188f1643f755755f84f6310584.ssl.cf2.rackcdn.com/841378/1/gate/tripleo-ci-centos-8-content-provider/e4b1117/logs/delorean_logs/component/validation/cc/ce/ccce4761654f0bc8477a5c14f1adba8bcf85ac67_dev/index.html

Pinged jpdovin about that - We are rechecking https://review.opendev.org/c/openstack/tripleo-validations/+/841378 - to issue if issue is consistent.

Yatin found the issue, jiri is fixing it.

~~~
<ykarel> https://github.com/openstack/cliff/commit/584352dcd008d58c433136539b22a6ae9d6c45cc is in 2.18.0+
<ykarel> but in train we have 2.16.0
<ykarel> jpodivin, the issue is reproducable easily with tox
<ykarel> too
<ykarel> with cliff 2.16.0
<ykarel> also iirc it was said stable/1.6 branch was for train release, but still i see it's testing with master u-c
ykarel> shouldn't it be using stable/train
<jpodivin> it should be
<ykarel> ok please fix then it should catch atleast error like this
<ykarel> https://github.com/openstack/requirements/blob/stable/train/upper-constraints.txt#L285
~~~

### Promotion
##### Master - 2022-05-06 - Out by 1 week - 2 known bugs (ovb instability and sc-10-kvm)
* Full tempest scenario failing - debug
* Rerunning failed jobs [here](https://review.rdoproject.org/r/c/testproject/+/39357)
##### Master component line
* Components which are lagging - No known bugs - Rerunning jobs [here](https://review.rdoproject.org/r/c/testproject/+/28537):-
* ~~Network~~ promoted via testproject
* ~~Security~~ promoted via testproject
* Validations
* ~~Tripleo~~ promoted via testproject

##### Wallaby c9 - 2022-05-10 - Out by 3 days now - 2 known bugs(ovb instalbility and neutron tempest bug)
* ~~Full-tempest-scenario~~ - passed via testproject
* Rerunning the failing jobs [here](https://review.rdoproject.org/r/c/testproject/+/39420)
* Need to discuss with Yatin and rodolfo - where are we with neutron bug.
* Spoke with yatin this morning - Running a testproject with node hold and cleanup script disabled for debugging
* https://review.rdoproject.org/r/c/testproject/+/36140/49/.zuul.yaml#9
##### Wallaby c9 components
* Components which are lagging
* Network(08th may) - ovb failing - Rerunning [here](https://review.rdoproject.org/r/c/testproject/+/42693/2/.zuul.yaml)

##### Wallaby c8 - 2022-05-10 - 2 known bugs(ovb instalbility and neutron tempest bug)
* ovb fs001/035 failing
* Need to discuss with Yatin and rodolfo - where are we with neutron bug.
* Spoke with yatin this morning - Running a testproject with node hold and cleanup script disabled for debugging
* https://review.rdoproject.org/r/c/testproject/+/36140/49/.zuul.yaml#9

##### Wallaby c8 components
* Components which are lagging behind:-
* ~~baremetal - Red - 26th April - ovb failing - need to debug~~
* https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-wallaby/4b8b904/job-output.txt
* Report and investigate
* Rerun: https://review.rdoproject.org/r/c/testproject/+/42813
If it gets to tempest (instead of failing on redfish), then we will promote this line.
* Promoted component

##### Victoria C8 - 2022-05-09
* Running victoria failed jobs [here](https://review.rdoproject.org/r/c/testproject/+/40101)

That hash is out only fs001 and fs035 - can choose to promote it (7 days out)
- project:
check:
jobs:
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria:
vars:
force_periodic: true
featureset_override:
dlrn_hash_tag: 527fdcb7ab243130b6f360da7f56c4f9
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-victoria:
vars:
force_periodic: true
featureset_override:
dlrn_hash_tag: 527fdcb7ab243130b6f360da7f56c4f9

##### Victoria C8 components
* Components which are lagging
* Baremetal - 30th April - rerunning [here](https://review.rdoproject.org/r/c/testproject/+/42693/2/.zuul.yaml)
* Tripleo - promoted on 11th May - Can wait for tonight run
* Have baremetal testproject in rerun
* promoting with fs001 removed

##### Train C8 - 2022-05-09
* Line running right now - let's see in few hours
+ https://review.rdoproject.org/r/c/testproject/+/36254/98
+ fs35 on train timed out, full-tempest-api-train succeeded and sc1 failed
* Few jobs failed - need to investigate and report if consistent
* Train passed all jobs except fs035 (timed out - skipped to promote)
* Train promoted

##### Train C8 components - All green

##### 16.2 - Promoted on 2022-05-13 - All Green
##### 16.2 Components - All Green

##### 17/9 - 2022-05-12 - Green

##### 17/9 components - Rerunning failed jobs [here](https://code.engineering.redhat.com/gerrit/c/testproject/+/201382)
* Components which are lagging
* tripleo
* ~~validations~~ promoted via testproject

##### 17/8 - Need to check yet

* containers build jobs passed but jobs are failing on ceph:
fpantano rlandy: we merged a downstream patch yesterday
https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/834352 is needed for quickstart
fpantano https://code.engineering.redhat.com/gerrit/c/python-tripleoclient/+/406914
fpantano rlandy: fultonj the revert ^

### New/Transient/No bug yet

### Bugs