# Ruck Rover 2022-11-25 - 2022-12-01
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Vn_zjslARMG5Crs4YwnSqw

##### ruck & rover: pooja & ananya

:::info

### Active Bugs
* ~~https://bugs.launchpad.net/tripleo/+bug/1997939~~ (c8 integration)
* ~~https://bugs.launchpad.net/tripleo/+bug/1998098~~ (validations)
* https://bugs.launchpad.net/tripleo/+bug/1998211 (fs64 zed overcloud deploy)
* https://bugs.launchpad.net/tripleo/+bug/1998227 (tempest c9 fs001)
* https://bugs.launchpad.net/tripleo/+bug/1998348 (selinux) (**TODO** close eod)
* https://bugzilla.redhat.com/show_bug.cgi?id=2148354 (rhos17/17.1 fs35)
* https://bugzilla.redhat.com/show_bug.cgi?id=2149963 (jenkins jobs 17.1 on rhel9)

:::
## Dec 01
### Gate

No particular job is red but I see some Retry limits (**TODO** monitor)

### New bugs
https://bugzilla.redhat.com/show_bug.cgi?id=2149963

### Upstream promotion pipelines
* Master: 2022-11-28
* https://bugs.launchpad.net/tripleo/+bug/1998227
* tp: https://review.rdoproject.org/r/c/testproject/+/46302
* Zed: 2022-11-25
* fs64:
* https://bugs.launchpad.net/tripleo/+bug/1998211
* tp: https://review.rdoproject.org/r/c/testproject/+/36254
* Wallaby c9: 30 nov
* tp: **TODO**
* Wallaby c8: 30 nov
* tp: **TODO**
* Train c8: 30 nov
* tp: **TODO**
### Upstream component pipelines
* c9:
* fs001s : https://bugs.launchpad.net/tripleo/+bug/1998227
* standalones: https://bugs.launchpad.net/tripleo/+bug/1998348 (this cleared on its own, will close by eod)
* c8:
* tp: **TODO**
### Downstream promotion pipelines
- 17/9 - ~~30th Nov (Promoted yesterday)~~ 01st Dec (promoted today)
- 17.1/9 - 29th Nov (Blocked on jenkins jobs)
- chasing promotions https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/55
- reported a bug for jenkins job failure https://bugzilla.redhat.com/show_bug.cgi?id=2149963
- 17.1/8 - 30th Nov (Promoted yesterday)
- 16.2/8 - 30th Nov (Promoted yesterday) No new hash

### Downstream component pipelines
- 17/9 - All green except network component
- Rerun https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/5
- 17.1/9 - All green except network component
- Rerun https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/5
- 17.1/8 - No jobs in criteria
- 16.2/8 - All green

## Nov 30
### Gate
No blockers
### New bugs
### Upstream promotion pipelines
* Master: 2022-11-28
* https://bugs.launchpad.net/tripleo/+bug/1998227
* tp: https://review.rdoproject.org/r/c/testproject/+/46302
* Zed: 2022-11-25
* fs64:
* https://bugs.launchpad.net/tripleo/+bug/1998211
* Wallaby c9: ~~2022-11-28~~ PROMOTED TODAY!!
* https://review.rdoproject.org/r/c/testproject/+/46256
* c8 ~~(wallaby 2022-11-23) (train 2022-11-24)~~ : PROMOTED TODAY!! https://review.rdoproject.org/r/c/testproject/+/46301

### Upstream component pipelines
* Master, Zed:
* https://bugs.launchpad.net/tripleo/+bug/1998348
* https://bugs.launchpad.net/tripleo/+bug/1998227
* Wallaby c9
* https://bugs.launchpad.net/tripleo/+bug/1998348

### Downstream promotion pipelines
- 17/9 - 30th Nov
- rerunning sc12 standalone https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/54
- 17.1/9 - 29th Nov
- 17.1/8 - 29th Nov
- 16.2/8 - 28th Nov
- rerunning tempest api https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/54

### Downstream component pipelines
- 17/9 -
- Security and network component
- Rerunning https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/3
- 17.1/9 -
- network component
- Rerunning https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/3
- 17.1/8 - No jobs in criteria
- 16.2/8 - All green

## Nov 29
### Gate
### New bugs
#### Upstream promotion pipelines
* Master:
* https://bugs.launchpad.net/tripleo/+bug/1998227

* Zed:
* https://review.rdoproject.org/r/c/testproject/+/46287
* fs64:
* https://bugs.launchpad.net/tripleo/+bug/1998211

* Wallaby c9:
* https://review.rdoproject.org/r/c/testproject/+/46256

#### Upstream Component pipelines

* https://review.rdoproject.org/r/c/testproject/+/46292
* https://review.rdoproject.org/r/c/testproject/+/46293
- Master
- validation
- https://review.rdoproject.org/r/c/testproject/+/46282
- https://review.rdoproject.org/r/c/testproject/+/33582
- Zed
- validation
- https://review.rdoproject.org/r/c/testproject/+/46283
- security
- fs39 failed with random failures
- network
- fs01 failed with random failures.
- Wallaby C9
- common
- standalone blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
- Wallaby C8
- tripleo
- standalone job blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
- tempest
- standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
- security
- standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
- network
- standalone job blocked https://bugs.launchpad.net/tripleo/+bug/1997939
- https://review.rdoproject.org/r/c/testproject/+/40897
- Train C8
- tripleo
- fs01 blocked on https://bugs.launchpad.net/tripleo/+bug/1997939
- common
- sc04 standalone blocked on https://bugs.launchpad.net/tripleo/+bug/1997939

#### Downstream promotion pipelines
- 17/9 - 28th Nov Line currently running
- 17.1/9 - 28th Nov fs35 and sc01 standalone failed
- Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/52
- 17.1/8 - 28th Nov All green
- 16.2/8 - 28th Nov All green
#### Downstream component promotion pipelines
- 17/9 - All green except sc04 common component job
- Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/2
- 17.1/9 - All green except tempest api network component job
- Rerun : https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/2
- 17.1/8 - No job in criteria
- 16.2/8 - All green

## Nov 28

### Gate:
### New bugs:
* https://bugs.launchpad.net/tripleo/+bug/1997939 (Affects wallaby and train c8)
* tp: https://review.rdoproject.org/r/c/testproject/+/46272
#### Upstream promotion pipelines
* Master: 28th
- https://review.rdoproject.org/r/c/testproject/+/46253
* Zed: 25th:
* https://review.rdoproject.org/r/c/testproject/+/46242
* featureset064:
* Could not create '/home/zuul/overcloud-hardened-uefi-full.raw': Permission denied
* Overcloud dceploy failed
* Wallaby c9: 27th
* https://review.rdoproject.org/r/c/testproject/+/46256
* Wallaby c8: RED
* https://bugs.launchpad.net/tripleo/+bug/1997939
* Train: RED
* https://bugs.launchpad.net/tripleo/+bug/1997939
#### Upstream Component pipelines
* Master
* tripleo (tempest tests failed)
* ~~https://review.rdoproject.org/r/c/testproject/+/46257~~
* validation
* https://bugs.launchpad.net/tripleo/+bug/1998098
* Zed
* periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-zed
* deploy-steps-tasks-step-0.yaml times out (**TODO** bug)
* https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-zed/0953c50/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz
* Wallaby c9:
* https://review.rdoproject.org/r/c/testproject/+/46269
* Walllaby c8:
* tempest, tripleo, network - standalone job failing with https://bugs.launchpad.net/tripleo/+bug/1997939
* Train c8:
* common -
* standalone - https://bugs.launchpad.net/tripleo/+bug/1997939
* tripleo
* fs001 - https://bugs.launchpad.net/tripleo/+bug/1997939
* sc04 standalone - https://bugs.launchpad.net/tripleo/+bug/1997939
#### Downstream promotion pipelines
- 17/9 - 28th Nov fs35 by skipping https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/865797
- 17.1/9 - 28th Nov fs35 by skipping https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/865797
- 17.1/8 - 26th Nov
- 16.2/8 - 28th Nov(Today) - ~~tempest-scenario and sc01 standalone failing with randaom tempest test.~~
- Rerun (got sucess) https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/50

#### Downstream component promotion pipelines
- 17/9 - All green except tripleo component jenkins job
- 17.1/9 - All green
- 17.1/8 - No job in criteria
- 16.2/8 - All green exceot tripleo component jenkins job

## Nov 25

### Gate:
Rechecked on https://review.opendev.org/c/openstack/heat/+/802139/ (failed once)

### New bugs:
* https://bugzilla.redhat.com/show_bug.cgi?id=2148354 periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1 failing consistently with set of keystone_tempest_plugin tests with error "Unexpected content type provided"
* https://bugs.launchpad.net/tripleo/+bug/1997939 tripleo-ci-centos-8-standalone in stable/wallaby consistently fails because of the failure during creating galera resource

#### Upstream promotion pipelines
- Master -
- https://review.rdoproject.org/r/c/testproject/+/46241
- Zed - promoted
- ~~https://review.rdoproject.org/r/c/testproject/+/46242~~
- C9 Wallaby -
- running

- C8 wallaby -
- https://review.rdoproject.org/r/c/testproject/+/46243 (rerunning the olds registry push job till Amol's patches merge)

- C8 Train -

#### Upstream Component pipelines
- Master
- clients
- fs001 ovb (random tempest tests failing)
- network
- fs001 ovb (random tempest tests failing)
- Reruns (https://review.rdoproject.org/r/c/testproject/+/33582)

- Zed
- Reruns (https://review.rdoproject.org/r/c/testproject/+/36719)
- C9 wallaby
- tempest - random tests failing
- C8 wallaby -
- Reruns (https://review.rdoproject.org/r/c/testproject/+/40897)
- C8 train -
- Reruns (https://review.rdoproject.org/r/c/testproject/+/36478)

#### Downstream promotion pipelines
- 17/9 - 22nd Nov (Line running)
- 17.1/9 - 22nd Nov - fs35 failing with tempest tests
- 17.1/8 - 22nd Nov (All green)
- 16.2/8 - 23rd Nov
- fs35 failed once [Rerun]https://code.engineering.redhat.com/gerrit/c/testproject/+/209874
(got a green run https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/ed489e014c584d8d833e9d070132ed6a)

#### Downstream component promotion pipelines
- 17/9 - All green
- 17.1/9 - network and tripleo [Rerun] https://code.engineering.redhat.com/gerrit/c/testproject/+/436232
(got a green run https://code.engineering.redhat.com/gerrit/c/testproject/+/436232/1#message-c943b28d5be1c620a8053a5ef8142341f5ea8c40)
- 17.1/8 - No job in criteria
- 16.2/8 - All green