# Ruck Rover 2022-08-19 to 2022-08-25

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/9b8XBCJYSDKf6QDDD9c2OQ

##### ruck: soniya, rover: doug

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

[Downstream promoter](http://10.0.110.143/promoter_logs/)

---

### Findings
* https://softwarefactory-project.io/weeder/tenant/rdoproject.org/info
### Ideas
* Promoter with conditional criteria, e.g.:
* fs035 OR fs035-internal
* an criteria item could be a colon or semicolon separeted list:
* -...featureset035-master;..featureset035-internal
* we could promote jobs by running the internal versions with the same hash
### Ongoing Bugs

#### Blocking:

Wallaby Check/Gate

* https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
* Xek proposed: https://review.opendev.org/853478
* Merged, rechicked security component
* It might need to be backported to fix wallaby too
* ipa job running in check itself - need to work with security team to continue debug/fixing.
* Wallaby component was Force promoted; We need to watch integration lines.

* https://bugs.launchpad.net/tripleo/+bug/1987092 - centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status
* job is in retry state, before closing we need to have the job stable

https://bugs.launchpad.net/tripleo/+bug/1987632 -
cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"

### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/1987323 - FIPS + Manila + Ceph Quincy
* Not a blocker, but we will create a CIX for it

* https://bugs.launchpad.net/tripleo/+bug/1987092 - centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'

* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0

* https://bugs.launchpad.net/tripleo/+bug/1987616 - cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
* Still needs investigation

* https://bugs.launchpad.net/tripleo/+bug/1987641 -
cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
* We need a fix or this revert: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542

---
## 2022-08-25

### General notes
* **Please revert if promoted**:
* train ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44655~~
* master ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44654~~
* Since tuesday we are facing lots of different issues on ovb jobs, which are most probably related to vexxhost infra:
* networking issues between nodes
* rdo zuul stuck without starting jobs (seems solved now)
* dns issues (seems solved)
* mirror issues (sometimes happens)
* on fs039 and fs064, which sets up a supplemental node for freeipa, we saw lots of failures on install packages. A simple loop mitigate this problem. Not sure if you folks can thing on a better solution. It is ok to promote with this, since it is just a workaround on our setup, but we might need a permanent fix.
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/854180
* security on wallaby cs9 was force promoted, fs039 isn't passing
* https://trunk.rdoproject.org/centos9-wallaby/component/security/promoted-components/commit.yaml
* You should try ovb jobs on internal if they are failing due to unknown reasons. We still don't have fs039 and fs064 on internal, but I (doug) am working on that.

### Promotions
#### Upstream
* Master - promoted on 19th - should promote today with skip in criteria
* hash 6d8a0601abbb10f1edfd2a5b8c5e6fe5 from 25th, missing some ovb jobs, but also scenario001 which is a real bug, so this hash can't move forward
* since we still need a fix for scenario001, and tomorrow is friday already, it is very likey that we have a promotion until next week
* We will promote **89fbcbce79c4a970207d7f5952937f1c** from 23th, which is failing for fs039 only atm.
* Wallaby C9 - promoted on 22nd
* chasing **2779b7691127687f9beeb36a4e686d3c**
* Wallaby C8 - promoted on 22nd
* missing fs001 - chasing ea64cc35fa13ba6fa94e0fbfb45fc4f8
* Train - promoted on 22nd - promoted today 25th
* tp:- https://review.rdoproject.org/r/c/testproject/+/40083
* on skip fs035 to promote - pls revert when done
#### Components
* Master: cinder/clients/cloudops/common/compute/glance/network/security/tempest/tripleo
* Wallaby c9: clients/network/~~security~~/tripleo
* Wallaby c8: all good
* Train: fs01 tripleo
* chasing here:- https://review.rdoproject.org/r/c/testproject/+/37029
#### Downstream
* OSP17 - RHEL9 - promoted on 25th
* OSP16.2 - RHEL8 - promoted on 25th
* OSP17-1 - RHEL9 - promoted on 25th

#### Components
* RHEL9 17 -
* https://code.engineering.redhat.com/gerrit/c/testproject/+/425376
* RHEL9 17.1
* RHEL8 16.2 -
* https://code.engineering.redhat.com/gerrit/c/testproject/+/425376

## 2022-08-24
### Promotions
#### Upstream
* Master - promoted on 19th
* missing fs039 only, but RDO Zuul is not starting jobs (even after zuul restart)

* Wallaby C9 - promoted on 22nd
* Wallaby C8 - promoted on 22nd
* Train - promoted on 22nd
* tp:- https://review.rdoproject.org/r/c/testproject/+/40083
#### Components
* Master: baremetal/~~common~~/~~network~~
* Wallaby c9: clients, network, validation
* on rerun https://review.rdoproject.org/r/c/testproject/+/37029
* Wallaby c8: tripleo
* Train: network
* chasing here:- https://review.rdoproject.org/r/c/testproject/+/37029
#### Downstream
* OSP17 - RHEL9 - promoted on 21st
* missing fs035 internal
* https://code.engineering.redhat.com/gerrit/c/testproject/+/310670
* OSP16.2 - RHEL8 - promoted on 22nd
* no new hash
* OSP17-1 - RHEL9 - promoted on 22nd
* missing fs001 https://code.engineering.redhat.com/gerrit/c/testproject/+/279836

#### Components
* RHEL9 17 - network
* https://code.engineering.redhat.com/gerrit/c/testproject/+/425376
* RHEL9 17.1
* RHEL8 16.2 - tripleo
* https://code.engineering.redhat.com/gerrit/c/testproject/+/425376

## 2022-08-23
### Promotions
#### Upstream

* Master - promoted on 19th
* missing fs039 only, but RDO Zuul is not starting jobs (even after zuul restart)
* hash: 89fbcbce79c4a970207d7f5952937f1c
* Wallaby C9 - promoted on 22nd
* Wallaby C8 - promoted on 22nd
* Train - promoted on 22nd
* tp:- https://review.rdoproject.org/r/c/testproject/+/40083
#### Components
* Master: baremetal/~~common~~/~~network~~
* Wallaby c9: all good
* Wallaby c8: tripleo
* Train: compute
* chasing here:- https://review.rdoproject.org/r/c/testproject/+/42595
#### Downstream
* OSP17 - RHEL9 - promoted on 21st
* OSP16.2 - RHEL8 - promoted on 22nd
* OSP17-1 - RHEL9 - promoted on 22nd

#### Components
* RHEL9 17 - all good
* RHEL9 17.1
* RHEL8 16.2 - network/cloudops
* chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/407601

## 2022-08-22

### Promotions
#### Upstream
##### Master
* fs01/fs35/fs39/fs64 - failing
* https://review.rdoproject.org/r/c/testproject/+/43560/
* (doug) - chasing 89fbcbce79c4a970207d7f5952937f1c
##### Wallaby C9 - Promoted on 22th
* full-tempest-api/fs64/full-tempest-scenario/fs35/fs39
* https://review.rdoproject.org/r/c/testproject/+/43560/

##### Wallaby C8 - Promoted on 22th
* fs01
* https://review.rdoproject.org/r/c/testproject/+/43560/

##### Train - Should promote, failed on promoted
* fs20/fs35/fs39
* https://review.rdoproject.org/r/c/testproject/+/43560/

#### Components
* Master: all good
* Wallaby c9: security/validation/compute
* Wallaby c8: tripleo/security
* Train: all good
#### Downstream
* OSP17 - RHEL9 - promoted on 21/8[yesterday]

* OSP16.2 - RHEL8 - promoted today
- all green
- except periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 disk full
* OSP17-1 - RHEL9 - promoted 21/8[yesterday], promoted on 22th
* waiting for pipeline to kick

#### Components
* RHEL9 17 - security due to https://bugs.launchpad.net/tripleo/+bug/1986755
* RHEL8 17 - fs01-interna-tripleo
* https://code.engineering.redhat.com/gerrit/c/testproject/+/407599
* RHEL8 16.2 - all good

## 2022-08-19
### Promotions

#### Upstream
##### Master - Promoted on 19th
* ~~chasing promotions here: https://review.rdoproject.org/r/c/testproject/+/35235~~
##### Wallaby C9 - Missing
* ~~sc04 - Block with ceph issue~~
* fs64/fs35 - overcloud deployment failed..rekikec here;- https://review.rdoproject.org/r/c/testproject/+/41063
##### Wallaby C8 - Promoted on 19th
* chasing fs001 here (with ovb affinity): https://review.rdoproject.org/r/c/testproject/+/36356
* node affinity not working for some jobs (needs debug)
* Try IBM cloud if it doesn't succeed on vexx
##### Train
* Chasing promotions here: https://review.rdoproject.org/r/c/testproject/+/35235
#### Components
* Master cs9
- validation/security/tripleo/network
- chasing here:- https://review.rdoproject.org/r/c/testproject/+/42692
* Wallaby cs9/cs8
* cs9
- multinode-ipa and fs39 security
- https://review.rdoproject.org/r/c/testproject/+/42595
* cs8
- fs01 component validation
- fs01 tripleo - https://review.rdoproject.org/r/c/testproject/+/42595
* Train cs8
- compute/tripleo - https://review.rdoproject.org/r/c/testproject/+/42693
#### Downstream
##### OSP17 - RHEL9 - Should promoted on 19th
~~* chasing https://code.engineering.redhat.com/gerrit/c/testproject/+/279836~~
##### OSP17 - RHEL8 - No new hash
##### OSP17-1 - RHEL9 Promoted on 19th
##### OSP16-2 - RHEL8 Promoted on 19th
#### Component
* missing security on OSP17 (due to https://bugs.launchpad.net/tripleo/+bug/1986755)
* missing compute on OSP16-2 (no run yet)