---
# System prepended metadata

title: Ruck Rover 2022-08-19 to 2022-08-25
tags: [ruck_rover]

---

# Ruck Rover 2022-08-19 to 2022-08-25

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/9b8XBCJYSDKf6QDDD9c2OQ

##### ruck: soniya, rover: doug

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

[Downstream promoter](http://10.0.110.143/promoter_logs/)

---

### Findings
 * https://softwarefactory-project.io/weeder/tenant/rdoproject.org/info
### Ideas
 * Promoter with conditional criteria, e.g.:
     * fs035 OR fs035-internal
         * an criteria item could be a colon or semicolon separeted list:
             *   -...featureset035-master;..featureset035-internal
     * we could promote jobs by running the internal versions with the same hash
### Ongoing Bugs

#### Blocking:

Wallaby Check/Gate


* https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path. 
    * Xek proposed: https://review.opendev.org/853478
        * Merged, rechicked security component
        * It might need to be backported to fix wallaby too
    * ipa job running in check itself - need to work with security team to continue debug/fixing.
    * Wallaby component was Force promoted; We need to watch integration lines.

* https://bugs.launchpad.net/tripleo/+bug/1987092 - centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status
    * job is in retry state, before closing we need to have the job stable

https://bugs.launchpad.net/tripleo/+bug/1987632 - 
cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"

### New Bugs
* https://bugs.launchpad.net/tripleo/+bug/1987323 - FIPS + Manila + Ceph Quincy
    * Not a blocker, but we will create a CIX for it

* https://bugs.launchpad.net/tripleo/+bug/1987092 - centos-9-standalone-validation-master fails on 'Check Keystone public endpoint status'

    * https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-standalone-validation-master&skip=0 

* https://bugs.launchpad.net/tripleo/+bug/1987616 - cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
    * Still needs investigation

* https://bugs.launchpad.net/tripleo/+bug/1987641 - 
cs9 sc01 master failing with "Error: can only create exec sessions on running containers: container state improper", "stderr_lines": ["Error: can only create exec sessions on running containers: container state improper"]
   * We need a fix or this revert: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/854542
    
---
## 2022-08-25

### General notes
* **Please revert if promoted**:
    * train ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44655~~
    * master ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44654~~
* Since tuesday we are facing lots of different issues on ovb jobs, which are most probably related to vexxhost infra:
    * networking issues between nodes
    * rdo zuul stuck without starting jobs (seems solved now)
    * dns issues (seems solved)
    * mirror issues (sometimes happens)
* on fs039 and fs064, which sets up a supplemental node for freeipa, we saw lots of failures on install packages. A simple loop mitigate this problem. Not sure if you folks can thing on a better solution. It is ok to promote with this, since it is just a workaround on our setup, but we might need a permanent fix.
    * https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/854180
* security on wallaby cs9 was force promoted, fs039 isn't passing
    * https://trunk.rdoproject.org/centos9-wallaby/component/security/promoted-components/commit.yaml
* You should try ovb jobs on internal if they are failing due to unknown reasons. We still don't have fs039 and fs064 on internal, but I (doug) am working on that.


### Promotions
#### Upstream
* Master - promoted on 19th - should promote today with skip in criteria
    * hash 6d8a0601abbb10f1edfd2a5b8c5e6fe5 from 25th, missing some ovb jobs, but also scenario001 which is a real bug, so this hash can't move forward
        * since we still need a fix for scenario001, and tomorrow is friday already, it is very likey that we have a promotion until next week
        * We will promote **89fbcbce79c4a970207d7f5952937f1c** from 23th, which is failing for fs039 only atm.
* Wallaby C9 - promoted on 22nd
    * chasing **2779b7691127687f9beeb36a4e686d3c**
* Wallaby C8 - promoted on 22nd
    * missing fs001 - chasing  ea64cc35fa13ba6fa94e0fbfb45fc4f8
* Train - promoted on 22nd - promoted today 25th
  * tp:- https://review.rdoproject.org/r/c/testproject/+/40083
  * on skip fs035 to promote - pls revert when done
#### Components
* Master: cinder/clients/cloudops/common/compute/glance/network/security/tempest/tripleo
* Wallaby c9: clients/network/~~security~~/tripleo
* Wallaby c8: all good
* Train: fs01 tripleo
    * chasing here:- https://review.rdoproject.org/r/c/testproject/+/37029
#### Downstream
* OSP17 - RHEL9 - promoted on 25th
* OSP16.2 - RHEL8 - promoted on 25th
* OSP17-1 - RHEL9 - promoted on 25th

#### Components
* RHEL9 17 - 
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/425376
* RHEL9 17.1
* RHEL8 16.2 - 
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/425376


## 2022-08-24
### Promotions
#### Upstream
* Master - promoted on 19th
    * missing fs039 only, but RDO Zuul is not starting jobs (even after zuul restart)
   
* Wallaby C9 - promoted on 22nd
* Wallaby C8 - promoted on 22nd
* Train - promoted on 22nd
  * tp:- https://review.rdoproject.org/r/c/testproject/+/40083
#### Components
* Master: baremetal/~~common~~/~~network~~
* Wallaby c9: clients, network, validation
    * on rerun https://review.rdoproject.org/r/c/testproject/+/37029
* Wallaby c8: tripleo
* Train: network
    * chasing here:- https://review.rdoproject.org/r/c/testproject/+/37029
#### Downstream
* OSP17 - RHEL9 - promoted on 21st
    * missing fs035 internal
        *  https://code.engineering.redhat.com/gerrit/c/testproject/+/310670
* OSP16.2 - RHEL8 - promoted on 22nd
    * no new hash
* OSP17-1 - RHEL9 - promoted on 22nd
    * missing fs001 https://code.engineering.redhat.com/gerrit/c/testproject/+/279836

#### Components
* RHEL9 17 - network
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/425376
* RHEL9 17.1
* RHEL8 16.2 - tripleo
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/425376

## 2022-08-23
### Promotions
#### Upstream

* Master - promoted on 19th
    * missing fs039 only, but RDO Zuul is not starting jobs (even after zuul restart)
    * hash: 89fbcbce79c4a970207d7f5952937f1c
* Wallaby C9 - promoted on 22nd
* Wallaby C8 - promoted on 22nd
* Train - promoted on 22nd
  * tp:- https://review.rdoproject.org/r/c/testproject/+/40083
#### Components
* Master: baremetal/~~common~~/~~network~~
* Wallaby c9: all good
* Wallaby c8: tripleo
* Train: compute
    * chasing here:- https://review.rdoproject.org/r/c/testproject/+/42595
#### Downstream
* OSP17 - RHEL9 - promoted on 21st
* OSP16.2 - RHEL8 - promoted on 22nd
* OSP17-1 - RHEL9 - promoted on 22nd

#### Components
* RHEL9 17 - all good
* RHEL9 17.1
* RHEL8 16.2 - network/cloudops
    * chasing here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/407601

## 2022-08-22

### Promotions
#### Upstream
##### Master 
* fs01/fs35/fs39/fs64 - failing
* https://review.rdoproject.org/r/c/testproject/+/43560/
* (doug) - chasing 89fbcbce79c4a970207d7f5952937f1c
##### Wallaby C9 - Promoted on 22th
* full-tempest-api/fs64/full-tempest-scenario/fs35/fs39
* https://review.rdoproject.org/r/c/testproject/+/43560/

##### Wallaby C8 - Promoted on 22th
* fs01
* https://review.rdoproject.org/r/c/testproject/+/43560/

##### Train - Should promote, failed on promoted 
* fs20/fs35/fs39
* https://review.rdoproject.org/r/c/testproject/+/43560/

#### Components
* Master: all good
* Wallaby c9: security/validation/compute
* Wallaby c8: tripleo/security
* Train: all good
#### Downstream
* OSP17 - RHEL9 - promoted on 21/8[yesterday]

* OSP16.2 - RHEL8 - promoted today
    - all green 
    - except periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2 disk full
* OSP17-1 - RHEL9 - promoted 21/8[yesterday], promoted on 22th
* waiting for pipeline to kick

#### Components
* RHEL9 17 - security due to https://bugs.launchpad.net/tripleo/+bug/1986755
* RHEL8 17 - fs01-interna-tripleo 
    * https://code.engineering.redhat.com/gerrit/c/testproject/+/407599
* RHEL8 16.2 - all good


## 2022-08-19
### Promotions

#### Upstream
##### Master - Promoted on 19th
* ~~chasing promotions here: https://review.rdoproject.org/r/c/testproject/+/35235~~
##### Wallaby C9 - Missing
* ~~sc04 - Block with ceph issue~~
* fs64/fs35 - overcloud deployment failed..rekikec here;- https://review.rdoproject.org/r/c/testproject/+/41063
##### Wallaby C8 - Promoted on 19th
* chasing fs001 here (with ovb affinity): https://review.rdoproject.org/r/c/testproject/+/36356
  * node affinity not working for some jobs (needs debug)
*  Try IBM cloud if it doesn't succeed on vexx
##### Train 
* Chasing promotions here: https://review.rdoproject.org/r/c/testproject/+/35235
#### Components
* Master cs9 
    - validation/security/tripleo/network
         - chasing here:- https://review.rdoproject.org/r/c/testproject/+/42692
* Wallaby cs9/cs8
     * cs9 
         - multinode-ipa and fs39 security
         - https://review.rdoproject.org/r/c/testproject/+/42595
    * cs8
        - fs01 component validation
        - fs01 tripleo - https://review.rdoproject.org/r/c/testproject/+/42595
* Train cs8
    - compute/tripleo - https://review.rdoproject.org/r/c/testproject/+/42693
#### Downstream
##### OSP17 - RHEL9 - Should promoted on 19th
~~* chasing https://code.engineering.redhat.com/gerrit/c/testproject/+/279836~~
##### OSP17 - RHEL8 - No new hash
##### OSP17-1 - RHEL9 Promoted on 19th
##### OSP16-2 - RHEL8 Promoted on 19th
#### Component
* missing security on OSP17 (due to https://bugs.launchpad.net/tripleo/+bug/1986755)
* missing compute on OSP16-2 (no run yet)