---
# System prepended metadata

title: Ruck Rover 2023-01-20 - 2023-01-26
tags: [ruck_rover]

---

# Ruck Rover 2023-01-20 - 2023-01-26
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)


Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/juRN31FtQkG8UYwR_bZHSg

##### ruck & rover: Bhagyashris and Tengu

### NOTE: Ongoing bugs

:::info

Check/Gate blockers

Intergation line blockers
* https://bugs.launchpad.net/tripleo/+bug/2003865 mixed OS wallaby cs8/9 fails tempest network_basic_ops, security_groups
* https://bugs.launchpad.net/tripleo/+bug/2003882 unable to configure haproxy container with puppet (probably linked to an update in puppetlabs-haproxy)
* https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops 

* https://bugs.launchpad.net/tripleo/+bug/2000061 :  CS-9 OVB fs001 tripleo master is unstable 
* ~~https://bugs.launchpad.net/tripleo/+bug/2003658  playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~ 

Downstream blockers:
* https://bugzilla.redhat.com/show_bug.cgi?id=2154179 periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy 
:::

## 26 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-26 (check bellow)
- Zed: 2023-01-25
- Wallaby (cs9): 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues
##### Wallaby
- tripleo-ci-centos-8-9-multinode-mixed-os started failing today
    - https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
    - https://bugs.launchpad.net/tripleo/+bug/2003865
    - patch is being worked on (check LP)
##### Master
- Network component on Master is apparently broken for ~1 or 2 days:
    - https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
    - https://bugs.launchpad.net/tripleo/+bug/2003884
    - Seems to be due to pyroute2
- TripleO component is hosed as well
    - https://bugs.launchpad.net/tripleo/+bug/2003882
    - Workaround: https://review.rdoproject.org/r/c/rdoinfo/+/46709
        - doesn't seem to work - investigating if the pin patch was correct or if anything more is needed. CIX card up-to-date.
    - Proper fix: https://review.opendev.org/c/openstack/puppet-tripleo/+/871807
    - testproject: https://review.rdoproject.org/r/c/testproject/+/46701
        - checks if pin is applied (for now)


### Downstream lines

## 25 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-24
- Zed: 2023-01-23
- Wallaby (cs9): 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues
- tripleo-ci-centos-8-9-multinode-mixed-os started failing today
    - https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
    - https://bugs.launchpad.net/tripleo/+bug/2003865
    - patch is being worked on (check LP)
- Network component on Master is apparently broken for ~1 or 2 days:
    - https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
    - http://dashboard-ci.tripleo.org/d/WWV51gO4k/component-pipeline?orgId=1&viewPanel=478
- TripleO component is hosed as well
    - https://bugs.launchpad.net/tripleo/+bug/2003882
    - probably due to puppetlabs-haproxy patch (check LP comments)



### Downstream promotion
  - RHOS-18 RH9 - 2023-01-24
  - RHOS-17.1 RH9 - 2023-01-15
      - running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs  
      - jenkins job failure:
        - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9  jobs are failing which are blocking  the promotions. could you plz check
        - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
          - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix 
  - RHOS-17.1 RH8 -  2023-01-24
      -  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
  - RHOS-17 RH9 - 2023-01-23
      - running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431169 [DNM] Test rhos17 on rhel9 failing job
  - RHOS-16-2 RH8 - 2023-01-25



## 24 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-23
- Zed: 2023-01-23
- Wallaby (cs9): ~~2023-01-19~~ 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content ) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues/blockers
- re-kicked missing promotion job for wallaby on cs9 (5 days old):
    - https://review.rdoproject.org/r/c/testproject/+/46698
    - Should promote shortly. **DONE**
- master promotion: 2 jobs are failing on tempest, apparently every time. Need investigations.
    - periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master: issue with TestNetworkBasicOps, leading to an API timeout on actions
    - periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master: cinder snapshot test, hitting some content format issue (?)
    - Re-kicked failed jobs:
        - https://review.rdoproject.org/r/c/testproject/+/46701




### Downstream promotion
  - RHOS-18 RH9 - 2023-01-21
  - RHOS-17.1 RH9 - 2023-01-15
      - jenkins job failure:
        - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9  jobs are failing which are blocking  the promotions. could you plz check
        - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
          - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix 
  - RHOS-17.1 RH8 -  2023-01-22
      -  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
  - RHOS-17 RH9 - 2023-01-23
      - run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
  - RHOS-16-2 RH8 - 2023-01-21
    - No new content

## 23 Jan
### Upstream lines
#### Promotions
- Master: 20/01/2023
- Zed: 20/01/2023
- Wallaby (cs9): 19/01/2023
- Wallaby (cs8): 21/01/2023
- Train: 20/01/2022
#### Issues/blockers
- ~~all upstream hosed, https://bugs.launchpad.net/tripleo/+bug/2003658 playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~ 
    - Re-kicked master pipeline (openstack-periodic-integration-main) in order to ensure the above issue is really solved
        - we're good! Promotion should happen shortly.



### Downstream promotion
  - RHOS-18 RH9 - 2023-01-21
  - RHOS-17.1 RH9 - 2023-01-15
      - jenkins job failure:
        - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9  jobs are failing which are blocking  the promotions. could you plz check
        - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
          - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix 
  - RHOS-17.1 RH8 -  2023-01-22
      -  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
  - RHOS-17 RH9 - 2023-01-23
      - run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
  - RHOS-16-2 RH8 - 2023-01-21
    - run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN 

## 20 Jan

### Upstream promotion
- Master CS9 - 2023-01-19
- Zed - 2023-01-18
- Wallaby c9 - 2023-01-18
- Wallaby c8 - 2023-01-19
- Train C8 - 2023-01-18
    - 10:54 < Tengu|rover> marios, bhagyashris|ruck https://review.rdoproject.org/r/c/testproject/+/46686  train re-run



### Downstream promotion
  - RHOS-17.1 RH9 - 2023-01-15
      - periodic-tripleo-rhel-9-upload-job-trigger-rhos-17.1 have passed
      - blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2161501
      -  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
  - RHOS-17.1 RH8 -  2023-01-14
      -  https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
  - RHOS-17 RH9 - 2023-01-19
  - RHOS-16-2 RH8 - 2023-01-19
    - https://code.engineering.redhat.com/gerrit/c/testproject/+/431140 


