# Ruck Rover 2023-01-20 - 2023-01-26
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/juRN31FtQkG8UYwR_bZHSg

##### ruck & rover: Bhagyashris and Tengu

### NOTE: Ongoing bugs

:::info

Check/Gate blockers

Intergation line blockers
* https://bugs.launchpad.net/tripleo/+bug/2003865 mixed OS wallaby cs8/9 fails tempest network_basic_ops, security_groups
* https://bugs.launchpad.net/tripleo/+bug/2003882 unable to configure haproxy container with puppet (probably linked to an update in puppetlabs-haproxy)
* https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops

* https://bugs.launchpad.net/tripleo/+bug/2000061 : CS-9 OVB fs001 tripleo master is unstable
* ~~https://bugs.launchpad.net/tripleo/+bug/2003658 playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~

Downstream blockers:
* https://bugzilla.redhat.com/show_bug.cgi?id=2154179 periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy
:::

## 26 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-26 (check bellow)
- Zed: 2023-01-25
- Wallaby (cs9): 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues
##### Wallaby
- tripleo-ci-centos-8-9-multinode-mixed-os started failing today
- https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
- https://bugs.launchpad.net/tripleo/+bug/2003865
- patch is being worked on (check LP)
##### Master
- Network component on Master is apparently broken for ~1 or 2 days:
- https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
- https://bugs.launchpad.net/tripleo/+bug/2003884
- Seems to be due to pyroute2
- TripleO component is hosed as well
- https://bugs.launchpad.net/tripleo/+bug/2003882
- Workaround: https://review.rdoproject.org/r/c/rdoinfo/+/46709
- doesn't seem to work - investigating if the pin patch was correct or if anything more is needed. CIX card up-to-date.
- Proper fix: https://review.opendev.org/c/openstack/puppet-tripleo/+/871807
- testproject: https://review.rdoproject.org/r/c/testproject/+/46701
- checks if pin is applied (for now)

### Downstream lines

## 25 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-24
- Zed: 2023-01-23
- Wallaby (cs9): 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues
- tripleo-ci-centos-8-9-multinode-mixed-os started failing today
- https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
- https://bugs.launchpad.net/tripleo/+bug/2003865
- patch is being worked on (check LP)
- Network component on Master is apparently broken for ~1 or 2 days:
- https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
- http://dashboard-ci.tripleo.org/d/WWV51gO4k/component-pipeline?orgId=1&viewPanel=478
- TripleO component is hosed as well
- https://bugs.launchpad.net/tripleo/+bug/2003882
- probably due to puppetlabs-haproxy patch (check LP comments)

### Downstream promotion
- RHOS-18 RH9 - 2023-01-24
- RHOS-17.1 RH9 - 2023-01-15
- running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs
- jenkins job failure:
- pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
- https://bugzilla.redhat.com/show_bug.cgi?id=2162653
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
- RHOS-17.1 RH8 - 2023-01-24
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
- RHOS-17 RH9 - 2023-01-23
- running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431169 [DNM] Test rhos17 on rhel9 failing job
- RHOS-16-2 RH8 - 2023-01-25

## 24 Jan
### Upstream lines
#### Promotions
- Master: 2023-01-23
- Zed: 2023-01-23
- Wallaby (cs9): ~~2023-01-19~~ 2023-01-24
- Wallaby (cs8): 2023-01-24
- Train: 2023-01-20 (no new content ) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5
#### Issues/blockers
- re-kicked missing promotion job for wallaby on cs9 (5 days old):
- https://review.rdoproject.org/r/c/testproject/+/46698
- Should promote shortly. **DONE**
- master promotion: 2 jobs are failing on tempest, apparently every time. Need investigations.
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master: issue with TestNetworkBasicOps, leading to an API timeout on actions
- periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master: cinder snapshot test, hitting some content format issue (?)
- Re-kicked failed jobs:
- https://review.rdoproject.org/r/c/testproject/+/46701

### Downstream promotion
- RHOS-18 RH9 - 2023-01-21
- RHOS-17.1 RH9 - 2023-01-15
- jenkins job failure:
- pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
- https://bugzilla.redhat.com/show_bug.cgi?id=2162653
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
- RHOS-17.1 RH8 - 2023-01-22
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
- RHOS-17 RH9 - 2023-01-23
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
- RHOS-16-2 RH8 - 2023-01-21
- No new content

## 23 Jan
### Upstream lines
#### Promotions
- Master: 20/01/2023
- Zed: 20/01/2023
- Wallaby (cs9): 19/01/2023
- Wallaby (cs8): 21/01/2023
- Train: 20/01/2022
#### Issues/blockers
- ~~all upstream hosed, https://bugs.launchpad.net/tripleo/+bug/2003658 playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~
- Re-kicked master pipeline (openstack-periodic-integration-main) in order to ensure the above issue is really solved
- we're good! Promotion should happen shortly.

### Downstream promotion
- RHOS-18 RH9 - 2023-01-21
- RHOS-17.1 RH9 - 2023-01-15
- jenkins job failure:
- pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
- https://bugzilla.redhat.com/show_bug.cgi?id=2162653
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
- RHOS-17.1 RH8 - 2023-01-22
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
- RHOS-17 RH9 - 2023-01-23
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
- RHOS-16-2 RH8 - 2023-01-21
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN

## 20 Jan

### Upstream promotion
- Master CS9 - 2023-01-19
- Zed - 2023-01-18
- Wallaby c9 - 2023-01-18
- Wallaby c8 - 2023-01-19
- Train C8 - 2023-01-18
- 10:54 < Tengu|rover> marios, bhagyashris|ruck https://review.rdoproject.org/r/c/testproject/+/46686 train re-run

### Downstream promotion
- RHOS-17.1 RH9 - 2023-01-15
- periodic-tripleo-rhel-9-upload-job-trigger-rhos-17.1 have passed
- blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2161501
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
- RHOS-17.1 RH8 - 2023-01-14
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
- RHOS-17 RH9 - 2023-01-19
- RHOS-16-2 RH8 - 2023-01-19
- https://code.engineering.redhat.com/gerrit/c/testproject/+/431140