# Ruck Rover 2022-12-02 - 2022-12-08
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/Y9s_hAQVQhq5LUKk1lNKPw

##### ruck & rover: marios & ronelle

:::info

### Active Bugs
* ~~https://bugs.launchpad.net/tripleo/+bug/1997939~~ (c8 integration)
* ~~https://bugs.launchpad.net/tripleo/+bug/1998098~~ (validations)
* ~~https://bugs.launchpad.net/tripleo/+bug/1998227~~ (tempest c9 fs001)
* ~~https://bugs.launchpad.net/tripleo/+bug/1998348~~ (selinux) (**TODO** close eod)
* https://bugs.launchpad.net/tripleo/+bug/1998211 (fs64 zed overcloud deploy)
* https://bugzilla.redhat.com/show_bug.cgi?id=2148354 (rhos17/17.1 fs35)
* https://bugzilla.redhat.com/show_bug.cgi?id=2149963 (jenkins jobs 17.1 on rhel9)
* https://bugs.launchpad.net/tripleo/+bug/1998762 * container quay-push jobs are failing push quay.rdoproject.org with 502 Bad Gateway
* https://bugs.launchpad.net/tripleo/+bug/1998770 * periodic multinode-oooq-container-updates-train fails minor update "TypeError: isinstance expected 2 arguments,
* https://bugs.launchpad.net/tripleo/+bug/1998954 scenario001 and scenario004 standalone test are failing - "Non-zero exit code 1 from systemctl start ceph - selinux enforcing"
:::
## Dec 08

### New
* https://bugs.launchpad.net/tripleo/+bug/1999131 * centos-9 content provider fails pulling ubi9 "unrecognized signature format, starting with binary 0x3c"
* https://bugs.launchpad.net/tripleo/+bug/1999140 wallaby & train 8 jobs NODE_FAILURE

### promotions/chasing:
* master nice buildset https://review.rdoproject.org/zuul/buildset/e2948fcc810949708a5a167d46b3c6ae * undercloud-upgrade fail & digging (2x) https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-master/d14f40a/logs/undercloud/home/zuul/undercloud_upgrade.log.txt.gz * related to https://bugs.launchpad.net/tripleo/+bug/1998501 ? * test with https://review.rdoproject.org/r/c/testproject/+/46375 & https://review.opendev.org/c/openstack/tripleo-heat-templates/+/865929
* passed... sanity checkign without depends-on tripleo-heat-templates/+/865929 running there https://review.rdoproject.org/r/c/testproject/+/46375/2#message-6c703397d73dd4c94670c3401d93abbadbe33696
* zed https://review.rdoproject.org/zuul/buildset/5d31938bd23247e69fc4bf870cd6845e * missing fs64/fs2 chasing https://review.rdoproject.org/r/c/testproject/+/46376
* train/8 * NODE_FAILURE killed the line https://review.rdoproject.org/zuul/buildset/9015360eb2694e949136219e2846cbe9 (also yesterday) * manually rekicked https://review.rdoproject.org/zuul/status#openstack-periodic-integration-stable4
* same result :/ https://review.rdoproject.org/zuul/build/91e76e98f2894d439848e95281ac4cb5
* manual kick again (running )
* https://review.rdoproject.org/r/c/rdo-jobs/+/46377 Temp remove ibm while under investigation (component lines) https://bugs.launchpad.net/tripleo/+bug/1999140
* train components (until network) in rerun https://review.rdoproject.org/r/c/testproject/+/25325

---
## Dec 07

* All tox jobs are failing - https://review.opendev.org/c/zuul/zuul-jobs/+/866928 see #openstack-discuss "Tox 4.0.0 breaking changes"
* (**Dec 08 also that for tqe** https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/866964)
* chasing RDO components: https://review.rdoproject.org/r/c/testproject/+/25325
* chasing downstream 17 and 17.1

### Promotions/chasing:
* Master: 2022-12-06
* Zed: 2022-12-06 (no new hash)
* Wallaby c9: 2022-12-07
* Wallaby c8: 2022-12-07
* Train c8: 2022-12-06 (no new hash)

- 17/9 - 2022-12-06
- 17.1/9 - 2022-12-06
- 17.1/8 - No jobs in criteria
- 16.2/8 - 2022-12-06 (no new hash)

---

## Dec 06
### New bugs
* https://bugs.launchpad.net/tripleo/+bug/1998954 scenario001 and scenario004 standalone test are failing - "Non-zero exit code 1 from systemctl start ceph - selinux enforcing"
* testing : https://review.opendev.org/c/openstack/tripleo-ci/+/866806 Ensure CentOS nodes are selinux Permissive
* may be the wrong approach
* pinged Cedric - he's the selinux expert

### Promotions/chasing:
* master * buildset @ https://review.rdoproject.org/zuul/buildset/7d2ec431e69e415698322c5cab3fdc64 * chasing with ~~https://review.rdoproject.org/r/c/testproject/+/46361~~ -> will promote
* 17.1/9 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/c2182f0c84d845e88c2ee7980acbe7cf (jenkins passed) * chasing fs1 https://code.engineering.redhat.com/gerrit/c/testproject/+/437138
* chasing zed: https://review.rdoproject.org/r/c/testproject/+/36254
* chasing 17: https://code.engineering.redhat.com/gerrit/c/testproject/+/431128

---
## Dec 05
### New bugs
* https://bugs.launchpad.net/tripleo/+bug/1998762 * container quay-push jobs are failing push quay.rdoproject.org with 502 Bad Gateway
* https://bugs.launchpad.net/tripleo/+bug/1998770 * periodic multinode-oooq-container-updates-train fails minor update "TypeError: isinstance expected 2 arguments, got 1"
* **Promoter not resolving new creds - ~~pls ping Ananya and dpawlik in the morning~~** (ack)

### Upstream component pipelines
* c9:
* tp: https://review.rdoproject.org/r/c/testproject/+/36254 (master network)
* both master and wallaby network components promoted
### Downstream promotion pipelines
- 17/9 - 01st Dec (new hash 6 hours ago - should promote tomorrow)
- 17.1/9 - 29th Nov (Blocked on jenkins jobs)
- promoting up donwstream change: https://code.engineering.redhat.com/gerrit/c/testproject/+/430999 - then rekicking 17.1 ine
https://bugzilla.redhat.com/show_bug.cgi?id=2149963
- 16.2/8 - 12-05
- tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/430582 for promotion

---
## Dec 02
### Upstream promotion pipelines
* Master: 2022-12-03
* https://bugs.launchpad.net/tripleo/+bug/1998227
* tp: https://review.rdoproject.org/r/c/testproject/+/46302
* Zed: 2022-12-02
* fs64:
* https://bugs.launchpad.net/tripleo/+bug/1998211
* tp: https://review.rdoproject.org/r/c/testproject/+/36254
* internal pass: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/6b3f45d3d1ca49f49ba9b18d8d9a7131
* Wallaby c9: 30 nov
* tp: https://review.rdoproject.org/r/c/testproject/+/36254
* Wallaby c8: 1 Dec
* https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train&skip=0 - possible real issue here:
* Train c8: 1 Dec
*
### Upstream component pipelines
* c9:
* tp: https://review.rdoproject.org/r/c/testproject/+/25325

* c8:
* tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/430999
### Downstream promotion pipelines
- 17/9 - 01st Dec (promoted today)
- 17.1/9 - 29th Nov (Blocked on jenkins jobs)
- chasing promotions https://code.engineering.redhat.com/gerrit/c/testproject/+/209874/55
- reported a bug for jenkins job failure https://bugzilla.redhat.com/show_bug.cgi?id=2149963
- 17.1/8 - 30th Nov (Promoted yesterday)
- 16.2/8 - 12-03 (Promoted yesterday) No new hash

### Downstream component pipelines
- 17/9 -
- 17.1/9 -
- 17.1/8 - No jobs in criteria
- 16.2/8 -