# Ruck Rover 2022-11-04 - 2022-11-10
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ

##### ruck & rover: marios & rcastillo
----
## Nov 10
#### New/transient no bug yet:
* promoter bug/alternative criteria ?
* http://promoter.rdoproject.org/promoter_logs/centos9_master_2022-11-10T08:20.log
* adaac75f69ae93d6ae76ee320b90dc0a
* Successful Jobs periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master
* Missing Jobs periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-master
* promotion chasing featureset001-internal-master
* confirmed ... now fs20 passed https://review.rdoproject.org/r/c/testproject/+/45996/8#message-abc19c3de9b35382169145f991b685940851f1cd but promoter still missing
* **master** fs20 tempest fail & recheck https://review.rdoproject.org/r/c/testproject/+/45996/8#message-e7cc80d54eaa78c28ffceac04321d76dc811b1db
* **train** * currently running good pending ovb https://review.rdoproject.org/zuul/buildset/72d30026eba5495f911ab3c62a6f8f23 * also chasing good candidate https://review.rdoproject.org/r/c/testproject/+/46049
* **16.2/8** good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/4ecef8e512924a989cb0d03c158f20ea missing tempest- * posted https://code.engineering.redhat.com/gerrit/c/testproject/+/434848
* **17.1/9** * good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/ada4c16426c44d1bb6ec01a011eb255a only scen4 * posted https://code.engineering.redhat.com/gerrit/c/testproject/+/434850 * also missing jenkins jobs triggered https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/106/ & https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph/104/
* jenkins fail might be real will see with rerun ^^ example: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/105/undercloud-0/home/stack/undercloud_install.log.gz
* confirmed filed https://bugzilla.redhat.com/show_bug.cgi?id=2141701

#### New bugs today
* https://bugzilla.redhat.com/show_bug.cgi?id=2141701 OSP 17.1/rhel9 jenkins jobs fail before undercloud install - Please check the access rights for: '/var/log/validations'
* https://trello.com/c/DcqS7k0P

----
## Nov 09
#### New/transient no bug yet:
* ~~gate fails (not widespread) RETRY/RETRY_LIMIT no logs~~
* stable/wallaby https://zuul.opendev.org/t/openstack/buildset/84b50a35e0544d79ae784f7bdddaa4c2
* master (tripleo-ci) https://zuul.opendev.org/t/openstack/buildset/b9c37b3ec8384711976af50f862129c1
* Didn't see during NA time, must have been a blip
* ~~sf-infra down ssh: connect to host review.rdoproject.org port 29418: Connection timed out~~ resolved
* ~~14:07 < marios|ruck> well looks like upstream gerrit too then not just rdo arxcruz maybe root related~~

* promotion chasing:
* wallaby/8 rerun line @ ~~https://review.rdoproject.org/r/c/testproject/+/46037~~ * ~~https://review.rdoproject.org/r/c/testproject/+/46037/1#message-85b47e5ce7426dd2608cca6e9cd91731caac3d4d~~ promoting
* wallaby/9 chasing 2 candidates fs20 & fs64 @ ~~https://review.rdoproject.org/r/c/testproject/+/45996/7#message-a872a4991f1eb0a328bbeb7278367f2811996b25~~ promoted
* master missing fs20 https://review.rdoproject.org/r/c/testproject/+/45996/7#message-a872a4991f1eb0a328bbeb7278367f2811996b25

----
## Nov 08
#### New/transient no bug yet:
* rdo/vexx issue still seen today
* openstack-periodic-integration-stable4 https://review.rdoproject.org/zuul/buildset/286e15de335d4e3bafcb6f2e272bea20
* https://bugs.launchpad.net/tripleo/+bug/1995851/comments/3
* https://review.rdoproject.org/zuul/buildset/fe7f638c57d94605881e25f02f055936 * Pipeline openstack-periodic-integration-stable1
* Failed to download packages: dmidecode-1:3.3-7.el9.x86_64: Cannot download, all mirrors were already tried without success", "results":
* promotion chasing:
* train just missing fs20 posted ~~https://review.rdoproject.org/r/c/testproject/+/46026~~ & promoted ~~https://review.rdoproject.org/r/c/testproject/+/46026/1#message-ff68295f6c0edd3d2671bfea1030d023a088825d~~
* Wallaby cs9 out two jobs, chasing: https://review.rdoproject.org/r/c/testproject/+/45996
* A couple of downstream components are out just a single jenkins job. I think I've rekicked them, but not sure if did it right.

----

## Nov 07
#### New bugs today:
* **https://bugs.launchpad.net/tripleo/+bug/1995851 Various periodic promotion jobs failing - NODE_FAILURE or POST_FAILURE**

#### New/transient no bug yet:
* promotion chasing:
* ~~master c9 failing on ovb, chasing two hashes~~
* today's hash: https://review.rdoproject.org/r/c/testproject/+/45996
* alt hash: https://review.rdoproject.org/r/c/testproject/+/45999
* newest hash promoted
* wallaby components https://review.rdoproject.org/r/c/testproject/+/45998 -> promoted https://review.rdoproject.org/r/c/testproject/+/45998/2#message-e43cdbd6aab4f3e3f2fe317739bf95a8b60fc761
* 1 job missing for network
* Job passed, not sure why it didn't promote yet?
* line kicked today but was hit by post_failure issue we're seeing https://review.rdoproject.org/zuul/build/bc04b0adf44c481eadaee942e8c0a837
* rekicked https://review.rdoproject.org/zuul/buildset/26fe00fa74d14e5faf8a0d24355afe22
* train (5 days) manually kicked openstack-periodic-integration-stable4 today & failed (3x-> filed https://bugs.launchpad.net/tripleo/+bug/1995851 )
* also chasing earlier hash (4th) ~~https://review.rdoproject.org/r/c/testproject/+/45956/2#message-b11bde9b78e8d1c427c3ccacdeeaca26c2bba10e~~ & promoted
* 16.2/8 (7 days) good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/d3972829640f4239b48b46d3a3934719 * missing jenkins jobs http://10.0.110.143/promoter_logs/redhat8_osp16-2.log
* ping rhos-ops/rlandy & retriggered those https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/multijob-trigger-dlrn-aggregate-16.2-rhel-8.4/377/
* another good cadidate 6d889c2b38355991aeffb09dfa0cb940 (from 05th) chasing with ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/434461~~ & promoted
* 17.1/9 (4 days) nice buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/84c1473cf96e4dc499e83eb457cb60a2 * chasing fs20 ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/434465~~ & promoted

----
## Nov 04

#### https://bugs.launchpad.net/tripleo/+bug/1995641 [CS9][Wallaby] featureset001-component-wallaby-validation failing with: "run_validations() got an unexpected keyword argument 'log_path'"
* https://review.opendev.org/c/openstack/python-tripleoclient/+/863559 https://review.rdoproject.org/r/c/testproject/+/36356/77#message-a3959ed56abc27ec910e47d217a46cc7fedde5e6
* https://trello.com/c/wUQUj2Nq/2772-cixlp1995641tripleociproa-cs9wallaby-featureset001-component-wallaby-validation-failing-with-runvalidations-got-an-unexpected-ke

#### https://bugs.launchpad.net/tripleo/+bug/1995719 wallaby component tripleo FATAL | Run init bundle puppet on the host
* https://review.opendev.org/c/openstack/puppet-pacemaker/+/863480

#### New/transient no bug yet:
* trunk.rdoproject.org a bit slow
* some jobs failing on dns issues https://logserver.rdoproject.org/46/45946/2/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/c6e18b6/job-output.txt

* chasing promotions:
* master https://review.rdoproject.org/r/c/testproject/+/45946
* train https://review.rdoproject.org/r/c/testproject/+/45956
* component/master -> network & glance (no consistent issue) chasing https://review.rdoproject.org/r/c/testproject/+/45954
* components/wallaby: tripleo & validation chasing https://review.rdoproject.org/r/c/testproject/+/45955
* will fail until https://bugs.launchpad.net/tripleo/+bug/1995719