---
# System prepended metadata

title: Ruck Rover 2022-11-04 - 2022-11-10
tags: [ruck_rover]

---

# Ruck Rover 2022-11-04 - 2022-11-10
[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)
[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

[Review List](/FGMoCiRfSNa8puA1BpTQ-Q)

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/2hB-P772SqyqDs0KKZzZEQ

##### ruck & rover: marios & rcastillo
----
## Nov 10
#### New/transient no bug yet:
 * promoter bug/alternative criteria ?
     * http://promoter.rdoproject.org/promoter_logs/centos9_master_2022-11-10T08:20.log
         * adaac75f69ae93d6ae76ee320b90dc0a 
             * Successful Jobs periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master
             * Missing Jobs periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-master
* promotion chasing featureset001-internal-master
        * confirmed ... now fs20 passed https://review.rdoproject.org/r/c/testproject/+/45996/8#message-abc19c3de9b35382169145f991b685940851f1cd but promoter still missing 
    * **master** fs20 tempest fail & recheck https://review.rdoproject.org/r/c/testproject/+/45996/8#message-e7cc80d54eaa78c28ffceac04321d76dc811b1db
    * **train** * currently running good pending ovb https://review.rdoproject.org/zuul/buildset/72d30026eba5495f911ab3c62a6f8f23 * also chasing good candidate https://review.rdoproject.org/r/c/testproject/+/46049
    * **16.2/8** good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/4ecef8e512924a989cb0d03c158f20ea missing tempest- * posted https://code.engineering.redhat.com/gerrit/c/testproject/+/434848
    * **17.1/9** * good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/ada4c16426c44d1bb6ec01a011eb255a only scen4 * posted https://code.engineering.redhat.com/gerrit/c/testproject/+/434850 * also missing jenkins jobs triggered https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/106/ & https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv6-geneve-ceph/104/
        * jenkins fail might be real will see with rerun ^^ example: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/105/undercloud-0/home/stack/undercloud_install.log.gz
            * confirmed filed https://bugzilla.redhat.com/show_bug.cgi?id=2141701

#### New bugs today
* https://bugzilla.redhat.com/show_bug.cgi?id=2141701 OSP 17.1/rhel9 jenkins jobs fail before undercloud install - Please check the access rights for: '/var/log/validations' 
    * https://trello.com/c/DcqS7k0P

----
## Nov 09
#### New/transient no bug yet:
* ~~gate fails (not widespread) RETRY/RETRY_LIMIT no logs~~
    * stable/wallaby https://zuul.opendev.org/t/openstack/buildset/84b50a35e0544d79ae784f7bdddaa4c2
    * master (tripleo-ci)  https://zuul.opendev.org/t/openstack/buildset/b9c37b3ec8384711976af50f862129c1 
    * Didn't see during NA time, must have been a blip
* ~~sf-infra down ssh: connect to host review.rdoproject.org port 29418: Connection timed out~~ resolved
    * ~~14:07 < marios|ruck> well looks like upstream gerrit too then not just rdo arxcruz maybe root related~~


* promotion chasing:
    * wallaby/8 rerun line @ ~~https://review.rdoproject.org/r/c/testproject/+/46037~~ * ~~https://review.rdoproject.org/r/c/testproject/+/46037/1#message-85b47e5ce7426dd2608cca6e9cd91731caac3d4d~~ promoting
    * wallaby/9 chasing 2 candidates fs20 & fs64 @ ~~https://review.rdoproject.org/r/c/testproject/+/45996/7#message-a872a4991f1eb0a328bbeb7278367f2811996b25~~ promoted
    * master missing fs20 https://review.rdoproject.org/r/c/testproject/+/45996/7#message-a872a4991f1eb0a328bbeb7278367f2811996b25

----
## Nov 08
#### New/transient no bug yet:
* rdo/vexx issue still seen today 
    * openstack-periodic-integration-stable4 https://review.rdoproject.org/zuul/buildset/286e15de335d4e3bafcb6f2e272bea20
    * https://bugs.launchpad.net/tripleo/+bug/1995851/comments/3
* https://review.rdoproject.org/zuul/buildset/fe7f638c57d94605881e25f02f055936    * Pipeline openstack-periodic-integration-stable1
    * Failed to download packages: dmidecode-1:3.3-7.el9.x86_64: Cannot download, all mirrors were already tried without success", "results":
* promotion chasing:
    * train just missing fs20 posted ~~https://review.rdoproject.org/r/c/testproject/+/46026~~ & promoted ~~https://review.rdoproject.org/r/c/testproject/+/46026/1#message-ff68295f6c0edd3d2671bfea1030d023a088825d~~
    * Wallaby cs9 out two jobs, chasing: https://review.rdoproject.org/r/c/testproject/+/45996
    * A couple of downstream components are out just a single jenkins job. I think I've rekicked them, but not sure if did it right.

----

## Nov 07
#### New bugs today:
* **https://bugs.launchpad.net/tripleo/+bug/1995851 Various periodic promotion jobs failing - NODE_FAILURE or POST_FAILURE**

#### New/transient no bug yet:
* promotion chasing:
    * ~~master c9 failing on ovb, chasing two hashes~~
        * today's hash: https://review.rdoproject.org/r/c/testproject/+/45996
        * alt hash: https://review.rdoproject.org/r/c/testproject/+/45999
        * newest hash promoted
    * wallaby components https://review.rdoproject.org/r/c/testproject/+/45998 -> promoted https://review.rdoproject.org/r/c/testproject/+/45998/2#message-e43cdbd6aab4f3e3f2fe317739bf95a8b60fc761 
        * 1 job missing for network
            * Job passed, not sure why it didn't promote yet?
            * line kicked today but was hit by post_failure issue we're seeing https://review.rdoproject.org/zuul/build/bc04b0adf44c481eadaee942e8c0a837
            * rekicked https://review.rdoproject.org/zuul/buildset/26fe00fa74d14e5faf8a0d24355afe22
    * train (5 days) manually kicked openstack-periodic-integration-stable4 today & failed (3x-> filed https://bugs.launchpad.net/tripleo/+bug/1995851 )
        * also chasing earlier hash (4th) ~~https://review.rdoproject.org/r/c/testproject/+/45956/2#message-b11bde9b78e8d1c427c3ccacdeeaca26c2bba10e~~  & promoted 
    * 16.2/8 (7 days) good buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/d3972829640f4239b48b46d3a3934719 * missing jenkins jobs http://10.0.110.143/promoter_logs/redhat8_osp16-2.log 
        * ping rhos-ops/rlandy & retriggered those https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/multijob-trigger-dlrn-aggregate-16.2-rhel-8.4/377/ 
        * another good cadidate 6d889c2b38355991aeffb09dfa0cb940 (from 05th) chasing with ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/434461~~ & promoted 
    * 17.1/9 (4 days) nice buildset https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/84c1473cf96e4dc499e83eb457cb60a2 * chasing fs20 ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/434465~~ & promoted

----
## Nov 04

#### https://bugs.launchpad.net/tripleo/+bug/1995641  [CS9][Wallaby] featureset001-component-wallaby-validation failing with: "run_validations() got an unexpected keyword argument 'log_path'" 
* https://review.opendev.org/c/openstack/python-tripleoclient/+/863559 https://review.rdoproject.org/r/c/testproject/+/36356/77#message-a3959ed56abc27ec910e47d217a46cc7fedde5e6
* https://trello.com/c/wUQUj2Nq/2772-cixlp1995641tripleociproa-cs9wallaby-featureset001-component-wallaby-validation-failing-with-runvalidations-got-an-unexpected-ke

#### https://bugs.launchpad.net/tripleo/+bug/1995719 wallaby component tripleo FATAL | Run init bundle puppet on the host
* https://review.opendev.org/c/openstack/puppet-pacemaker/+/863480

#### New/transient no bug yet:
* trunk.rdoproject.org  a bit slow 
    * some jobs failing on dns issues https://logserver.rdoproject.org/46/45946/2/check/periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-master/c6e18b6/job-output.txt

* chasing promotions:
    * master https://review.rdoproject.org/r/c/testproject/+/45946 
    * train https://review.rdoproject.org/r/c/testproject/+/45956
    * component/master -> network & glance (no consistent issue) chasing https://review.rdoproject.org/r/c/testproject/+/45954
    * components/wallaby: tripleo & validation chasing https://review.rdoproject.org/r/c/testproject/+/45955
        * will fail until  https://bugs.launchpad.net/tripleo/+bug/1995719
