# Ruck Rover 2022-07-08 to 2022-07-14

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g?edit

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

---

## 14th Jul, 2022

UPstream check and
* Gate blocker:
* https://bugs.launchpad.net/tripleo/+bug/1981634
* https://review.opendev.org/c/openstack/tripleo-ansible/+/849732 - in the gate

CS9 Master
* promoted 12th july
* Rekicked: https://review.rdoproject.org/r/c/testproject/+/42692

CS9 Wallaby
* promoted yesterday
* Rekicked:- https://review.rdoproject.org/r/c/testproject/+/42676

Cs8 Wallaby
* promoted on 12th july
* Rekicked:- https://review.rdoproject.org/r/c/testproject/+/41228

CS8 Train
* promoted on 13th july
* rekicked:- https://review.rdoproject.org/r/c/testproject/+/40932

RHEL9 17
* promoted on 13th july
*
RHEL8 17
* promoted on 13th july

RHEL8 16.2
* promoted on 12th july
* Rekicked:- https://code.engineering.redhat.com/gerrit/c/testproject/+/397264

Component lines
* Master:
* validation:- https://review.rdoproject.org/r/c/testproject/+/42693
* Wallaby c8:
* https://review.rdoproject.org/r/c/testproject/+/42595
## 13th Jul, 2022

Upstream check and gate
* Gate blocker:
* https://bugs.launchpad.net/tripleo/+bug/1981634
* trying revert a recent ceph change:
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/849755
* Giulio is aware, fultonj and fmount are on pto
* Pls follow up on this in the morning

CS9 Master
* promoted yesterday
* tp: https://review.rdoproject.org/r/c/testproject/+/43560

CS9 Wallaby
* promoted yesterday
* Rekicked failures:- https://review.rdoproject.org/r/c/testproject/+/42676
* fs01, fs20, fs35 + kvm internal missing

CS8 Wallaby
* promoted today

CS8 train:
* fs01 is still missing
* https://bugs.launchpad.net/tripleo/+bug/1981478
* **Promoted**
* fs001 and fs035 are passing on internal, so we changed the criteria to consider internal jobs:
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43987
* https://code.engineering.redhat.com/gerrit/c/testproject/+/418639

RHEL9 17
* promoted yesterday

RHEL8 17
* promoted on 11/07
* https://code.engineering.redhat.com/gerrit/c/testproject/+/407599

RHEL8 16.2
* promoted yesterday

Component lines
* Master:- all seems good with component, security got promoted today
* Wallaby cs9:
* network: https://review.rdoproject.org/r/c/testproject/+/42595
* manila: is good
* tripleo: fs39 may fail because of ovb issue - 'Failed to delete port with name or ID '171053d0-d2a2-40dc-aa89-c06773b16ce6': No Port found for 171053d0-d2a2-40dc-aa89-c06773b16ce6'
- https://review.rdoproject.org/r/c/testproject/+/40083
## 12th Jul, 2022

** OVB jobs failing to create stacks in downstream - pls alert rhos-ops if this is still happening tomorrow **

Upstream check and gate
* tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0 is green now
* heat-functional failed once in gate

CS9 Master
* tp:- https://review.rdoproject.org/r/c/testproject/+/43560
* fs35, fs39, fs64, fs01 are failing
* ~~https://bugs.launchpad.net/tripleo/+bug/1981424 -
periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master fails consistently on tempest~~
* https://bugs.launchpad.net/tripleo/+bug/1981460 - cs9 fs35 master is failing on tempest tests with error 'tempest.exceptions.BuildErrorException: Server failed to build and is in ERROR status'

CS9 Wallaby
* Rekicked here:- https://review.rdoproject.org/r/c/testproject/+/42676
fs20, fs35, fs01, fs39 + kvm-internal are missing

CS8 Wallaby
* Green - promoted yesterday

CS8 train
* fs01 missing
* New bug: https://bugs.launchpad.net/tripleo/+bug/1981478
* tp:- https://review.rdoproject.org/r/c/testproject/+/40932
* chasing new hash: 10fe06ba12e547934c687990503db51b

RHEL9 17
* promoted today
RHEL8 17
* promoted yesterday
RHEL8 16.2
* promoted yesterday, green

Component lines:
* master - tripleo, network, security
- tripleo/network:- https://review.rdoproject.org/r/c/testproject/+/42692
- security:- not failing anything from criteria, but didn't get promotion over it
* train - tripleo
- tripleo:- https://review.rdoproject.org/r/c/testproject/+/41063

## 11th Jul, 2022

~~**Vexxhost upgrade over weekend - promoter and cockpit are down**~~

Upstream check and gate
* watching https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0 (issues was reported on Friday - looks not bad today)

CS9 master
* Rekicked here:- https://review.rdoproject.org/r/q/topic:chasing_master_promotion
* Missing 3 jobs + kvm-internal
* https://review.rdoproject.org/r/c/testproject/+/37029/41/.zuul.yaml

CS9 wallaby - Promoted on 07/11
* Rekicked here:- https://review.rdoproject.org/r/q/topic:chasing-wallaby-failures
fs35 failing with different issues
- scenario010-kvm failing on: https://bugs.launchpad.net/tripleo/+bug/1979546
- skipping scn010-kvm to promote
- **promoted**

CS8 wallaby - Promoted on 07/11
* CS8 wallaby is good, promotor is down
* **promoted**

CS8 train
* Rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40932
- ~~fs20~~, fs01, fs35 are failing here
- **needs investigation**

Component Line:
* ~~CS9 wallaby:- all good, except fs01 tripleo wallaby~~
- Tested here:- https://review.rdoproject.org/r/c/testproject/+/40083
- https://bugs.launchpad.net/tripleo/+bug/1981337
- can we skip fs01 tripleo wallaby? if yes then dviroel can you put the patch for the same.
- Passed on rerun, should promote.

RHEL9 17 promoted on 07///
* ~~last promoted on 07/07~~
* yesterday's run: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/30789405db464f1abc38274687906dd5 failed fs035 and scenario010
* today's run: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/b8dd0123176a49a3aaf68cc0aafce7e6
* ~~need to investigate - how long failing fs035 and scenario010~~
* periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17 - last test still failing - in rerun

RHEL8 17 - Promoted on 07/11
* Jenkins tests are failing - need the Jenkins tests to complete promotion
afazekas: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_integration-pcci-17_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/409/undercloud-0/home/stack/overcloud_install.log.gz
* **DONE - promoted**
* rerunning components

RHEL8 16.2 - Promoted on 07/11
* promoted today: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/05f9cabaeae3435294e66e7d84dbfdfe
* check if two open bugs are valid (soniya)

Known bugs:

* https://bugzilla.redhat.com/show_bug.cgi?id=2105408 RHOSP16.2 - Octavia tempest test LoadBalancerScenarioTest.test_load_balancer_ipv4_CRUD failing on load balancer creation: timeout waiting compute instance
referencing: https://bugs.launchpad.net/tripleo/+bug/1979546
* this is affecting scenario010 on centos-8 and centos-9

* https://bugzilla.redhat.com/show_bug.cgi?id=2104931 RHOSP16.2 Neutron QOS related tempest test failure and traceback observed in neutron server log: "ERROR neutron.plugins.ml2.managers TypeError: qos_del() got an unexpected keyword argument 'if_exists'"

* https://bugs.launchpad.net/tripleo/+bug/1980255

## 09th Jul, 2022

Upstream Check and gate
- tripleo-ci-centos-9-content-provider-wallaby fails because of missing ceph container - https://bugs.launchpad.net/tripleo/+bug/1981167

## 08th Jul, 2022

Check/Gate
- all standalone jobs are runnig the same tempest tests:
- revert to fix the issue: https://review.opendev.org/c/openstack/tripleo-ci/+/849006

CS9 wallaby
- Rechecked the tp: https://review.rdoproject.org/r/c/testproject/+/35235
- fs001, fs020: missing jobs to promote, no lucky today, different failures on each rerun

CS8 train
- rerun fs035 - https://review.rdoproject.org/r/c/testproject/+/43930
- fs035 does not pass, needs investigation

RH9 RHOS-17
- pipeline is still running

RH8 RHOS-17
- https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17/5d7beb3/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-3rt_k_ja/cephadm/cephadm_command.log

The task includes an option with an undefined variable. The error was: 'tripleo_cephadm_alertmanager_container_image' is undefined

- need tripleo component promotion once dlrn is enabled back for rhel8.
- https://bugzilla.redhat.com/show_bug.cgi?id=2104372

RH8 RHOS-16.2
- Full tempest api bug https://bugzilla.redhat.com/show_bug.cgi?id=2104931
- fix is merged now, need a network component promotion and then integration line
- TP network component: https://code.engineering.redhat.com/gerrit/c/testproject/+/398758

- Tempest test failure on scenario 010 (test_load_balancer_ipv4_CRUD): https://bugzilla.redhat.com/show_bug.cgi?id=2105408