Ruck Rover 2022-07-08 to 2022-07-14

tags: ruck_rover

Previous RR notes: https://hackmd.io/pRC9TDaoQLWqRSQioTUl5g?edit

Cockpit

Downstream cockpit

OpenStack Program Meeting 2022

14th Jul, 2022

UPstream check and

• Gate blocker:
  • https://bugs.launchpad.net/tripleo/+bug/1981634
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/849732 - in the gate

CS9 Master

• promoted 12th july
• Rekicked: https://review.rdoproject.org/r/c/testproject/+/42692

CS9 Wallaby

• promoted yesterday
• Rekicked:- https://review.rdoproject.org/r/c/testproject/+/42676

Cs8 Wallaby

• promoted on 12th july
• Rekicked:- https://review.rdoproject.org/r/c/testproject/+/41228

CS8 Train

• promoted on 13th july
• rekicked:- https://review.rdoproject.org/r/c/testproject/+/40932

RHEL9 17

• promoted on 13th july

RHEL8 17

• promoted on 13th july

RHEL8 16.2

• promoted on 12th july
• Rekicked:- https://code.engineering.redhat.com/gerrit/c/testproject/+/397264

Component lines

• Master:
  • validation:- https://review.rdoproject.org/r/c/testproject/+/42693
• Wallaby c8:
  • https://review.rdoproject.org/r/c/testproject/+/42595

13th Jul, 2022

Upstream check and gate

• Gate blocker:
  • https://bugs.launchpad.net/tripleo/+bug/1981634
  • trying revert a recent ceph change:
    • https://review.opendev.org/c/openstack/tripleo-heat-templates/+/849755
  • Giulio is aware, fultonj and fmount are on pto
  • Pls follow up on this in the morning

CS9 Master

• promoted yesterday
• tp: https://review.rdoproject.org/r/c/testproject/+/43560

CS9 Wallaby

• promoted yesterday
• Rekicked failures:- https://review.rdoproject.org/r/c/testproject/+/42676
• fs01, fs20, fs35 + kvm internal missing

CS8 Wallaby

• promoted today

CS8 train:

• fs01 is still missing
• https://bugs.launchpad.net/tripleo/+bug/1981478
• Promoted
  • fs001 and fs035 are passing on internal, so we changed the criteria to consider internal jobs:
    • https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43987
    • https://code.engineering.redhat.com/gerrit/c/testproject/+/418639

RHEL9 17

• promoted yesterday

RHEL8 17

• promoted on 11/07
• https://code.engineering.redhat.com/gerrit/c/testproject/+/407599

RHEL8 16.2

• promoted yesterday

Component lines

• Master:- all seems good with component, security got promoted today
• Wallaby cs9:
  • network: https://review.rdoproject.org/r/c/testproject/+/42595
  • manila: is good
  • tripleo: fs39 may fail because of ovb issue - 'Failed to delete port with name or ID '171053d0-d2a2-40dc-aa89-c06773b16ce6': No Port found for 171053d0-d2a2-40dc-aa89-c06773b16ce6'
    • https://review.rdoproject.org/r/c/testproject/+/40083

12th Jul, 2022

** OVB jobs failing to create stacks in downstream - pls alert rhos-ops if this is still happening tomorrow **

Upstream check and gate

• tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0 is green now
• heat-functional failed once in gate

CS9 Master

• tp:- https://review.rdoproject.org/r/c/testproject/+/43560
• fs35, fs39, fs64, fs01 are failing
• https://bugs.launchpad.net/tripleo/+bug/1981424 - periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master fails consistently on tempest
• https://bugs.launchpad.net/tripleo/+bug/1981460 - cs9 fs35 master is failing on tempest tests with error 'tempest.exceptions.BuildErrorException: Server failed to build and is in ERROR status'

CS9 Wallaby

• Rekicked here:- https://review.rdoproject.org/r/c/testproject/+/42676 fs20, fs35, fs01, fs39 + kvm-internal are missing

CS8 Wallaby

• Green - promoted yesterday

CS8 train

• fs01 missing
  • New bug: https://bugs.launchpad.net/tripleo/+bug/1981478
• tp:- https://review.rdoproject.org/r/c/testproject/+/40932
• chasing new hash: 10fe06ba12e547934c687990503db51b

RHEL9 17

• promoted today RHEL8 17
• promoted yesterday RHEL8 16.2
• promoted yesterday, green

Component lines:

• master - tripleo, network, security
  • tripleo/network:- https://review.rdoproject.org/r/c/testproject/+/42692
  • security:- not failing anything from criteria, but didn't get promotion over it
• train - tripleo
  • tripleo:- https://review.rdoproject.org/r/c/testproject/+/41063

11th Jul, 2022

Vexxhost upgrade over weekend - promoter and cockpit are down

Upstream check and gate

• watching https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-standalone-on-multinode-ipa&skip=0 (issues was reported on Friday - looks not bad today)

CS9 master

• Rekicked here:- https://review.rdoproject.org/r/q/topic:chasing_master_promotion
• Missing 3 jobs + kvm-internal
  • https://review.rdoproject.org/r/c/testproject/+/37029/41/.zuul.yaml

CS9 wallaby - Promoted on 07/11

• Rekicked here:- https://review.rdoproject.org/r/q/topic:chasing-wallaby-failures fs35 failing with different issues

• scenario010-kvm failing on: https://bugs.launchpad.net/tripleo/+bug/1979546
• skipping scn010-kvm to promote
• promoted

CS8 wallaby - Promoted on 07/11

• CS8 wallaby is good, promotor is down
• promoted

CS8 train

• Rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40932
  • fs20, fs01, fs35 are failing here
  • needs investigation

Component Line:

• CS9 wallaby:- all good, except fs01 tripleo wallaby
  • Tested here:- https://review.rdoproject.org/r/c/testproject/+/40083
  • https://bugs.launchpad.net/tripleo/+bug/1981337
  • can we skip fs01 tripleo wallaby? if yes then dviroel can you put the patch for the same.
  • Passed on rerun, should promote.

RHEL9 17 promoted on 07///

• last promoted on 07/07
• yesterday's run: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/30789405db464f1abc38274687906dd5 failed fs035 and scenario010
• today's run: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/b8dd0123176a49a3aaf68cc0aafce7e6
• need to investigate - how long failing fs035 and scenario010
• periodic-tripleo-ci-rhel-9-scenario010-standalone-rhos-17 - last test still failing - in rerun

RHEL8 17 - Promoted on 07/11

• Jenkins tests are failing - need the Jenkins tests to complete promotion afazekas: http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/pipeline_integration-pcci-17_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/409/undercloud-0/home/stack/overcloud_install.log.gz
• DONE - promoted
• rerunning components

RHEL8 16.2 - Promoted on 07/11

• promoted today: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/05f9cabaeae3435294e66e7d84dbfdfe
• check if two open bugs are valid (soniya)

Known bugs:

• https://bugzilla.redhat.com/show_bug.cgi?id=2105408 RHOSP16.2 - Octavia tempest test LoadBalancerScenarioTest.test_load_balancer_ipv4_CRUD failing on load balancer creation: timeout waiting compute instance referencing: https://bugs.launchpad.net/tripleo/+bug/1979546

  • this is affecting scenario010 on centos-8 and centos-9

• https://bugzilla.redhat.com/show_bug.cgi?id=2104931 RHOSP16.2 Neutron QOS related tempest test failure and traceback observed in neutron server log: "ERROR neutron.plugins.ml2.managers TypeError: qos_del() got an unexpected keyword argument 'if_exists'"

• https://bugs.launchpad.net/tripleo/+bug/1980255

09th Jul, 2022

Upstream Check and gate

• tripleo-ci-centos-9-content-provider-wallaby fails because of missing ceph container - https://bugs.launchpad.net/tripleo/+bug/1981167

08th Jul, 2022

Check/Gate

• all standalone jobs are runnig the same tempest tests:
  • revert to fix the issue: https://review.opendev.org/c/openstack/tripleo-ci/+/849006

CS9 wallaby

• Rechecked the tp: https://review.rdoproject.org/r/c/testproject/+/35235
• fs001, fs020: missing jobs to promote, no lucky today, different failures on each rerun

CS8 train

• rerun fs035 - https://review.rdoproject.org/r/c/testproject/+/43930
  • fs035 does not pass, needs investigation

RH9 RHOS-17

• pipeline is still running

RH8 RHOS-17

• https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17/5d7beb3/logs/undercloud/home/zuul/tripleo-deploy/standalone-ansible-3rt_k_ja/cephadm/cephadm_command.log

The task includes an option with an undefined variable. The error was: 'tripleo_cephadm_alertmanager_container_image' is undefined

• need tripleo component promotion once dlrn is enabled back for rhel8.
• https://bugzilla.redhat.com/show_bug.cgi?id=2104372

RH8 RHOS-16.2

• Full tempest api bug https://bugzilla.redhat.com/show_bug.cgi?id=2104931

  • fix is merged now, need a network component promotion and then integration line
  • TP network component: https://code.engineering.redhat.com/gerrit/c/testproject/+/398758

• Tempest test failure on scenario 010 (test_load_balancer_ipv4_CRUD): https://bugzilla.redhat.com/show_bug.cgi?id=2105408