Ruck Rover 2022-09-13 to 2022-09-15

tags: ruck_rover

Next RR notes: https://hackmd.io/a92u74hDQhe5afE_LUtbRw

Previous RR notes: https://hackmd.io/s4TgnCY-QQGKv2ONxTjOZA

ruck: Bhagyashris, rover: Jakob

RDO Cockpit / RHOS Cockpit

RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

• opendev.org:openstack
• rdoproject.org:rdoproject.org
• redhat.com:tripleo-ci-internal

2022-09-15

TLDR

• Upstream

  • TODO

• Downstream

  • TODO

Reruns and Investigations:

(crossed out := known bugs recorded above)

NOTE: Watch for running testproject jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

Integration Lines

• centos-9 master

  • last_promotion=2022-09-13

  • hash 8c71d10c1dacd8b2b5fc59dbdd753186

    • periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master:

      • Overcloud stack: FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master:

      • Overcloud stack: FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master:

      • N/A

• centos-9 wallaby

  • last_promotion=2022-09-13

  • hash 164135bf6866a2568241e186f56b903b

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby:

      • Overcloud stack: FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby:

      • N/A

    • periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-wallaby:

      • Tempest tests FAILED. Reason: unknown

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby:

      • Tempest tests FAILED. Reason: unknown

• centos-8 wallaby

  • last_promotion=2022-09-14
  • hash 5b62e82eebd71259ed086d27d7a2abae

• centos-8 train

  • last_promotion=2022-09-12

  • hash 54e3db5e5134970e993eeb7d124da84c

    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train:
      • Tempest tests FAILED. Reason: unknown

• rhel-9 osp17-1

  • last_promotion=2022-09-14
  • hash fb7fef6ff353625dadab13dfa1c2c80e

• rhel-9 osp17

  • last_promotion=2022-09-11
  • hash 0990788a06ae1c3505956399b88fd630

• rhel-8 osp16-2

  • last_promotion=2022-09-13

  • hash 546f5261237fd7e18a39ce31ae980932

    • periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2:
      • N/A

Component Lines

• centos-9 master

  • baremetal

    • last_promotion=2022-09-13

    • hash a99bf274e4baec8e585bc9979e492bb8d85d17b5

      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master:
        • Overcloud stack: FAILED. Reason: unknown

  • cinder

    • last_promotion=2022-09-14

    • hash 87205a242bbf1c0c39490a1d518d6918d08addca

      • periodic-tripleo-ci-centos-9-scenario004-standalone-cinder-master:
        • N/A

  • common

    • last_promotion=2022-09-14
    • hash ee5d8644c328e850c561c99b2098aeba2bf1b471

  • compute

    • last_promotion=2022-09-07

    • hash 299c898cf4af1d96a4c683305d3c262285eccdaf

      • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master:
        • Tempest tests FAILED. Reason: unknown

  • security

    • last_promotion=2022-09-13
    • hash 44cdbc68658d8fe3013d1dd253a268bec343c087

  • tempest

    • last_promotion=2022-09-05
    • hash b39bbcc0bc5be39cc4d9e3cf2addf1924c8df49a

  • tripleo

    • last_promotion=2022-09-14
    • hash 948a5f23d866e2d30a5d4c2e723a63d1aae637e6

• centos-9 wallaby

  • tempest

    • last_promotion=2022-09-07

    • hash d5fc8ad86bb895ebf59ff73bdcf36a9a6b7eff4f

      • periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-tempest-wallaby:
        • Tempest tests FAILED. Reason: unknown

  • tripleo

    • last_promotion=2022-09-12

    • hash 39be6cfe5cc0e18bbea85705f593b7608cadd0a6

      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby:
        • Tempest tests FAILED. Reason: unknown

• centos-8 wallaby

  • network

    • last_promotion=2022-09-09

    • hash b125402309516d25e40113f3963d1bdf0a25421b

      • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby:
        • MAC conflict between hosts. Overcloud stack: FAILED. Reason: infra

  • tripleo

    • last_promotion=2022-09-13

    • hash 39be6cfe5cc0e18bbea85705f593b7608cadd0a6

      • periodic-tripleo-ci-centos-8-standalone-tripleo-wallaby:

        • N/A

      • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby:

        • Overcloud stack: FAILED. Reason: unknown

• centos-8 train

  • tripleo

    • last_promotion=2022-09-10

    • hash f7fa1d52a7f5d7ae868bc6a57da5e7cae1c84748

      • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train:
        • Timeout. Reason: unknown

• rhel-9 osp17-1

• rhel-9 osp17

• rhel-8 osp16-2

Known Bugs

• https://bugs.launchpad.net/tripleo/+bug/1960355

  • Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

  • affecting OSP17.1 as well

• https://bugs.launchpad.net/tripleo/+bug/1987092

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  • Pacemaker performance causes intermittent galera issues in loaded CI env
  • jobs fail on Check Keystone public endpoint status or in other places
  • zuul/builds
  • affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• https://bugzilla.redhat.com/2105408

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

  • RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)

  • reproducible only on RHEL 8.2 and RHEL 8.4 … RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.

  • latest status is tracked in cix card
  • blocking integration lines of c9 master (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master) and c9 wallaby (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby)
  • workaround (specifying cpu_models in libvirt) is currently being tested

• https://bugs.launchpad.net/tripleo/+bug/1988514

  • Trello
  • On CIX board in lane Waiting on external dependencies

• https://bugs.launchpad.net/tripleo/+bug/1987323

  • FIPS + Manila + Ceph Quincy

  • … known issue on the ceph side, … already merged in master/quincy … 17.2.4 should solve this issue, so we'll check during the next promotion.

  • On CIX board in lane Waiting on external dependencies
  • Not a blocker

• https://bugs.launchpad.net/tripleo/+bug/1988810

  • Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788

• https://bugs.launchpad.net/tripleo/+bug/1989197

  • Tempest test neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON fails on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

• https://bugs.launchpad.net/tripleo/+bug/1989341

  • Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch

• https://bugs.launchpad.net/tripleo/+bug/1989247

  • overcloud deploy fails with sudo: unable to send audit message: Operation not permitted

  • Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master

• https://bugs.launchpad.net/tripleo/+bug/1989452

  • multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host

• https://bugs.launchpad.net/tripleo/+bug/1989606

  • container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use"

Intermittent Failures

• https://bugs.launchpad.net/tripleo/+bug/1987616

  • cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'

  • no CIX card

• https://bugs.launchpad.net/tripleo/+bug/1938987

  • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train

  • ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train

• TASK [freeipa-setup : Run the tripleo-ipa preparation script]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby

• Connection failure: The read operation timed out in TASK [Get current autoholds]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby

• Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • this has different root causes, watch out for log details!

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby
  • FAILURE in tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby

• FATAL | Wait for puppet host configuration to finish | undercloud | error…

  • TASK [tripleo.operator.tripleo_undercloud_install : undercloud install] in job-output.txt
  • puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages: in logs/undercloud/home/zuul/undercloud_install.log.txt.gz
  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds" during overcloud deploy

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

Fixed

Downstream:

• rhos16.2 on rhel8:

  • Promotion: 13-sept-2022
  • In todays run conatiner build job failed so re-running the job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400
    • container build job failing due to this change https://review.opendev.org/c/openstack/tripleo-ci/+/850736
    • job passed on revert change https://review.opendev.org/c/openstack/tripleo-ci/+/857776 job result: periodic-tripleo-build-containers-rhel-8-internal-build-push-upload-rhos-16.2_1 https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/39d84ef71b86408aa433b5edbd452acc : SUCCESS in 1h 08m 23s

• rhos17.1 on rhel8

  • Promotion: 14-sept-2022

• rhos17 on rhel9:

  • Promotion: 11-Sept-2022
    • Note: we don't have new content to promote.

• rhos17.1 on rhel9:

  • Promoted: 15-sept-2022
  • Waiting for jekins jobs result, integration line is GREEN

Downstream component:

2022-09-14

Notes

• doug: hi o/ - we merged a small change on sshd_config that affects centos-9 images (https://softwarefactory-project.io/r/c/config/+/26113) - note that this image isn't build on daily basis anymore, so our jobs won't get this update unless we update the base image and request rhos-infra team to unpause the build. Just in case you need a new image, keep an eye on related ssh issues with missing kex alghoritm curve25519.

Reminder:

Please update https://docs.engineering.redhat.com/display/PRODCHAIN/2022-09-12+Meeting+Notes for the program call. wallaby c8, c9, train, 16.2, 17 , 17.1 on rhel-9 and rhel-8

Reruns and Investigations:

(crossed out := known bugs recorded above)

NOTE: Watch for running testproject jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

• Promotions

• C9 Master

  • last_promotion: 2022-09-13

  • hash 573ed7f477dfe3ed338408f28dac7035

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master:

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master:

      • Overcloud deploy failed with ERROR: Can't run container glance_api_db_sync
      • Overcloud stack: FAILED. Reason: ERROR: Can't run container rabbitmq_bootstrap

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master:

      • undercloud install failed with ensure apache is installed...Failed to download packages: centos-logos-httpd-***el9.noarch: Cannot download, all mirrors were already tried without success
      • Overcloud stack: FAILED. Reason: ConnectionRefusedError: [Errno 111] Connection refused

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master:

      • intermittent failure TASK [freeipa-setup : Run the tripleo-ipa preparation script]
      • Overcloud stack: FAILED. Reason: ERROR: Can't run container nova_db_sync

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master:

      • Overcloud deploy failed with ERROR: Can't run container mysql_wait_bundle
      • Tempest tests FAILED

    • periodic-tripleo-ci-centos-9-scenario001-standalone-master:

      • TODO

    • periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master:

      • LP#2105408?
      • LP#2105408?

    • reruns 1

    • reruns 2 (internal)

• C9 Wallaby

  • last_promotion: 2022-09-13
  • hash 84acb1bac5b6c640738a43e63c532c2c

    • periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby:

      • TODO

    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-wallaby:

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby:

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby:

      • TODO

    • periodic-tripleo-ci-centos-8-9-multinode-mixed-os (not in criteria):

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby:

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-internal-wallaby (not in criteria):

      • TODO

    • periodic-tripleo-ci-centos-9-8-multinode-mixed-os (not in criteria):

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby (not in criteria):

      • TODO

    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby (not in criteria):

      • TODO

    • reruns 1

• C8 Wallaby

  • last_promotion: 2022-09-11
  • hash 5b62e82eebd71259ed086d27d7a2abae
    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-wallaby:
      • TASK [run-test : run toci_gate_test.sh] failed with HTTPConnectionPool(host='mirror.regionone.ibm-bm2-nodepool.rdoproject.org', port=80): Read timed out.
      • overcloud deploy failed with Error: 'rabbitmq-bundle' already exists. Too many tries
      • overcloud deploy failed with tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time
    • reruns 1

• C8 Train

  • last_promotion: 2022-09-12
  • hash 54e3db5e5134970e993eeb7d124da84c
    • periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train:
      • tempest failures
      • other (?) tempest failures
    • reruns
    • hash: 4e3db5e5134970e993eeb7d124da84c missing on fs035 - rerun https://review.rdoproject.org/r/c/testproject/+/31165

• RHOS-17.1 on RHEL-9 Refer to Bhagyashris' notes below

• RHOS-17.1 on RHEL-8 Refer to Bhagyashris' notes below

• RHOS-17.0 on RHEL-9 Refer to Bhagyashris' notes below

• RHOS-16.2 on RHEL-8 Refer to Bhagyashris' notes below

Component Lines

• C9 Master:

  • baremetal

    • last_promotion: 2022-09-13
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master:
      • TODO

  • clients

    • last_promotion: 2022-09-09
    • periodic-tripleo-ci-centos-9-undercloud-upgrade-clients-master (not in criteria):
      • TODO

  • common

    • last_promotion: 2022-09-11
    • periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master (not in criteria):
      • TODO

  • compute

    • last_promotion: 2022-09-07
    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-compute-master:
      • TODO

  • manila

    • last_promotion: 2022-09-12
    • periodic-tripleo-ci-centos-9-scenario004-standalone-manila-master:
      • TODO

  • tempest

    • last_promotion: 2022-09-05
    • periodic-tripleo-ci-centos-9-standalone-tempest-master:
      • TODO
    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-master:
      • TODO

  • tripleo

    • last_promotion: 2022-09-12
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-master:
      • TODO

  • reruns

• C9 Wallaby:

  • common

    • last_promotion: 2022-09-10
    • periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby (not in criteria):
      • TODO

  • network

    • last_promotion: 2022-09-13
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby:
      • TODO

  • tempest

    • last_promotion: 2022-09-07
    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-tempest-wallaby:
      • TODO

  • tripleo

    • last_promotion: 2022-09-12
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby:
      • TODO

  • reruns

• C8 Wallaby:

  • network

    • last_promotion: 2022-09-09
    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby:
      • TODO

  • tripleo

    • last_promotion: 2022-09-13
    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-wallaby:
      • TODO

  • reruns

• C8 Train:

  • network

    • last_promotion: 2022-09-09
    • periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train (not in criteria):
      • TODO

  • tripleo

    • last_promotion: 2022-09-10
    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train:
      • TODO

  • reruns

• RHOS-17.0 on RHEL-9:

  • Refer to Bhagyashris' notes below

• RHOS-16.2 on RHEL-8:

  • Refer to Bhagyashris' notes below

Downstream:

• rhos16.2 on rhel8:

  • Promotion: 13-sept-2022
    • blocked due to fule-tempest api , scenario and fs020 re-running jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400

• rhos17.1 on rhel8

  • Promotion: 14-sept-2022
    • buildimage overcloud full and tempest api job failed re-running here: https://code.engineering.redhat.com/gerrit/c/testproject/+/424992

• rhos17 on rhel9:

  • Promotion: 14-Sept-2022
    • Note: we don't have new content to promote.

• rhos17.1 on rhel9:

  • Promoted: 12-sept-2022

Downstream component:

Known Bugs

• https://bugs.launchpad.net/tripleo/+bug/1960355

  • Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

  • affecting OSP17.1 as well

• https://bugs.launchpad.net/tripleo/+bug/1987092

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  • Pacemaker performance causes intermittent galera issues in loaded CI env
  • jobs fail on Check Keystone public endpoint status or in other places
  • zuul/builds
  • affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• https://bugzilla.redhat.com/2105408

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

  • RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)

  • reproducible only on RHEL 8.2 and RHEL 8.4 … RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.

  • latest status is tracked in cix card
  • blocking integration lines of c9 master (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master) and c9 wallaby (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby)
  • workaround (specifying cpu_models in libvirt) is currently being tested

• https://bugs.launchpad.net/tripleo/+bug/1988514

  • Trello
  • On CIX board in lane Waiting on external dependencies

• https://bugs.launchpad.net/tripleo/+bug/1987323

  • FIPS + Manila + Ceph Quincy

  • … known issue on the ceph side, … already merged in master/quincy … 17.2.4 should solve this issue, so we'll check during the next promotion.

  • On CIX board in lane Waiting on external dependencies
  • Not a blocker

• https://bugs.launchpad.net/tripleo/+bug/1988810

  • Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788

• https://bugs.launchpad.net/tripleo/+bug/1989197

  • Tempest test neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON fails on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

• https://bugs.launchpad.net/tripleo/+bug/1989341

  • Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch

• https://bugs.launchpad.net/tripleo/+bug/1989247

  • overcloud deploy fails with sudo: unable to send audit message: Operation not permitted

  • Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master

• https://bugs.launchpad.net/tripleo/+bug/1989452

  • multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host

• https://bugs.launchpad.net/tripleo/+bug/1989606

  • container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use"

Intermittent Failures

• https://bugs.launchpad.net/tripleo/+bug/1987616

  • cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'

  • no CIX card

• https://bugs.launchpad.net/tripleo/+bug/1938987

  • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train

  • ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train

• TASK [freeipa-setup : Run the tripleo-ipa preparation script]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby

• Connection failure: The read operation timed out in TASK [Get current autoholds]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby

• Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • this has different root causes, watch out for log details!

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby
  • FAILURE in tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby

• FATAL | Wait for puppet host configuration to finish | undercloud | error…

  • TASK [tripleo.operator.tripleo_undercloud_install : undercloud install] in job-output.txt
  • puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages: in logs/undercloud/home/zuul/undercloud_install.log.txt.gz
  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds" during overcloud deploy

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

Fixed

2022-09-13

Reruns and Investigations:

(crossed out := known bugs recorded above)

NOTE: Watch for running testproject jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

• Promotions

  • C9 Master

    • last_promotion: 2022-09-09
    • hash 71abe3aa5c8ad3da6d894dc93d9c0273
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master:
        • overcloud deploy failed with Clean up legacy Cinder keystone catalog entries
        • intermittent failure TASK [freeipa-setup : Deploy FreeIPA]
      • periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master:
        • tempest failures
        • other tempest failures
      • periodic-tripleo-ci-centos-9-standalone-full-tempest-scenario-master:
        • other tempest failures
      • periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master:
        • overcloud deploy failed with tripleoclient.exceptions.HeatPodMessageQueueException: Message queue for ephemeral heat not created in time
        • undercloud install failed in Manage package | undercloud with Failed to download packages: double-conversion-3.1.5-6.el9.x86_64: Cannot download, all mirrors were already tried without success
      • periodic-tripleo-ci-centos-9-scenario010-kvm-standalone-master (not in criteria):
        • failed in TASK [standalone : Write containers-prepare-parameters.yaml] with Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-5 (not in criteria):
        • overcloud deploy failed with ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud ConnectionRefusedError: [Errno 111] Connection refused
      • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master:
        • other² tempest failures
        • standalone deploy failed in ensure apache is installed with Failed to download packages: apr-1.7.0-11.el9.x86_64: Cannot download, all mirrors were already tried without success
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master:
        • other³ tempest failures
        • LP#1987092
      • periodic-tripleo-ci-centos-9-bm_envC-3ctlr_1comp-featureset001-master (not in criteria):
        • cannot install the best update candidate for package gnutls-utils-3.7.3-10.el9.x86_64
          • will be fixed by chandan's revert
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm-2 (not in criteria):
        • node provisioning failed
      • reruns 1
      • reruns 2 (internal)
      • reruns 3
      • reruns 4
      • reruns 5
      • reruns 6

  • C9 Wallaby

    • last_promotion: 2022-09-09 2022-09-13
    • hash f6749f9cda54d017021ab56bf4ec0958:
      • yesterdays hash has passed today
    • hash ea8639e875ec8d7ee7158bce2cd3fa2b
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby:
        • TODO
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-rbac-wallaby (not in criteria):
        • TODO
      • periodic-tripleo-ci-centos-9-8-multinode-mixed-os (not in criteria):
        • TODO
      • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby:
        • TODO
      • reruns 1

  • C8 Wallaby

    • last_promotion: 2022-09-11
    • hash 21d2e33704a2f14d611cf4edff39f70e
      • periodic-tripleo-ci-centos-8-skipped-standalone-wallaby (not in criteria):
        • TODO

  • C8 Train

    • last_promotion: 2022-09-12
    • hash 54e3db5e5134970e993eeb7d124da84c
      • periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train:
        • TODO
      • periodic-tripleo-centos-8-buildimage-overcloud-full-train:
        • TODO
      • reruns

  • RHOS-17.1 on RHEL-9 Refer to Bhagyashris' notes below

  • RHOS-17.1 on RHEL-8 Refer to Bhagyashris' notes below

  • RHOS-17.0 on RHEL-9 Refer to Bhagyashris' notes below

  • RHOS-16.2 on RHEL-8 Refer to Bhagyashris' notes below

• C9 Master:

  • baremetal
    • last_promotion: 2022-09-12
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-master:
      • tempest failures
      • other tempest failures
  • network
    • last_promotion: 2022-09-08
    • periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master (not in criteria):
      • TODO
    • periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-master:
      • older runs:
        • 2022-09-12: LP#1989197
        • 2022-09-12: LP#1989197
      • LP#1989197
      • LP#1989197
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-master:
      • tempest failures
      • overcloud deploy failed with ERROR: Can't run container rabbitmq_bootstrap
  • security
    • last_promotion: 2022-09-11
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master:
      • TODO
  • reruns

• C9 Wallaby:

  • common
    • last_promotion: 2022-09-10
    • periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby (not in criteria):
      • TODO
  • tripleo
    • last_promotion: 2022-09-12
    • periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby:
      • overcloud deploy failed with ERROR: Can't run container container-puppet-ovn_controller
  • reruns

• C8 Wallaby:

  • all GREEN 😱
  • reruns

• C8 Train:

  • network
    • last_promotion: 2022-09-09
    • periodic-tripleo-ci-centos-8-standalone-full-tempest-api-network-train (not in criteria):
      • TODO
  • tripleo
    • last_promotion: 2022-09-10
    • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train:
      • overcloud introspection failed
  • reruns

• RHOS-17.0 on RHEL-9:

  • Refer to Bhagyashris' notes below

• RHOS-16.2 on RHEL-8:

  • Refer to Bhagyashris' notes below

Downstream:

• rhos16.2 on rhel8:

  • Promotion: 13-sept-2022
    • blocked due to sc010 re-running jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/ PASSED

• rhos17.1 on rhel8

  • Promotion: 12-sept-2022

• rhos17 on rhel9:

  • Promotion: 11-Sept-2022
    • failed fs001 job so re-ran here : https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : PASSED

• rhos17.1 on rhel9:

  • Promoted: 12-sept-2022
  • blocked due full tempest api so re-ran here: https://code.engineering.redhat.com/gerrit/c/testproject/+/422400/10#message-60300b428a51d8e0c5639502640ead15c847b402 : PASSED
  • jenkins job not triggerd:
    • Note : Pinged atila: - After discussion with atila we got to know that currently we don't have trigger jobs for 17.1 on rhel9 and 8 so i am adding trigger job for 17.1 on 9 and 8 here: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/428003
    • ~~fix for trigger jekins component rhso17.1 on rhel9 jobs is here: https://code.engineering.redhat.com/gerrit/c/rhos-qe-jenkins/+/428056 ~~

Downstream component:

Known Bugs

• https://bugs.launchpad.net/tripleo/+bug/1960355

  • Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)

  • affecting OSP17.1 as well

• https://bugs.launchpad.net/tripleo/+bug/1987092

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  • Pacemaker performance causes intermittent galera issues in loaded CI env
  • jobs fail on Check Keystone public endpoint status or in other places
  • zuul/builds
  • affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• https://bugzilla.redhat.com/2105408

  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →
  BLOCKER
  Image Not Showing Possible Reasons

  • The image file may be corrupted
  • The server hosting the image is unavailable
  • The image path is incorrect
  • The image format is not supported

  Learn More →

  • RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)

  • reproducible only on RHEL 8.2 and RHEL 8.4 … RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.

  • latest status is tracked in cix card
  • blocking integration lines of c9 master (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master) and c9 wallaby (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby)
  • workaround (specifying cpu_models in libvirt) is currently being tested

• https://bugs.launchpad.net/tripleo/+bug/1988514

  • Trello
  • On CIX board in lane Waiting on external dependencies

• https://bugs.launchpad.net/tripleo/+bug/1987323

  • FIPS + Manila + Ceph Quincy

  • … known issue on the ceph side, … already merged in master/quincy … 17.2.4 should solve this issue, so we'll check during the next promotion.

  • On CIX board in lane Waiting on external dependencies
  • Not a blocker

• https://bugs.launchpad.net/tripleo/+bug/1988810

  • Revert https://review.rdoproject.org/r/c/rdo-jobs/+/44788

• https://bugs.launchpad.net/tripleo/+bug/1989197

  • Tempest test neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON fails on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

• https://bugs.launchpad.net/tripleo/+bug/1989341

  • Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch

• https://bugs.launchpad.net/tripleo/+bug/1989247

  • overcloud deploy fails with sudo: unable to send audit message: Operation not permitted

  • Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master

• https://bugs.launchpad.net/tripleo/+bug/1989452

  • multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host

Intermittent Failures

• https://bugs.launchpad.net/tripleo/+bug/1987616

  • cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'

  • no CIX card

• https://bugs.launchpad.net/tripleo/+bug/1938987

  • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train

  • ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')

  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
  • FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train

• TASK [freeipa-setup : Run the tripleo-ipa preparation script]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby
  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-wallaby

• Connection failure: The read operation timed out in TASK [Get current autoholds]

  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby

• Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)

  • this has different root causes, watch out for log details!

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby
  • FAILURE in tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby

• FATAL | Wait for puppet host configuration to finish | undercloud | error…

  • TASK [tripleo.operator.tripleo_undercloud_install : undercloud install] in job-output.txt
  • puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages: in logs/undercloud/home/zuul/undercloud_install.log.txt.gz
  • POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby

• Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds" during overcloud deploy

  • FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

Fixed