Ruck Rover - 2022-06-17 - 2022-06-23

tags: ruck_rover

Previous RR notes: https://hackmd.io/gw8f6Y3hQcatR_l2PhhY9g

Next RR notes: https://hackmd.io/9hv3vTNlST2rw014LSDqcg

Cockpit

Downstream cockpit

June 25

• Ceph patches merged in both master and wallaby, sc001 and sc004 should be green now.
• Missing Ceph container in quay.rdoproject.org:
  • Master content provider is failing due to: "Failed to pull following containers: ['quay.rdoproject.org/tripleomastercentos9/daemon:current-ceph']
  • https://quay.rdoproject.org/repository/tripleomastercentos9/daemon?tab=history
    • Why "current-ceph" tag was deleted?
    • Need to check with Daniel, he is the responsible for the prune tags in quay.rdoproject.org. For now @arxcruz restore the tag.
  • content provider failure log
  • Is the fix supposed to be a manual restore of the missing tag?
    • manually restored (twice Thursday and Saturday)
    • should be fixed by https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/25329 - Merged Monday
    • 27 Jun: we expend this to be fixed but will ping dpawlik if it is missing again

June 24

Patches requiring attention

• https://bugs.launchpad.net/tripleo/+bug/1979665

  • Patch below affected by this bug
  • Saw it on tripleo-ci-centos-9-standalone
  • recheck'd (known intermittent issue)

• tripleo_cephadm_ceph_cli is undefined

  • https://bugs.launchpad.net/tripleo/+bug/1979651
  • fixed by https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2
    • failed tripleo-ci-centos-9-standalone because of https://bugs.launchpad.net/tripleo/+bug/1979665
  • tested in master TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847357
    • sc001: Success
    • sc004: Success
  • tested in wallaby TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847329
    • sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
    • sc004: Success
  • tested in wallaby periodic https://review.rdoproject.org/r/c/testproject/+/37973
    • sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
    • sc004: Success
  • tested in master periodic https://review.rdoproject.org/r/c/testproject/+/36256
    • sc001: Success
    • sc004: Success~~
  • Updates:
    • Master patch (847323) in gate
    • Wallaby patch (847257) -> gate

• Periodic Job Failure (gate)

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231
    • intermittently failed for LP 1979093 (above)
    • Tested by https://review.rdoproject.org/r/c/testproject/+/36256

June 23

Patches requiring attention

• https://bugs.launchpad.net/tripleo/+bug/1979665

  • Patch below affected by this bug
  • Saw it on tripleo-ci-centos-9-standalone
  • recheck'd (known intermittent issue)

• tripleo_cephadm_ceph_cli is undefined

  • https://bugs.launchpad.net/tripleo/+bug/1979651
  • fixed by https://review.opendev.org/q/I137e335abeedccad801cdc03feee654c3e42a0e2
    • failed tripleo-ci-centos-9-standalone because of https://bugs.launchpad.net/tripleo/+bug/1979665
  • tested in master TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847357
    • sc001: Success
    • sc004: Success
  • tested in wallaby TripleO https://review.opendev.org/c/openstack/tripleo-heat-templates/+/847329
    • sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
    • sc004: Success
  • tested in wallaby periodic https://review.rdoproject.org/r/c/testproject/+/37973
    • sc001: Failure (Failed containers: gnocchi_db_sync, ceilometer_gnocchi_upgrade)
    • sc004: Success
  • tested in master periodic https://review.rdoproject.org/r/c/testproject/+/36256
    • sc001: Success
    • sc004: Success
  • Updates:
    • Master patch (847323) in gate

• https://bugs.launchpad.net/tripleo/+bug/1979093 Intermittent failure adding user 'ceph-admin'

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530
  • needs to be tested in wallaby before merging in master

• Periodic Job Failure (gate)

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231
    • intermittently failed for LP 1979093 (above)
    • Tested by https://review.rdoproject.org/r/c/testproject/+/36256

• 001/002 standalone failing on gnocchi bug (fixed)

  • https://bugs.launchpad.net/tripleo/+bug/1978997
  • See the job run without pacemaker 2.1.3-2.el9
    • 2.1.4 exists so we were downgrading back to the bad version
    • https://review.opendev.org/c/openstack/tripleo-common/+/846287 - revert https://review.opendev.org/c/openstack/tripleo-common/+/847166 fails
    • https://review.opendev.org/c/openstack/tripleo-common/+/847222 - Merged

Active Bugs

• https://bugs.launchpad.net/tripleo/+bug/1979646 fs01 network wallaby component failing on neutron/dhcp-agent - reported today, need neutron/hardprov team involvement to resolve this

• https://bugs.launchpad.net/tripleo/+bug/1979665 - standalone network wallaby failing on network tempest tests

• https://bugs.launchpad.net/tripleo/+bug/1979546 - scenario010 kvm internal failing on octavia tests with error status

• https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing

  • This is also affecting other jobs

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis

  • Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

• https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

• https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004

  • merged

Promotions downsteam

• osp16.2 rhel8 - today (23 June 2022)
• osp17 rhel8 - 20th June 2022
• osp17 rhel9 - 21st June 2022

June 22

Patches requiring attention

• 001/002 standalone failing on gnocchi bug

  • https://bugs.launchpad.net/tripleo/+bug/1978997
  • Innocent victim: https://review.opendev.org/c/openstack/puppet-tripleo/+/845854 2022-22-06 18:34:38 UTC
  • Before we disable gnocchi https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474
  • See the job run without pacemaker 2.1.3-2.el9
    • conjecture: 2.1.4 exists now so we're downgrading back to the bad version
    • https://review.opendev.org/c/openstack/tripleo-common/+/846287 - revert https://review.opendev.org/c/openstack/tripleo-common/+/847166 fails
    • https://review.opendev.org/c/openstack/tripleo-common/+/847222 - passes - pls merge

• Wallaby: Ceph Failing on sc001 and sc004 (not master):

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 merged
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846940 backport (merged)
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846950 merged
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/847163 backport (after 846940 merges) (merged)

• https://bugs.launchpad.net/tripleo/+bug/1979093 Intermittent failure adding user 'ceph-admin'

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

• Periodic Job Failure

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 (in check now)
    • intermittently failed for LP 1979093 (above)

Infra and missing ceph contianer

• mirror issue prevented merging of patches - see #opendev

• not here either: https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/

  • fixed

• 'podman pull quay.rdoproject.org/tripleowallabycentos9/daemon:current-ceph' failed so arxcruz manually went in the repository to restore the tag

Active Bugs

• https://bugs.launchpad.net/tripleo/+bug/1979546 - scenario010 kvm internal failing on octavia tests with error status

• https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing

  • This is also affecting other jobs

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis

  • Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

• https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

• https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004

  • merged

June 21

• mirror issue preverting merging of patches - pls see #opendev
• not here either: https://mirror.facebook.net/centos-stream/9-stream/BaseOS/x86_64/os/repodata/

Patches requiring attention

• Need to merge https://review.opendev.org/c/openstack/tripleo-ansible/+/846999 to unblock scenario001 and scenario004

  • +w'd

• Wallaby is failing on sc001 and sc004 as well:

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846950
  • we need to backport this change ^ once it is merged

• Is https://review.rdoproject.org/r/c/testproject/+/36256 not using redis-bundle container with RPMs from https://review.opendev.org/c/openstack/tripleo-common/+/846287 ?

  • This will confirm the fix for https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap
  • So we don't need https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474 (LP 1978997 - WA disable gnocchi)
  • If we add in the testptoject under vars add "build_container_images: true", will it stop periodic from pulling the usptream ceph?

Active Bugs

• https://bugs.launchpad.net/tripleo/+bug/1979276 - puppet-glance-tripleo-standalone job failing

  • This is also affecting other jobs

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis

  • Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

• https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user ‘ceph-admin’, exit code: 9

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530

Backlog (bugs fixed with workaround)

• https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

June 20

Active Bugs

• https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis

  • Patch to disable legacy telemetry - https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

Backlog (bugs fixed with workaround)

• https://bugs.launchpad.net/tripleo/+bug/1978969 - tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'

  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 - merged

• https://bugs.launchpad.net/tripleo/+bug/1978929 - resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different

  • CI is unblocked for today, might come tomorrow also.

• https://bugzilla.redhat.com/show_bug.cgi?id=2097443 - [RHOS-17][RHEL-9]Overcloud image builds are failing with "libselinux-x-3.4-2" install/update errors

  • Fix is merged now, waiting on tripleo wallaby component promotion and rhos-17 rhel-9 tripleo component promotion

June 17

Active Bugs

Phase1

• https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007 is causing phase1 to fail because quay.rdoproject.org is not tagging current-tripleo

Ceph

• https://bugs.launchpad.net/tripleo/+bug/1978956 - branch-override jobs are failing Generate Ceph Spec step

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231

MaybeCeph

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis
  • [fultonj] Asked mrunge for input

NotCeph

• https://bugs.launchpad.net/tripleo/+bug/1978996 - multinode-ipa job is failing with "No package openssl-3.0.1-5.el9 available"

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125/ - merged

• https://bugs.launchpad.net/tripleo/+bug/1978969 - tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'

  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 - merged

• https://bugs.launchpad.net/tripleo/+bug/1978929 - resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different

  • CI is unblocked for today, might come tomorrow also.

• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

  • Need a proper bug report based on fs039 failures
  • Hit a testproject just to confirm the root cause in case in future you see these jobs failing

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

• https://bugzilla.redhat.com/show_bug.cgi?id=2097443 - [RHOS-17][RHEL-9]Overcloud image builds are failing with "libselinux-x-3.4-2" install/update errors

  • Fix is merged now, waiting on tripleo wallaby component promotion and rhos-17 rhel-9 tripleo component promotion

CIXes to close or revisit

• https://trello.com/c/CQRTV134/2558-cixlp1977873tripleociproa-master-wallaby-deployments-are-failing-with-selinux-boolean-osenablevtpm-does-not-exist
• https://trello.com/c/9PnDFQ0i/2559-cixlp1977888tripleociproa-cics8train-diskimage-builder-requires-a-different-python-368-not-in-38
• https://trello.com/c/mLDYyWxC/2555-cixbz2094257osp17rhel9additional-partition-dev-sda4-in-rhel-guest-image-90-202204200x8664qcow2-guest-image-is-breaking-the-conve

Jobs to actively monitor

• kvm internal, fs035, fs01, fs039

Promotion Status

• Master :- 15 june
• wallaby c9:- 13 june
• wallaby c8:- 13 june
• train c8:- 13 june

Downstream

• RHOS-17 RHEL-9 - 13th
• RHOS-17 RHEL-8 - 15th
• RHOS-16 RHEL-8 - 14th - needs https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/pipeline/job/pipeline_integration-pcci-16.2_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/ to promote - Pooja pls watch this

Active Bugs

• Check handover bugs list

16 June

• Promote tripleo wallaby component to get https://review.opendev.org/c/openstack/diskimage-builder/+/846052

  • https://review.rdoproject.org/r/c/testproject/+/43621

• cs9 wallaby promotion re-run fs039 mirror issue: https://review.rdoproject.org/r/c/testproject/+/43622

• train promotion: https://review.rdoproject.org/r/c/testproject/+/43623

component pipelines

• Rekicking c9 components:- https://review.rdoproject.org/r/c/testproject/+/40171
• Rekicking c9 integration line failures:- https://review.rdoproject.org/r/c/testproject/+/40102

Ignore everything below this line

15 June

Active Bugs

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 image build issue in ovb jobs
• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'
• https://bugs.launchpad.net/tripleo/+bug/1978482 - Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[keystone_wsgi]
• https://bugs.launchpad.net/tripleo/+bug/1972163 - cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
• https://bugs.launchpad.net/tripleo/+bug/1978319 - cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
• https://bugzilla.redhat.com/show_bug.cgi?id=2096125 [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

cs9 wallaby tempest component

• https://bugs.launchpad.net/tripleo/+bug/1978553 - ExternalGatewayForFloatingIPNotFound: External network is not reachable from subnet Therefore, cannot associate Port with a Floating IP
• https://bugs.launchpad.net/tripleo/+bug/1978556 - Multiple tempest tests failure with ValueError: Unable to setup floating IP for validation: port not found on tenant network RHOS-17 RHEL-8 promoted today!

component pipelines

• master(compute, security)
  • compute line is running
  • validation needs a rerun - rekcicked line
  • security - fs39 is failing - rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40083
• c9 wallaby(compute)- got promoted yesterday
• c8 wallaby(network, tripleo)
  • https://review.rdoproject.org/r/c/testproject/+/40932
  • https://review.rdoproject.org/r/c/testproject/+/42595

CIX cards we can clear today

• https://trello.com/c/5nFneypz/2566-cixlp1978456tripleociproa-cs9-fs02-ovb-job-is-failing-at-image-build-dib-run-parts-ignoring-non-executable-files-05-selinux-9-st
• https://trello.com/c/sZFnUP6h/2569-cixlp1978131tripleociproa-tripleo-puppet-fails-to-set-up-keystone-container-during-standalone-deployment
• https://trello.com/c/EuVXuVVO/2570-cixlp1978553tripleociproa-externalgatewayforfloatingipnotfound-external-network-is-not-reachable-from-subnet-therefore-cannot-as
• https://trello.com/c/lQNtKdRh/2571-cixlp1978556tripleociproa-multiple-tempest-tests-failure-with-valueerror-unable-to-setup-floating-ip-for-validation-port-not-fou
• https://trello.com/c/oT1GERap/2568-cixlp1978482tripleociproa-evaluation-error-error-while-evaluating-a-resource-statement-apachevhostkeystonewsgi
• https://trello.com/c/PGuOikZG/2565-cixbz2096125rhos-17rhel-8overcloud-image-build-jobs-are-failing-openstack-ironic-python-agent-builder-1280-020220327061952e0b51e
• https://trello.com/c/wiOeCsdg/2563-cixlp1978319tripleociproa-cs8-octavia-cloudops-tempest-common-network-tripleo-validation-wallaby-fails-with-error-running-contai
• https://trello.com/c/2VHETeKz/2554-cixlp1977716tripleociproa-the-conditional-check-negativeresults-in-namevalue-failed-during-setfact

14 June

components pipeline

• master (validation, tripleo, security, baremetal, network) - Testproject:- https://review.rdoproject.org/r/c/testproject/+/40083 - tripleo..https://bugs.launchpad.net/tripleo/+bug/1978482

• wallaby c8(tripleo, network) - [network]Testproject:- https://review.rdoproject.org/r/c/testproject/+/40932 - [tripleo]Testproject:- https://review.rdoproject.org/r/c/testproject/+/42595

  • All components promoted

• Chasing master and train: https://review.rdoproject.org/r/c/testproject/+/39960

• Chasing master and rhos17 on rhel8 - https://code.engineering.redhat.com/gerrit/c/testproject/+/398466

• Issue with https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone&skip=0 - fixed now

13 June

Promotions

• wallaby c9 promoted
• wallaby c8 should promote with current testproject run
• rhos-17 on rhel 9 should promote with current testproject run
• working on promoting wallaby c9 components (promoted so far: security, validation, baremetal, network) tempest and tripleo are in rerun

patches to clear fs02//fs01 / image build

• 845573: Make sure 05-selinux-9-stream is executable | https://review.opendev.org/c/openstack/tripleo-ci/+/845573
  • testproject: https://review.rdoproject.org/r/c/testproject/+/43586

Tempest patches

• 43404: Backport wait_until_sshable_pingable patches | https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/43404
• https://code.engineering.redhat.com/gerrit/q/topic:wait_until_sshable_pingable
  • testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/414808
  • Ping mkopec or tosky to get it merged.

CS9 Master:

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded… iPXE boot failed, retrying… .'

• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

• https://bugs.launchpad.net/tripleo/+bug/1978482 - cs9-multinode/multinode-ipa/standalone/multinode-tripleo-master-validation/sc004-standalone/sc000-multinode-oooq-container-updates tripleo-master fails with RuntimeError: Ansible execution failed

Active Bugs

CS9 Wallaby / Master

• https://bugs.launchpad.net/tripleo/+bug/1977873

  • master/wallaby deployments are failing with "SELinux boolean os_enable_vtpm does not exist."
  • https://review.opendev.org/c/openstack/diskimage-builder/+/845189/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/845

  1 - revert of workaround

  • issue seen in compute component- 8th june log
  • compute component - 9th june
  • waiting for fresh run

CS9 Master

• https://bugs.launchpad.net/tripleo/+bug/1972163
  • cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
• https://bugs.launchpad.net/tripleo/+bug/1978298
• Rekicked master component jobs here:- https://review.rdoproject.org/r/c/testproject/+/42693

CS9 Wallaby

• https://bugs.launchpad.net/tripleo/+bug/1977716

  • The conditional check ''negative_results' in name.value ' failed during set_fact
  • fix: https://review.opendev.org/c/openstack/validations-common/+/844823
  • test project: https://review.rdoproject.org/r/c/testproject/+/42319
  • Follow up on this

• https://bugs.launchpad.net/tripleo/+bug/1964940

  • Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
  • verify this bug

CS8 Wallaby

• https://bugs.launchpad.net/tripleo/+bug/1978319
  • cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
  • testproject: https://review.rdoproject.org/r/c/testproject/+/40101
  • Assigned to @frenzyfriday
  • fire a testproj and confirm

CS8 Train

• https://bugs.launchpad.net/tripleo/+bug/1977888
  • fix: https://review.opendev.org/c/openstack/tripleo-ci/+/845081
  • test project: https://review.rdoproject.org/r/c/testproject/+/42319
  • chasing fs035: https://review.rdoproject.org/r/c/testproject/+/36255

RHEL 9 RHOS17

• https://bugzilla.redhat.com/show_bug.cgi?id=2094257
  • fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952
  • test project: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027
  • Rekicked 17 rhel9 jobs here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398340
  • cross-verify

RHEL 8 RHOS17

• https://bugzilla.redhat.com/show_bug.cgi?id=2096125 [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
  • https://review.opendev.org/c/openstack/tripleo-ci/+/845544
    • https://code.engineering.redhat.com/gerrit/c/testproject/+/414668

Other

• https://bugs.launchpad.net/bugs/1971465
  • fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

13 June

CS9 Master:

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded… iPXE boot failed, retrying… .'
• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

Fri, Jun 10

Promotions

CS9 Wallaby / Master

• https://review.rdoproject.org/r/c/testproject/+/36254
  • skip fs002: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43579
• components: https://review.rdoproject.org/r/c/testproject/+/4239
  • security component is failing due to libselinux 3.4-1 issues.

CS8 Wallaby

• https://review.rdoproject.org/r/c/testproject/+/40101

RHEL-8 RHOS-16.2

• Chasing two hashes: https://code.engineering.redhat.com/gerrit/c/testproject/+/398466 (last promo 06/06)

Thu Jun 09

Promotions

CS9 Wallaby / Master

• https://review.rdoproject.org/r/c/testproject/+/36254/
• https://review.rdoproject.org/r/c/testproject/+/42374
• waiting to see what tempest on fs001 looks like before promoting. Tons of failures there atm
• master also failng fs039 and fs064 and internal kvm
• wallaby only fs001
• https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43570 - to promote wallaby c9

13 June

https://bugs.launchpad.net/tripleo/+bug/1978298

• reckick the job

Transient Bugs

fs039 - series of unknown issues, difficult to triage 1

They all seem to be unrelated, but they're causing havoc to the line.

"Can't run container"

2 2022-06-09 22:58:35 | 2022-06-09 22:58:35.006643 | | WARNING | ERROR: Can't run container nova_api_ensure_default_cells […] 2022-06-09 22:58:35 | 2022-06-09 22:58:35.010171 | | WARNING | ERROR: Can't run container placement_api_db_sync […]

Internal Server Error Keystone

3 2022-06-10 15:08:13 | 2022-06-10 15:08:13.341227 | fa163e2d-7640-c8bc-5072-00000000a20c | TIMING | tripleo_keystone_resources : Create identity service | undercloud | 0:29:54.730355 | 1.67s 2022-06-10 15:08:13 | 2022-06-10 15:08:13.352842 | fa163e2d-7640-c8bc-5072-00000000a20d | TASK | Create identity public endpoint 2022-06-10 15:08:17 | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)

4 [Fri Jun 10 15:08:17.218323 2022] [wsgi:error] [pid 17:tid 38] [remote 172.17.0.184:33542] mod_wsgi (pid=17): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/keystone'.

SSH Permission denied

5 2022-06-10 18:05:50 | 2022-06-10 18:05:50.412897 | fa163e10-0939-a651-5d12-000000001759 | FATAL | Run tripleo_os_net_config_module with network_config | overcloud-controller-2 | error={"msg": "Data could not be sent to remote host "192.168.24.30". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.30' (ED25519) to the list of known hosts.\r\nheat-admin@192.168.24.30: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive).\r\n"}

Cannot download ansible-macros

6 2022-06-09 15:03:03 | Error: Error downloading packages: 2022-06-09 15:03:03 | ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success

Failed to download packages: mod_lua-2.4.51-8

7 2022-06-06 22:01:03.474740 | fa163ec0-bb0e-7746-1f99-000000000cb2 | FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: mod_lua-2.4.51-8.el9.x86_64: Cannot download, all mirrors were already tried without success", "results": []}

overcloud-2 didn't start

8 One of 3 overcloud nodes didn't start.