Ruck Rover - 2022-06-10 - 2022-06-16

tags: ruck_rover

Previous RR notes: https://hackmd.io/HMjD8MouQXmGWMJP5o76JQ

Cockpit

Downstream cockpit

Handover Notes

Active Bugs

Ceph

• https://bugs.launchpad.net/tripleo/+bug/1979093 - Intermittent failure adding user 'ceph-admin', exit code: 9

  • https://review.opendev.org/c/openstack/tripleo-ansible/+/846530
  • (moved to 17-23 board)

• https://bugs.launchpad.net/tripleo/+bug/1978998 - periodic-tripleo-ci-centos-9-scenario001-standalone failed to download the ceph container during bootstrap

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846231 - green ready for review
  • Tested by https://review.rdoproject.org/r/c/testproject/+/36256
  • Which was green w/ depends-on to https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474 (LP 1978997 - WA disable gnocchi)
  • However, test project periodic job was still using newer PCS containers and needs to be changed to use downgraded PCS containers; next step update job to use old PCS containers and see result (moved to 17-23 board)

• https://bugs.launchpad.net/tripleo/+bug/1978956 - branch-override jobs are failing Generate Ceph Spec step

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846159 merged

NotCeph

• [IMP][No Fix]https://bugs.launchpad.net/tripleo/+bug/1978997 - tripleo-ci-centos-9-scenario001-standalone failed during step5 because gnocchi couldn't connect to redis

  • https://review.opendev.org/c/openstack/tripleo-common/+/846287 - merged
  • Still seeing bug; have a WA to disable telemetry https://review.opendev.org/c/openstack/tripleo-heat-templates/+/846474

• https://bugs.launchpad.net/tripleo/+bug/1978996 - multinode-ipa job is failing with "No package openssl-3.0.1-5.el9 available"

  • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/846125/ - merged

• https://bugs.launchpad.net/tripleo/+bug/1978969 - tripleo-ci-centos-9-standalone fails tempest test with error - 'Failed to attach network adapter device'

  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/846194 - merged

• https://bugs.launchpad.net/tripleo/+bug/1978929 - resource-agents-4.10.0-17.el9.x86_64.rpm: Downloading successful, but checksum doesn't match. Calculated and expected checksum different

  • CI is unblocked for today, might come tomorrow also.

• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

  • Need a proper bug report based on fs039 failures
  • Hit a testproject just to confirm the root cause in case in future you see these jobs failing

• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

  • Track the health of fs01 and fs035

• https://bugzilla.redhat.com/show_bug.cgi?id=2097443 - [RHOS-17][RHEL-9]Overcloud image builds are failing with "libselinux-x-3.4-2" install/update errors

  • Fix is merged now, waiting on tripleo wallaby component promotion and rhos-17 rhel-9 tripleo component promotion

CIXes to close or revisit

• https://trello.com/c/CQRTV134/2558-cixlp1977873tripleociproa-master-wallaby-deployments-are-failing-with-selinux-boolean-osenablevtpm-does-not-exist
• https://trello.com/c/9PnDFQ0i/2559-cixlp1977888tripleociproa-cics8train-diskimage-builder-requires-a-different-python-368-not-in-38
• https://trello.com/c/mLDYyWxC/2555-cixbz2094257osp17rhel9additional-partition-dev-sda4-in-rhel-guest-image-90-202204200x8664qcow2-guest-image-is-breaking-the-conve

Jobs to actively monitor

• kvm internal, fs035, fs01, fs039

Promotion Status

• Master :- 15 june
• wallaby c9:- 13 june
• wallaby c8:- 13 june
• train c8:- 13 june

Downstream

• RHOS-17 RHEL-9 - 13th
• RHOS-17 RHEL-8 - 15th
• RHOS-16 RHEL-8 - 14th

Active Bugs

• Check handover bugs list

16 June

• Promote tripleo wallaby component to get https://review.opendev.org/c/openstack/diskimage-builder/+/846052

  • https://review.rdoproject.org/r/c/testproject/+/43621

• cs9 wallaby promotion re-run fs039 mirror issue: https://review.rdoproject.org/r/c/testproject/+/43622

• train promotion: https://review.rdoproject.org/r/c/testproject/+/43623

component pipelines

• Rekicking c9 components:- https://review.rdoproject.org/r/c/testproject/+/40171
• Rekicking c9 integration line failures:- https://review.rdoproject.org/r/c/testproject/+/40102

Ignore everything below this line

15 June

Active Bugs

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 image build issue in ovb jobs
• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'
• https://bugs.launchpad.net/tripleo/+bug/1978482 - Evaluation Error: Error while evaluating a Resource Statement, Apache::Vhost[keystone_wsgi]
• https://bugs.launchpad.net/tripleo/+bug/1972163 - cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
• https://bugs.launchpad.net/tripleo/+bug/1978319 - cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
• https://bugzilla.redhat.com/show_bug.cgi?id=2096125 [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
• https://bugs.launchpad.net/bugs/1971465 - fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

cs9 wallaby tempest component

• https://bugs.launchpad.net/tripleo/+bug/1978553 - ExternalGatewayForFloatingIPNotFound: External network is not reachable from subnet Therefore, cannot associate Port with a Floating IP
• https://bugs.launchpad.net/tripleo/+bug/1978556 - Multiple tempest tests failure with ValueError: Unable to setup floating IP for validation: port not found on tenant network RHOS-17 RHEL-8 promoted today!

component pipelines

• master(compute, security)
  • compute line is running
  • validation needs a rerun - rekcicked line
  • security - fs39 is failing - rekicked here:- https://review.rdoproject.org/r/c/testproject/+/40083
• c9 wallaby(compute)- got promoted yesterday
• c8 wallaby(network, tripleo)
  • https://review.rdoproject.org/r/c/testproject/+/40932
  • https://review.rdoproject.org/r/c/testproject/+/42595

CIX cards we can clear today

• https://trello.com/c/5nFneypz/2566-cixlp1978456tripleociproa-cs9-fs02-ovb-job-is-failing-at-image-build-dib-run-parts-ignoring-non-executable-files-05-selinux-9-st
• https://trello.com/c/sZFnUP6h/2569-cixlp1978131tripleociproa-tripleo-puppet-fails-to-set-up-keystone-container-during-standalone-deployment
• https://trello.com/c/EuVXuVVO/2570-cixlp1978553tripleociproa-externalgatewayforfloatingipnotfound-external-network-is-not-reachable-from-subnet-therefore-cannot-as
• https://trello.com/c/lQNtKdRh/2571-cixlp1978556tripleociproa-multiple-tempest-tests-failure-with-valueerror-unable-to-setup-floating-ip-for-validation-port-not-fou
• https://trello.com/c/oT1GERap/2568-cixlp1978482tripleociproa-evaluation-error-error-while-evaluating-a-resource-statement-apachevhostkeystonewsgi
• https://trello.com/c/PGuOikZG/2565-cixbz2096125rhos-17rhel-8overcloud-image-build-jobs-are-failing-openstack-ironic-python-agent-builder-1280-020220327061952e0b51e
• https://trello.com/c/wiOeCsdg/2563-cixlp1978319tripleociproa-cs8-octavia-cloudops-tempest-common-network-tripleo-validation-wallaby-fails-with-error-running-contai
• https://trello.com/c/2VHETeKz/2554-cixlp1977716tripleociproa-the-conditional-check-negativeresults-in-namevalue-failed-during-setfact

14 June

components pipeline

• master (validation, tripleo, security, baremetal, network) - Testproject:- https://review.rdoproject.org/r/c/testproject/+/40083 - tripleo..https://bugs.launchpad.net/tripleo/+bug/1978482

• wallaby c8(tripleo, network) - [network]Testproject:- https://review.rdoproject.org/r/c/testproject/+/40932 - [tripleo]Testproject:- https://review.rdoproject.org/r/c/testproject/+/42595

  • All components promoted

• Chasing master and train: https://review.rdoproject.org/r/c/testproject/+/39960

• Chasing master and rhos17 on rhel8 - https://code.engineering.redhat.com/gerrit/c/testproject/+/398466

• Issue with https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-standalone&skip=0 - fixed now

13 June

Promotions

• wallaby c9 promoted
• wallaby c8 should promote with current testproject run
• rhos-17 on rhel 9 should promote with current testproject run
• working on promoting wallaby c9 components (promoted so far: security, validation, baremetal, network) tempest and tripleo are in rerun

patches to clear fs02//fs01 / image build

• 845573: Make sure 05-selinux-9-stream is executable | https://review.opendev.org/c/openstack/tripleo-ci/+/845573
  • testproject: https://review.rdoproject.org/r/c/testproject/+/43586

Tempest patches

• 43404: Backport wait_until_sshable_pingable patches | https://review.rdoproject.org/r/c/openstack/tempest-distgit/+/43404
• https://code.engineering.redhat.com/gerrit/q/topic:wait_until_sshable_pingable
  • testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/414808
  • Ping mkopec or tosky to get it merged.

CS9 Master:

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded… iPXE boot failed, retrying… .'

• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

• https://bugs.launchpad.net/tripleo/+bug/1978482 - cs9-multinode/multinode-ipa/standalone/multinode-tripleo-master-validation/sc004-standalone/sc000-multinode-oooq-container-updates tripleo-master fails with RuntimeError: Ansible execution failed

Active Bugs

CS9 Wallaby / Master

• https://bugs.launchpad.net/tripleo/+bug/1977873

  • master/wallaby deployments are failing with "SELinux boolean os_enable_vtpm does not exist."
  • https://review.opendev.org/c/openstack/diskimage-builder/+/845189/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/845137/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/843007/
  • https://review.opendev.org/c/openstack/tripleo-quickstart/+/845

  1 - revert of workaround

  • issue seen in compute component- 8th june log
  • compute component - 9th june
  • waiting for fresh run

CS9 Master

• https://bugs.launchpad.net/tripleo/+bug/1972163
  • cinder tempest.api.compute.admin.test_volumes_negative* tempest tests failing randomly in multiple branches.
• https://bugs.launchpad.net/tripleo/+bug/1978298
• Rekicked master component jobs here:- https://review.rdoproject.org/r/c/testproject/+/42693

CS9 Wallaby

• https://bugs.launchpad.net/tripleo/+bug/1977716

  • The conditional check ''negative_results' in name.value ' failed during set_fact
  • fix: https://review.opendev.org/c/openstack/validations-common/+/844823
  • test project: https://review.rdoproject.org/r/c/testproject/+/42319
  • Follow up on this

• https://bugs.launchpad.net/tripleo/+bug/1964940

  • Compute tests are failing with failed to reach ACTIVE status and task state "None" within the required time.
  • verify this bug

CS8 Wallaby

• https://bugs.launchpad.net/tripleo/+bug/1978319
  • cs8 octavia/cloudops/tempest/common/network/tripleo/validation wallaby fails with "Error running container image prepare"
  • testproject: https://review.rdoproject.org/r/c/testproject/+/40101
  • Assigned to @frenzyfriday
  • fire a testproj and confirm

CS8 Train

• https://bugs.launchpad.net/tripleo/+bug/1977888
  • fix: https://review.opendev.org/c/openstack/tripleo-ci/+/845081
  • test project: https://review.rdoproject.org/r/c/testproject/+/42319
  • chasing fs035: https://review.rdoproject.org/r/c/testproject/+/36255

RHEL 9 RHOS17

• https://bugzilla.redhat.com/show_bug.cgi?id=2094257
  • fix: https://review.opendev.org/c/openstack/tripleo-quickstart/+/844952
  • test project: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027
  • Rekicked 17 rhel9 jobs here:- https://code.engineering.redhat.com/gerrit/c/testproject/+/398340
  • cross-verify

RHEL 8 RHOS17

• https://bugzilla.redhat.com/show_bug.cgi?id=2096125 [RHOS-17][RHEL-8] overcloud image build jobs are failing - openstack-ironic-python-agent-builder-1:2.8.0-0.20220327061952.e0b51e0.el8osttrunk.noarch requires diskimage-builder >= 3.4.0
  • https://review.opendev.org/c/openstack/tripleo-ci/+/845544
    • https://code.engineering.redhat.com/gerrit/c/testproject/+/414668

Other

• https://bugs.launchpad.net/bugs/1971465
  • fs001 and fs035 OVB jobs failing tempest - identity/haproxy connection errors

13 June

CS9 Master:

• https://bugs.launchpad.net/tripleo/+bug/1978456 - cs9 fs02 ovb job is failing with error - 'No configuration methods succeeded… iPXE boot failed, retrying… .'
• https://bugs.launchpad.net/tripleo/+bug/1978458 - cs9 fs39 ovb job is failing with 'RuntimeError: Ansible execution failed.'

Fri, Jun 10

Promotions

CS9 Wallaby / Master

• https://review.rdoproject.org/r/c/testproject/+/36254
  • skip fs002: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43579
• components: https://review.rdoproject.org/r/c/testproject/+/4239
  • security component is failing due to libselinux 3.4-1 issues.

CS8 Wallaby

• https://review.rdoproject.org/r/c/testproject/+/40101

RHEL-8 RHOS-16.2

• Chasing two hashes: https://code.engineering.redhat.com/gerrit/c/testproject/+/398466 (last promo 06/06)

Thu Jun 09

Promotions

CS9 Wallaby / Master

• https://review.rdoproject.org/r/c/testproject/+/36254/
• https://review.rdoproject.org/r/c/testproject/+/42374
• waiting to see what tempest on fs001 looks like before promoting. Tons of failures there atm
• master also failng fs039 and fs064 and internal kvm
• wallaby only fs001
• https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43570 - to promote wallaby c9

13 June

https://bugs.launchpad.net/tripleo/+bug/1978298

• reckick the job

Transient Bugs

fs039 - series of unknown issues, difficult to triage 1

They all seem to be unrelated, but they're causing havoc to the line.

"Can't run container"

2 2022-06-09 22:58:35 | 2022-06-09 22:58:35.006643 | | WARNING | ERROR: Can't run container nova_api_ensure_default_cells […] 2022-06-09 22:58:35 | 2022-06-09 22:58:35.010171 | | WARNING | ERROR: Can't run container placement_api_db_sync […]

Internal Server Error Keystone

3 2022-06-10 15:08:13 | 2022-06-10 15:08:13.341227 | fa163e2d-7640-c8bc-5072-00000000a20c | TIMING | tripleo_keystone_resources : Create identity service | undercloud | 0:29:54.730355 | 1.67s 2022-06-10 15:08:13 | 2022-06-10 15:08:13.352842 | fa163e2d-7640-c8bc-5072-00000000a20d | TASK | Create identity public endpoint 2022-06-10 15:08:17 | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)

4 [Fri Jun 10 15:08:17.218323 2022] [wsgi:error] [pid 17:tid 38] [remote 172.17.0.184:33542] mod_wsgi (pid=17): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/keystone'.

SSH Permission denied

5 2022-06-10 18:05:50 | 2022-06-10 18:05:50.412897 | fa163e10-0939-a651-5d12-000000001759 | FATAL | Run tripleo_os_net_config_module with network_config | overcloud-controller-2 | error={"msg": "Data could not be sent to remote host "192.168.24.30". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.30' (ED25519) to the list of known hosts.\r\nheat-admin@192.168.24.30: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,keyboard-interactive).\r\n"}

Cannot download ansible-macros

6 2022-06-09 15:03:03 | Error: Error downloading packages: 2022-06-09 15:03:03 | ansible-macros-2021.1.2-2.el9s.noarch: Cannot download, all mirrors were already tried without success

Failed to download packages: mod_lua-2.4.51-8

7 2022-06-06 22:01:03.474740 | fa163ec0-bb0e-7746-1f99-000000000cb2 | FATAL | ensure apache is installed | undercloud | error={"changed": false, "msg": "Failed to download packages: mod_lua-2.4.51-8.el9.x86_64: Cannot download, all mirrors were already tried without success", "results": []}

overcloud-2 didn't start

8 One of 3 overcloud nodes didn't start.