Ruck Rover 2022-08-05 to 2022-08-11

# Ruck Rover 2022-08-05 to 2022-08-11 ###### tags: `ruck_rover` ###### Previous RR notes: https://hackmd.io/H9CSoXvlTm6nTZ4bsJkeRg [Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) [Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com) [OpenStack Program Meeting 2022]( https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit) [Downstream promoter](http://10.0.110.143/promoter_logs/) --- ## 2022-08-11 ### New/Transient no bug yet ### New bugs today #### ~~https://bugs.launchpad.net/tripleo/+bug/1985031~~ tripleo-tox-molecule consistently failing with failed: [localhost] (item=https://images.rdoproject.org/centos8/train/rdo_trunk/current-tripleo/ironic-python-agent.tar) ### Gate ongoing blocker https://bugs.launchpad.net/tripleo/+bug/1984175/comments/9 ### Check Jobs ### master blocked cix https://bugs.launchpad.net/bugs/1984184 ### wallaby c9 blocked cix https://bugs.launchpad.net/tripleo/+bug/1984453 ### wallaby c8 chasing https://review.rdoproject.org/r/c/testproject/+/44517 Run missing fs1 job for wallaby/8 c36ce5aec4d97093b683c04f3ee56212 ### train chasing https://review.rdoproject.org/r/c/testproject/+/44516 Run missing train jobs for 29018450b497197a282c6d4463b4c745 ### rhos17.1 on rhel9 #### https://bugzilla.redhat.com/show_bug.cgi?id=2116287 - major blocker in d/stream atm for across all releases. We havent promoted single line due to this infra issue. * chasing for promotion : https://code.engineering.redhat.com/gerrit/c/testproject/+/286307 (https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/d1d6dfe2a5b14ca9ab1830a33deee460) ### Promotions : (https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit#heading=h.hqhtw5tvhd63) * OSP 16.2 RHEL-8 promoted on Aug 4, 2022 * OSP 17 RHEL-8 Promoted on Aug 5, 2022 * OSP-17 RHEL-9 Promoted on Aug 7, 2022 * OSP-17.1 RHEL-9 Promoted on July 27, 2022 ### rhos17 on rhel9 * chasing for promotion : https://code.engineering.redhat.com/gerrit/c/testproject/+/209874 ### rhos16.2 * chasing for promotion : https://code.engineering.redhat.com/gerrit/c/testproject/+/276230 ### Upstream components #### master #### wallaby #### wallaby c8 #### train ### Downstream components #### rhos17 on rhel9 * network and tripleo components are promoted on 6 Aug * now cloudops and tempest components are lagging - not consistent issues so rerunning with testproject : https://code.engineering.redhat.com/gerrit/c/testproject/+/266024 #### rhos16.2 * network and tripleo still lagging due to fs01 with node_failures atm. * rerunning sc10 as issue is not consistent one : https://code.engineering.redhat.com/gerrit/c/testproject/+/266024 --- ## 2022-08-10 ### New/Transient no bug yet * ~~https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-undercloud-upgrade&skip=0~~ https://bugs.launchpad.net/tripleo/+bug/1984175 ### New bugs today #### https://bugs.launchpad.net/tripleo/+bug/1984175 tripleo-ci-centos-9-undercloud-upgrade - cannot install both NetworkManager-1:1.39.5-1.el9.x86_64 and NetworkManager-1:1.39.12 #### https://bugs.launchpad.net/tripleo/+bug/1984453 periodic-tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset002-master is failing image build - dependency fence-agents-common = 4.10.0-28.el9 #### https://bugs.launchpad.net/tripleo/+bug/1984184 fs001 and fs035 OVB jobs failing to set up private network for os_tempest ( patch under test https://review.rdoproject.org/r/c/testproject/+/36254) #### (NON VOTING) https://bugs.launchpad.net/tripleo/+bug/1984237 [FIPS] Standalone deploy failing with: "Error in GnuTLS initialization: Error while performing self checks" ### Gate ~~green~~ * looks like new issue undercloud-upgrade ### Check Jobs ### master ongoing upstream RETRY bug https://bugs.launchpad.net/tripleo/+bug/1983817/comments/10 ### wallaby c9 ### wallaby c8 ### train ### rhos17.1 on rhel9 ### rhos17 on rhel9 ### rhos17 on rhel8 ### rhos16.2 ### Upstream components #### master #### wallaby c9 #### wallaby c8 #### train ### Downstream components #### rhos17 on rhel9 #### rhos17 on rhel8 #### rhos16.2 --- ## 2022-08-09 ### New bugs today #### https://bugs.launchpad.net/tripleo/+bug/1984035 seen on cs8 wallaby while uploading the image ### Gate green ### Check Jobs ### master ongoing issue all branches https://bugs.launchpad.net/tripleo/+bug/1983817/comments/5 * vexx ticket: https://support.vexxhost.com/hc/en-us/requests/362706 * miss 1/35 recheck https://review.rdoproject.org/r/c/testproject/+/44460/2#message-1fa61d1a0d0fca87173d4b04842ebdfdb69d3c0d * even with this we have to skip for https://code.engineering.redhat.com/gerrit/c/testproject/+/423882 (https://bugzilla.redhat.com/show_bug.cgi?id=2116287) * another hash here also 1/35 https://review.rdoproject.org/r/c/testproject/+/44475 for more comparison so we can skip * latest failures on OVB are all: "Failed to download metadata for repo 'quickstart-centos-appstreams'" https://opendev.org/openstack/openstack-ansible-os_tempest/commit/f8c8a1ed6c59dcbf1fbe66d137d715e53af2ff51 looks like the latest commit taken by openstack-os_tempest ... marked as the master branch as of 07/13. https://opendev.org/openstack/openstack-ansible-os_tempest/graph show other available commits. Maybe check with Arx impact of this change - or moving the branch. Tried these jobs on IBM cloud - fails at the same point. ### wallaby c9 * missing fs64 there https://review.rdoproject.org/r/c/testproject/+/44472 ### wallaby c8 ### train * lets go with https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44477 * miss fs35 https://review.rdoproject.org/r/c/testproject/+/44442/4#message-829ecd515910cd3ac3dc1d9417243f3b1081b29d * miss fs1 https://review.rdoproject.org/r/c/testproject/+/44462/3#message-a19d911453cfa39a2174e88a08dc3bf6c98c8dc1 ### rhos17.1 on rhel9 * watching failed jobs with latest hash https://code.engineering.redhat.com/gerrit/c/testproject/+/286307 * still waiting for a response from kforde. Please check in with daniel tomorrow - node falures continues - no ovb stack ### rhos17 on rhel9 ### rhos17 on rhel8 ### rhos16.2 ### Upstream components #### master #### wallaby c9 #### wallaby c8 #### train ### Downstream components #### rhos17 on rhel9 #### rhos17 on rhel8 #### rhos16.2 * periodic-tripleo-ci-rhel-8-scenario010-standalone-network-rhos-16.2 only latest build failed previous green history so rechecking. * periodic-tripleo-ci-rhel-8-scenario007-standalone-network-rhos-16.2 latest one build history is good, only one failure with retry-limit and one with node_failure thats why rechecking. * periodic-tripleo-ci-rhel-8-standalone-network-rhos-16.2 build history good except one retry limit and one node failure thats why rechecking * periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2 right now failing with retry_limit the real issue we actually facing now. https://bugzilla.redhat.com/show_bug.cgi?id=2116287 * periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2 (currently failing with node_failures/retry_limits) ``` 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.Controller.2.Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500" 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.Controller.2]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: Build of instance 278b9381-5621-435c-9b8e-0fd6e83e4898 aborted: Failed to provision instance 278b9381-5621-435c-9b8e-0fd6e83e4898: Failed to prepare to de 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.ComputeIpListMap.EnabledServicesValue]: CREATE_IN_PROGRESS state changed 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.ComputeIpListMap.EnabledServicesValue]: CREATE_COMPLETE state changed 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.ComputeIpListMap]: CREATE_COMPLETE Stack CREATE completed successfully 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.Controller.2]: CREATE_FAILED ResourceInError: resources[2].resources.Controller: Went to status ERROR due to "Message: Build of instance 278b9381-5621-435c-9b8e-0fd6e83e4898 aborted: Failed to provision instance 278b9381-5621-435c-9b8e-0fd6e83e4898: Failed to prepare to deploy: IPMI 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.Controller]: UPDATE_FAILED Resource CREATE failed: ResourceInError: resources[2].resources.Controller: Went to status ERROR due to "Message: Build of instance 278b9381-5621-435c-9b8e-0fd6e83e4898 aborted: Failed to provision instance 278b9381-5621-435c-9b8e-0fd6e83e4898: Failed to 2022-08-05 15:20:16 | 2022-08-05 15:20:01Z [overcloud.ComputeIpListMap]: CREATE_COMPLETE state changed 2022-08-05 15:20:16 | 2022-08-05 15:20:02Z [overcloud.Controller]: CREATE_FAILED resources.Controller: Resource CREATE failed: ResourceInError: resources[2].resources.Controller: Went to status ERROR due to "Message: Build of instance 278b9381-5621-435c-9b8e-0fd6e83e4898 aborted: Failed to provision instance 278b9381-5621-435c-9b8e-0f 2022-08-05 15:20:16 | 2022-08-05 15:20:02Z [overcloud]: CREATE_FAILED Resource CREATE failed: resources.Controller: Resource CREATE failed: ResourceInError: resources[2].resources.Controller: Went to status ERROR due to "Message: Build of instance 278b9381-5621-435c-9b8e-0fd6e83e4898 aborted: Failed to provision instance 27 ``` ### New/Transient no bug yet * promoter missing images for c8/wallaby (holding on revert for https://review.rdoproject.org/r/c/rdo-jobs/+/44441 ) * http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log-20220807 * chkumar working on re-push images --- ## 2022-08-08 ### New Bugs today: #### https://bugs.launchpad.net/tripleo/+bug/1983817 periodic integration all branches RETRY Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org #### https://bugzilla.redhat.com/show_bug.cgi?id=2116287 No promotions occurs due to NODE_FAILURE at downstream on weekend since 7th August ### Gate green ### Check Jobs ### master chasing master 3acadba0c3986b5a074088073042b411 https://review.rdoproject.org/r/c/testproject/+/44460 https://code.engineering.redhat.com/gerrit/c/testproject/+/423882 looks like a real issue on fs001 and fs035: TASK [os_tempest : Ensure private network exists]503 Service Unavailable (example logs in testproject) https://logserver.rdoproject.org/60/44460/2/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-master/b2eee76/job-output.txt ### wallaby c9 Chasing failures at: https://review.rdoproject.org/r/c/testproject/+/36255 f40013fd0fe770cbca616a1b53663936 is missing: internal-kvm (rerun in https://code.engineering.redhat.com/gerrit/c/testproject/+/423882), fs020 and fs039 (two reruns in testproject above so can compare results) ### wallaby c8 Hash_under_test=https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=31a61a906cdcd8fccc322d597aa1dd5f All tests passed ... **promoter is pushing containers but failing to find images**: http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log-20220807 pinged Amol and Chandan - pls follow up. ### train chasing train * recheck https://review.rdoproject.org/r/c/testproject/+/44442/4#message-14062031d9b8e0b9bebbd95150f76ce3f09f2a74 (just fs35 * chase second hash posted https://review.rdoproject.org/r/c/testproject/+/44462 04837c814032381ae4e9d817c276a22 - rerunning with fs001 and scenario007 For hash: e04837c814032381ae4e9d817c276a22 - only missing fs001: comparing two logs: https://logserver.rdoproject.org/openstack-periodic-integration-stable4/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/1f0a584/logs/undercloud/var/log/tempest/stestr_results.html.gz and https://logserver.rdoproject.org/62/44462/1/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train/75ded79/logs/undercloud/var/log/tempest/stestr_results.html.gz inconsistent test failures - **can skip promote this hash** (latest one to run in train) ### rhos17.1 on rhel9 * rerunning few jobs with old hash to promote https://code.engineering.redhat.com/gerrit/c/testproject/+/286307 (https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/6b594ee62d9d40729ebd1a7e6108e255) ### rhos17 on rhel9 * promoted yesterday(7th august) ### rhos17 on rhel8 * https://code.engineering.redhat.com/gerrit/c/testproject/+/209874 ### rhos16.2 * rerunning few jobs with old hash https://code.engineering.redhat.com/gerrit/c/testproject/+/276230 (https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/buildset/cd1bd2bee295456bb009084f3a2229d7) ### Upstream components #### master #### wallaby c9 #### wallaby c8 #### train ### Downstream components #### rhos17 on rhel9 * no blockers #### rhos17 on rhel8 * tripleo component laggig * periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17 (posted test run - https://code.engineering.redhat.com/gerrit/c/testproject/+/266024) * https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/job/pipeline_component-tripleo-pcci-17_dlrn-rhel-8.4-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph/ (pinged attila - job having consistent failure) #### rhos16.2 * network component laggig * https://code.engineering.redhat.com/gerrit/c/testproject/+/266024 ### New/Transient no bug yet * ~~d/stream all builds BUILD_FAILURE (https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?skip=0)~~ https://bugzilla.redhat.com/show_bug.cgi?id=2116287 * unable to login horizon/down? (ping rhos-ops dpawlik) * ~~u/stream all integration builds hit by Could not resolve host: mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org~~ https://bugs.launchpad.net/tripleo/+bug/1983817 * eg https://review.rdoproject.org/zuul/buildset/73812fd92e0f42978a461fd1cb890697 * 12:49 < ysandeep> marios|ruck, pojadhav|rover ^^ jfyi.. old stack cleanup script was disabled on vexx for debug(causing the list of old stack to grow and we start hitting ""Quota exceeded for resources"), we just enabled it back. --- ## 2022-08-05 ### New Bugs today: #### https://bugzilla.redhat.com/show_bug.cgi?id=2115778 #### https://bugs.launchpad.net/tripleo/+bug/1983718 periodic master scen1 standalone fails/timeout 'manage firewall rules' **Component: validations: - 12 days out** https://review.rdoproject.org/r/c/testproject/+/44413 real issue: https://logserver.rdoproject.org/13/44413/1/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation/a965b91/job-output.txt pinged Jiri ### Gate green https://lists.openstack.org/pipermail/openstack-discuss/2022-August/029865.html ### Check Jobs ### master - blocked new bug https://bugs.launchpad.net/tripleo/+bug/1983718 ### wallaby c9 - promoted today ### wallaby c8 ### train - chasing https://review.rdoproject.org/r/c/testproject/+/44442 Run missing jobs train 8cf307aefe47066dfa2b89be39b174f8 && https://code.engineering.redhat.com/gerrit/c/testproject/+/423713 Run periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-train * train fs035 may have a legit issue now: see https://logserver.rdoproject.org/42/44442/4/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train/d31038e/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz. Pls bug if it repeats ### rhos17.1 on rhel9 * Testing a patch : https://code.engineering.redhat.com/gerrit/c/testproject/+/420272 (failed sc01 and sc02 standalone with same issue ceilometer_gnocchi_upgrade) ### rhos17 on rhel9 * https://code.engineering.redhat.com/gerrit/c/testproject/+/286307 ### rhos17 on rhel8 * https://code.engineering.redhat.com/gerrit/c/testproject/+/286307 ### rhos16.2 * No blockers ### Upstream components #### master #### wallaby c9 #### wallaby c8 #### train ### Downstream components #### rhos17 on rhel9 * network component lagging * periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-network-rhos-17 * periodic-tripleo-ci-rhel-9-scenario010-standalone-network-rhos-17 #### rhos17 on rhel8 * reported a bug for tripleo component https://bugzilla.redhat.com/show_bug.cgi?id=2115778 Tried https://code.engineering.redhat.com/gerrit/c/testproject/+/418637 with depends-on: https://code.engineering.redhat.com/gerrit/c/openstack-tripleo-heat-templates/+/423661. Looks like that setting is on the controller but the same error is still there. Do we need to set it somewhere else? #### rhos16.2 * network component lagging - https://code.engineering.redhat.com/gerrit/c/testproject/+/266024/ ## 2022-08-04 previous ruck|rover shift - notes @ https://hackmd.io/H9CSoXvlTm6nTZ4bsJkeRg

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.