Ruck and rover notes #27

# Ruck and rover notes #27 ###### tags: `ruck_rover` :::info Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ) **Ruck Rover - Unified Sprint 27** Dates: May 6 - May 26 Tripleo CI team ruck|rover: Folco (rfolco) / Bhagyashris (bhagyashris) OSP CI team ruck|rover (April 24 - May 15): Waldek (wznoinsk) / Avi (TalmoR) Previous notes: https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg ::: [TOC] --- ## on-going issues :::danger ## TripleO * https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images) here is the fix: https://review.opendev.org/731498 (Added image sanity check condition) ### gate ### RDO CI * watch next periodic runs - https://review.opendev.org/#/c/730533 merged ## OSP * OSP17 still without attention (!) because of fires in OSP<16 * foreign jobs still invading p1/p2 views * not yet tested https://code.engineering.redhat.com/gerrit/#/c/198375 ::: --- :::info add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections. ::: ### Launchpad Bugs Reported :::spoiler | Bugzilla | Name | status | Review | | -------- | ---- |------- | ------ | | [1878101](https://bugs.launchpad.net/tripleo/+bug/1878101) | ping br-ctlplane is failing too often, "Trying to ping default gateway" | Complete |[727942](https://review.opendev.org/#/c/727942/) | | [1878190](https://bugs.launchpad.net/tripleo/+bug/1878190) | periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master job is consistently failing because of some tesmpest test are failing | Triged | [727192](https://review.opendev.org/#/c/727192/) | | [1878197](https://bugs.launchpad.net/tripleo/+bug/1878197) | periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train is failing and time_out on tempest execute | Triged | | | [1878150]( https://bugs.launchpad.net/tripleo/+bug/1878150) | tox-linters jobs failing with AttributeError | In Progress | [727113](https://review.opendev.org/#/c/727113/)| | [1878248](https://bugs.launchpad.net/tripleo/+bug/1878248 ) | NetworkSecGroupTest failing on fs020 stein/train | In Progress | [727287](https://review.opendev.org/727287)| | [1877031](https://bugs.launchpad.net/tripleo/+bug/1877031 ) | queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version | In Progress | [727696](https://review.opendev.org/#/c/727696/)| | [1879267](https://bugs.launchpad.net/tripleo/+bug/1879267) | Error: Failed to download metadata for repo \'advanced-virtualization\'\n level=debug msg="error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \\"centos-binary-swift-container-working-container\\": error while running runtime: exit status 1 failing standlone deployment jobs | Complete | [728761](https://review.opendev.org/728761)| | [1879638](https://bugs.launchpad.net/tripleo/+bug/1879638) | [Train Only] Error: No matching repo to modify: epel failing container build push on centos-8 train | Complete | [729519](https://review.opendev.org/#/c/729519)| | [1880383](https://bugs.launchpad.net/tripleo/+bug/1880383 ) | ERROR! Unable to retrieve file contents, Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller. failing tripleo quickstart deployment on centos-8 (master, ussuri and train) | Complete | [730533](https://review.opendev.org/#/c/730533/ )| ::: ## May 28th ### Tripleo * Train and Stein get promoted recently. * Master promotion is blocked because of fs001 failure. * RDO CI Failures: - https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images) https://review.opendev.org/731498 (Added image sanity check condition) - **Master** : fs001 is failing on master and blocking promotion. Issue : https://bugs.launchpad.net/tripleo/+bug/1879766 (master ovb jobs failing on Destination directory /etc/pki/tls/private does not exist) - **Ussuri**: "periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri" is failing because of tempest test failures : https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri/903029a/logs/undercloud/var/log/tempest/stestr_results.html.gz "periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri" is getting time out consistently at Execute tempest test task. Submitted patch to increase the time out here : https://review.rdoproject.org/r/#/c/27811/ and testing the jobs: https://review.rdoproject.org/r/#/c/27789/ - **Train C7**: Currently most of the jobs are failing because of https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images), once this will patche will merge https://review.opendev.org/731498 (Added image sanity check condition) it will go green. ### OSP ## May 27th ### Tripleo * Gate: - scenario10 on stable/ussuri check pipeline: ~~(https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone&pipeline=check#) error log: The specified regex doesn't match with anything (https://88e1de9a81e55d590d5b-26f184bb59af339cfe698349cbda4177.ssl.cf5.rackcdn.com/731036/1/check/tripleo-ci-centos-8-scenario010-standalone/e355cdc/logs/undercloud/var/log/tempest/tempest_run.log) this is failing because these two files are not in sync https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_master.yml and https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_ussuri.yml~~ ~~Here is the fix: https://review.opendev.org/#/c/731149/ (Remove octavia tempest tests from skiplist)~~ * RDO CI Failures: - ussuri failing on scen010, fs001 and fs020 - scen010 --> https://bugs.launchpad.net/tripleo/+bug/1880981 - fs001/fs020 tempest tests - **Stein C7:** fs020 and fs035 is failing and blocking the promotion because of below issue and same jobs are passing on test patch https://review.rdoproject.org/r/#/c/27709/ . So as this is know and random failure. ~~~ 2020-05-27 07:21:18 | [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- 2020-05-27 07:21:18 | controller-0: Failed to connect to the host via ssh: Warning: Permanently added 2020-05-27 07:21:18 | '192.168.24.25' (ECDSA) to the list of known hosts. Permission denied 2020-05-27 07:21:18 | (publickey,gssapi-keyex,gssapi-with-mic). 2020-05-27 07:21:18 | 2020-05-27 07:21:18 | 2020-05-27 07:21:18 | TASK [Gathering Facts] ********************************************************* 2020-05-27 07:21:18 | fatal: [overcloud-novacompute-0]: UNREACHABLE! => { 2020-05-27 07:21:18 | "changed": false, 2020-05-27 07:21:18 | "unreachable": true 2020-05-27 07:21:18 | } 2020-05-27 07:21:18 | 2020-05-27 07:21:18 | MSG: 2020-05-27 07:21:18 | 2020-05-27 07:21:18 | Data could not be sent to remote host "192.168.24.15". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.15' (ECDSA) to the list of known hosts. 2020-05-27 07:21:18 | Permission denied (publickey,gssapi-keyex,gssapi-with-mic). ~~~ ### OSP ## May 26th ### Tripleo * Gate: - On cockpit Most of the failures are from patch: https://review.opendev.org/#/c/725623/ (Add ansible hieradata file) and all the failures are random one those are listed yesterday. * RDO CI failures: - **Stein c7** : [featureset020](https://review.rdoproject.org/zuul/builds?pipeline=%09openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein) is failing randomly , recently it failed because of below error, **Below error is known and random one so will need to keep eye on next run** Test patch: https://review.rdoproject.org/r/#/c/27709/ got **SUCCESS** on test patch. ~~~ 2020-05-26 07:14:15 | TASK [Gathering Facts] ********************************************************* 2020-05-26 07:14:15 | fatal: [overcloud-novacompute-1]: UNREACHABLE! => { 2020-05-26 07:14:15 | "changed": false, 2020-05-26 07:14:15 | "unreachable": true 2020-05-26 07:14:15 | } 2020-05-26 07:14:15 | 2020-05-26 07:14:15 | MSG: 2020-05-26 07:14:15 | 2020-05-26 07:14:15 | Data could not be sent to remote host "192.168.24.21". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.21' (ECDSA) to the list of known hosts. 2020-05-26 07:14:15 | Permission denied (publickey,gssapi-keyex,gssapi-with-mic). 2020-05-26 07:14:15 | 2020-05-26 07:14:15 | fatal: [overcloud-novacompute-0]: UNREACHABLE! => { 2020-05-26 07:14:15 | "changed": false, 2020-05-26 07:14:15 | "unreachable": true 2020-05-26 07:14:15 | } ~~~ - **Train c7:** [featureset020](https://review.rdoproject.org/zuul/builds?pipeline=%09openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train) failed recenlty after long time because of tempest test failure https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/2629c18/logs/undercloud/var/log/tempest/stestr_results.html.gz **will need to keep eye on this in next run** test patch: https://review.rdoproject.org/r/#/c/27657/ got **SUCCESS** on test patch. - **Train C8:** [standalone-full-tempest-api](https://review.rdoproject.org/zuul/builds?pipeline=%09openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train) job got failed after this patch [Revert "[install-deps] Use venv instead of virtualenv when possible"](https://review.opendev.org/#/c/730533) got merged because of tempest test failure https://logserver.rdoproject.org/53/27753/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/7ef9d7b/logs/undercloud/var/log/tempest/stestr_results.html.gz Test patch here : https://review.rdoproject.org/r/#/c/27753/ -> Same tempest test is failing on the patch : https://logserver.rdoproject.org/53/27753/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/7ef9d7b/logs/undercloud/var/log/tempest/stestr_results.html.gz . Will check for the next run result on test patch. Got **SUCCESS** on next run. Hope will pass on openstack-periodic-24hr pipeline as well. - **Ussuri**: fs001 is failing randomly with https://bugs.launchpad.net/tripleo/+bug/1879766 (ovb jobs failing on Destination directory /etc/pki/tls/private does not exist) https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/8f8eeda/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz Here doing regression testing : https://review.rdoproject.org/r/#/c/27750/ - **Master**: periodic-tripleo-centos-8-buildimage-overcloud-full-master is failing on current running openstack-peridic-master pipeline but it passed after that on testproject patch https://review.rdoproject.org/r/#/c/27754/**SUCCESS**. Test patch got **SUCCESS** in regression testing as well. Hope this will pass on next run on openstack-periodic-master pipeline. ### OSP ## May 25th ### Tripleo #### gate 13 failures 05-25-2020 3:50 UTC - no pattern, random failures, examples: - tempest.scenario.test_snapshot_pattern.TestSnapshotPattern failed on scen001 standalone (centos8) - Unable to disable service iscsid.socket on containers multinode (stein) - image prepare failed on containers multinode (train) - TIME OUT on undercloud-containers, containers-multinode, scen000 (train) - container-puppet tasks failed on centos7 standalone (train) - Wait for containers to start for step 3 using paunch >> scen003 standalone (train) - puppet host configuration failed on scen010 (train) - patches rechecked #### RDO CI Failures - ~~https://bugs.launchpad.net/tripleo/+bug/1880383 (ERROR! Unable to retrieve file contents, Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller. failing tripleo quickstart deployment on centos-8 (master, ussuri and train))~~ - ~~Fix: https://review.opendev.org/#/c/730533/ - Revert "[install-deps] Use venv instead of virtualenv when possible"~~ - ~~Test patch is here : https://review.rdoproject.org/r/#/c/27734/ -> **SUCCESSS**~~ ### ~~undercloud-upgrade failing in gate~~ ~~https://bugs.launchpad.net/tripleo/+bug/1876893~~ * ~~https://review.opendev.org/#/c/726008/2/tripleo_ansible/roles/tripleo-container-rm/tasks/tripleo_podman_container_rm.yml~~ * ~~https://review.opendev.org/#/c/725939/~~ * ~~Above patch is Abandoned in favor of https://review.opendev.org/#/c/726008/2 and this patch is merged~~ * ~~https://review.opendev.org/#/c/725944/ this patch is also merged~~ ### OSP ## May 22th ### TripleO * Gate failures: * python-tripleoclient * https://bugs.launchpad.net/tripleo/+bug/1879392 [build-test-packages] ERROR:dlrn:Received exception Error in build_rpm_wrapper for python-tripleoclient * tripleo-ci-centos-8-scenario001-standalone: this job is mostly failing becuase of tempest test failure and standalone deployment failure * Tempest failure: (https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ec/725250/6/gate/tripleo-ci-centos-8-scenario001-standalone/5ec570c/logs/undercloud/var/log/tempest/stestr_results.html) * Standalone deployment failure: https://6e5de493453a596bf54b-e12a251c5f6363a4d35eb8aac39c4442.ssl.cf2.rackcdn.com/729508/1/gate/tripleo-ci-centos-8-scenario001-standalone/270ea66/logs/undercloud/home/zuul/standalone_deploy.log * timeoute increase: https://review.opendev.org/730416 Increase scenario001 standalone timeout * RDO CI failures: train promotion failed due to: - periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train - https://bugs.launchpad.net/tripleo/+bug/1880236 - @rfolco fyi, This job is not a promotion blocker https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini#L32 - tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train - https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train# - TIME OUTs: https://review.rdoproject.org/r/27724 Increase timeout for fs035 train - @rfolco fyi, If you see, it's never time out here https://review.rdoproject.org/r/#/c/27644/ (test patch on testproject), even in openstack-periodic-24hr pipeline it got SUCCESS https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train master: * periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master: is failing consistently because of https://bugs.launchpad.net/tripleo/+bug/1879766 (master ovb jobs failing on Destination directory /etc/pki/tls/private does not exist): Not sure why this bug is marked as invalid. * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein: is getting time out because of some tempest test are running out of time. ### OSP ## May 21th ### Tripleo * Gate failures: * tripleo-ci-centos-7-scenario010-standalone is failing on gate/check https://zuul.opendev.org/t/openstack/builds?pipeline=gate&job_name=tripleo-ci-centos-7-scenario010-standalone%20 https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario010-standalone because of setUpClass(octavia_tempest_plugin.tests.scenario.v2.test_load_balancer) tempest test failure https://55e769a6be92d5846621-6d67d1055c684f7187f360a048e3d766.ssl.cf5.rackcdn.com/721577/1/gate/tripleo-ci-centos-7-scenario010-standalone/5ead591/logs/undercloud/var/log/tempest/stestr_results.html * Bug: https://bugs.launchpad.net/tripleo/+bug/1879941 ( tempest.lib.exceptions.NotFound: Object not found causes failure on stable/train tripleo-ci-centos-7-scenario010-standalone) * Fix: https://review.opendev.org/#/c/729926/ (Use tempest_tempestconf_profile_overrides for extra overrides) * Test patch: https://review.opendev.org/#/c/729930/ ([DNM] sc10 train gate) * RDO CI failures * periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train job is getting timeout https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train but on testproject it's working fine https://review.rdoproject.org/r/#/c/27644/ ### OSP ## May 20th ### Tripleo :::spoiler rfolco notes on gate failures #### msg: '[''mysql_init_bundle''] failed to start * https://47fba6cbbdf7110dae32-95a2136c445cc9f6f88a9d6c838f9ef5.ssl.cf2.rackcdn.com/728390/5/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/fcba19d/logs/undercloud/home/zuul/overcloud_update_run_Controller.log #### RUN END RESULT_TIMED_OUT * https://a266683f54be05fa2a12-0d0531abd29396cae25ad5b41413c8a1.ssl.cf5.rackcdn.com/727655/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/73ab7eb/job-output.txt * https://529238dce23327b6a1dc-af78d673c12faab1c666b4915d91f2fd.ssl.cf1.rackcdn.com/726875/1/gate/tripleo-ci-centos-8-containers-multinode/d8c33db/job-output.txt * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b44/726875/1/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/b44e6ad/job-output.txt * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_cb4/725623/1/gate/tripleo-ci-centos-7-containers-multinode/cb44c0d/job-output.txt #### async task did not complete within the requested time - 5700s * https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_a10/728891/1/gate/tripleo-ci-centos-8-scenario004-standalone/a10573d/job-output.txt #### Unable to start service tripleo_rabbitmq.service * https://f92cf8ddf16f455f65fe-3e4526f4e1e568120af910c41c844d04.ssl.cf2.rackcdn.com/729352/1/gate/tripleo-ci-centos-7-undercloud-containers/d8b34e2/logs/undercloud/home/zuul/install-undercloud.log #### FAILED - RETRYING: Wait for puppet host configuration to finish * https://88d7e0c140b2b7046387-cdf523d3b16150ab0b9ddc512a79d512.ssl.cf5.rackcdn.com/726875/1/gate/tripleo-ci-centos-8-scenario010-standalone/b8d254e/logs/undercloud/home/zuul/standalone_deploy.log #### tempest (network_basic_ops) https://bce0317ca743b3733203-ca4b089b4b338eb03b97f6a00e3061e2.ssl.cf5.rackcdn.com/729105/1/gate/tripleo-ci-centos-8-scenario004-standalone/494b55d/logs/undercloud/var/log/tempest/stestr_results.html ::: * master * fs001 https://bugs.launchpad.net/tripleo/+bug/1879766 * Container build push is failing on c8 train * ~~Reported Bug: https://bugs.launchpad.net/tripleo/+bug/1879638 ([Train Only] Error: No matching repo to modify: epel failing container build push on centos-8 train)~~ * ~~Fix: https://review.opendev.org/#/c/729519/ ([Train only] Fix for CentOS8 containers build)~~ * ~~Now build containers are failing here: https://logserver.rdoproject.org/34/27634/4/check/periodic-tripleo-centos-8-train-containers-build-push/b8bf039/logs/buildah-builds/kolla-builds/aed6abec-8abe-4cbd-883d-a26ed33d44a6/docker/cinder/cinder-base/cinder-base-build.log as the ceph-common is getting disabled because of this change https://review.opendev.org/#/c/721329/13/docker/base/ceph.repo~~ ``` STEP 1: FROM trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-openstack-base:b74e95c27fff46a3abf661fd6348b604cabf47db_34d2c7ae STEP 2: LABEL maintainer="Kolla Project (https://launchpad.net/kolla)" name="cinder-base" build-date="20200520" STEP 3: RUN usermod --append --home /var/lib/cinder --groups kolla cinder && mkdir -p /var/lib/cinder && chown -R 42407:42407 /var/lib/cinder STEP 4: RUN dnf -y install ceph-common lvm2 cryptsetup openstack-cinder python3-automaton python3-oslo-vmware && dnf clean all && rm -rf /var/cache/dnf Repository centos-opstools is listed more than once in the configuration delorean-openstack-keystone-b74e95c27fff46a3abf 3.7 MB/s | 1.2 MB 00:00 CentOS-8 - AppStream 7.2 MB/s | 7.0 MB 00:00 CentOS-8 - Base 2.2 MB/s | 2.2 MB 00:00 CentOS-8 - Extras 32 kB/s | 5.9 kB 00:00 CentOS-8 - HA 1.7 MB/s | 564 kB 00:00 CentOS-8 - NFS Ganesha 2.8 44 kB/s | 11 kB 00:00 CentOS-8 - OpsTools - collectd 231 kB/s | 117 kB 00:00 CentOS-8 - PowerTools 667 kB/s | 2.0 MB 00:03 dlrn-train-testing 2.4 MB/s | 976 kB 00:00 dlrn-train-build-deps 1.4 MB/s | 395 kB 00:00 Advanced Virtualization mirror 116 kB/s | 72 kB 00:00 Messaging RabbitMQ 257 kB/s | 80 kB 00:00 No match for argument: ceph-common Package lvm2-8:2.03.05-5.el8.0.1.x86_64 is already installed. Error: Unable to find a match: ceph-common ``` * ~~We will need to do some changes here to add those repos: https://opendev.org/openstack/tripleo-repos/src/branch/master/tripleo_repos/main.py#L46~~ * ~~where we get ceph for train ? https://trunk.rdoproject.org/centos8-master/deps/storage/storage8-ceph-nautilus/~~ * ~~added some work arounds solution on patch https://review.opendev.org/#/c/729519/ ~~ * ~~https://review.opendev.org/729893 qdrouterd: ignore failure of disabling epel repos~~ ### OSP ## May 19th ### TripleO Looks like stein fs020 hitting issue in overcloud-deploy.. waiting on repeat.. https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/27d5e4f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz ``` 2020-05-19 07:08:47 | 2020-05-19 07:08:47 | novacompute-1: Failed to connect to the host via ssh: Warning: Permanently 2020-05-19 07:08:47 | added '192.168.24.24' (ECDSA) to the list of known hosts. Permission denied 2020-05-19 07:08:47 | (publickey,gssapi-keyex,gssapi-with-mic). ``` ### OSP ## May 18th ### Tripleo * https://bugs.launchpad.net/tripleo/+bug/1879392 [build-test-packages] Run DLRN: Unexpected DLRN return code * https://bugs.launchpad.net/tripleo/+bug/1879379 - OVB failures on Vexx cloud: none of the requested nodes are available and match the resource class baremetal * https://bugs.launchpad.net/tripleo/+bug/1879365 - container build issue (found on gate) * ~~**Affecting check/gate** https://bugs.launchpad.net/tripleo/+bug/1879267 [Error: Failed to download metadata for repo \'advanced-virtualization\'\n level=debug msg="error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \\"centos-binary-swift-container-working-container\\": error while running runtime: exit status 1 failing standlone deployment jobs] https://review.opendev.org/728761 [Use centos proxy mirror for centos8 repos]- Patch is up~~ * ~~LP Bug: https://bugs.launchpad.net/tripleo/+bug/1879255 [openstack overcloud node introspect --all-manageable timing out and failing periodic tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master]~~ ~~Temporary reschedule openstack-periodic-master:https://review.rdoproject.org/r/#/c/27586/~~ * ~~Related patches:~~ ~~https://review.opendev.org/#/c/724822/ (Use latest version of python construct)~~ ~~https://review.rdoproject.org/r/#/c/27549/ (Update minimal version of construct to 2.9.39)~~ ~~https://review.rdoproject.org/r/#/c/27585/ ([DNM] promote baremetal component to unblock current-tripleo promotions)~~ * LP Bug: https://bugs.launchpad.net/tripleo/+bug/1879292 [ERROR paunch [ ] Error running ['podman', 'run', '--name', 'rabbitmq_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=rabbitmq_init_bundle' failing periodic tripleo-ci-centos-8 scenario004-standalone-train and ovb-1ctlr_1comp-featureset002-train] * Note: This is the consistent failure and not the promotion blocker. * wes, if possible add rabbit debug commands to https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/defaults/main.yml#L174 * rabbitmqctl status ### OSP ## May 15th ### TripleO :::spoiler Pipeline: [openstack-periodic-latest-released](https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released#) * ERROR: "Could not find or access '/home/zuul/workspace/.quickstart/vars/tempest_skip_ussuri.yml'" :::spoiler * Failing jobs: 1. periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri 2. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri 3. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-ussuri 4. periodic-tripleo-ci-centos-8-scenario012-standalone-ussuri 5. periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri 6. periodic-tripleo-ci-centos-8-scenario007-standalone-ussuri 7. periodic-tripleo-ci-centos-8-scenario008-standalone-ussuri 8. periodic-tripleo-ci-centos-8-scenario003-standalone-ussuri 9. periodic-tripleo-ci-centos-8-scenario002-standalone-ussuri 10. periodic-tripleo-ci-centos-8-scenario001-standalone-ussuri 11. periodic-tripleo-ci-centos-8-standalone-ussuri 12. periodic-tripleo-ci-centos-8-undercloud-containers-ussuri * [Error log](https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/1bf60ef/job-output.txt) ~~~ 2020-05-14 23:03:09.383163 | primary | TASK [Load tempest skiplist file] ********************************************** 2020-05-14 23:03:09.383171 | primary | Thursday 14 May 2020 23:03:09 +0000 (0:00:00.052) 0:44:05.425 ********** 2020-05-14 23:03:09.425360 | primary | fatal: [undercloud]: FAILED! => { 2020-05-14 23:03:09.425423 | primary | "ansible_facts": {}, 2020-05-14 23:03:09.425433 | primary | "ansible_included_var_files": [], 2020-05-14 23:03:09.425440 | primary | "changed": false, 2020-05-14 23:03:09.425448 | primary | "message": "Could not find or access '/home/zuul/workspace/.quickstart/vars/tempest_skip_ussuri.yml' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option" 2020-05-14 23:03:09.425470 | primary | } ~~~ * This issue will be fixed by https://review.opendev.org/#/c/728134/ and https://review.opendev.org/#/c/728133/ (add ussuri tempest skiplist) * Below jobs are failing bacuse of emit_releases_file.py: error: argument --stable-release: invalid choice: 'ussuri' (choose from 'newton', 'ocata', 'pike', 'queens', 'rocky', 'stein', 'train', 'master') * [Error log](https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri/1bf60ef/job-output.txt) ~~~ 2020-05-14 22:12:11.131318 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:113): main(): basename /home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset037.yml 2020-05-14 22:12:11.132520 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:113): main(): python3 /home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py --stable-release ussuri --featureset-file /home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset037.yml --output-file /home/zuul/workspace/logs/releases.sh --log-file /home/zuul/workspace/logs/emit_releases_file.log --distro-name centos --distro-version 8 --is-periodic 2020-05-14 22:12:11.316668 | primary | usage: emit_releases_file.py [-h] --stable-release 2020-05-14 22:12:11.316747 | primary | {newton,ocata,pike,queens,rocky,stein,train,master} 2020-05-14 22:12:11.316756 | primary | --distro-name {centos} --distro-version {7,8} 2020-05-14 22:12:11.316763 | primary | --featureset-file FEATURESET_FILE 2020-05-14 22:12:11.316770 | primary | [--output-file OUTPUT_FILE] [--log-file LOG_FILE] 2020-05-14 22:12:11.316776 | primary | [--upgrade-from] [--is-periodic] 2020-05-14 22:12:11.317979 | primary | emit_releases_file.py: error: argument --stable-release: invalid choice: 'ussuri' (choose from 'newton', 'ocata', 'pike', 'queens', 'rocky', 'stein', 'train', 'master') 2020-05-14 22:12:12.165646 | primary | ERROR 2020-05-14 22:12:12.165914 | primary | { 2020-05-14 22:12:12.165958 | primary | "delta": "0:00:03.616096", 2020-05-14 22:12:12.165991 | primary | "end": "2020-05-14 22:12:11.334283", 2020-05-14 22:12:12.166019 | primary | "msg": "non-zero return code", 2020-05-14 22:12:12.166078 | primary | "rc": 2, 2020-05-14 22:12:12.166111 | primary | "start": "2020-05-14 22:12:07.718187" 2020-05-14 22:12:12.166140 | primary | } 2020-05-14 22:12:12.214538 | 2020-05-14 22:12:12.214686 | PLAY RECAP 2020-05-14 22:12:12.214748 | primary | ok: 8 changed: 5 unreachable: 0 failed: 1 skipped: 11 rescued: 0 ignored: 0 2020-05-14 22:12:12.214779 | 2020-05-14 22:12:12.454582 | RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master] 2020-05-14 22:12:12.454795 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master] ~~~ * Failed job list: ~~~ 1. periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri 2. periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-ussuri ~~~ * Fix: https://review.opendev.org/#/c/723905/ (Add ussuri support for emit_releases_file.py --- ) ::: ### OSP * 16.0 - latest content went trhu (20200513.n.1) + phase3 ran * 16.1 - 20200511 p2 almost complete - p3 run today * 1 octavia Tempest failure TBD * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/49/ * needs investigation and escalation * 13.0 - as below (y-day)+ * work on CIX continues * latest BZ update: https://bugzilla.redhat.com/show_bug.cgi?id=1835828#c3 * on-going long-term isues: * OSP17 still without attention (!) because of fires in OSP<16 * foreign jobs still invading p1/p2 views not yet tested https://code.engineering.redhat.com/gerrit/#/c/198375 ## May 14th ### TripleO :::spoiler Pipeline: periodic-tripleo-centos-7-rocky-containers-build-push is getting time out from TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/build.log] https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-rocky-containers-build-push# * Here this job is getting time out from this task: ~~~ 2020-05-13 08:27:23.219557 | TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/build.log] 2020-05-13 10:21:06.985995 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/run.yaml@master] 2020-05-13 10:21:06.986377 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master] ~~~ * if you see here the containers are not build successfully: https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/cbf360b/logs/build.log * I have increase the time out here https://review.rdoproject.org/r/#/c/27542/ ([DNM] Test periodic-tripleo-centos-7-rocky-containers-build-push) and tested this job -> it's pass with increase timeout. * So one of the process is taking to much of time to push the build container that i can't fin till now (where exactly and which container is taking more time to build and push) * Work around: https://review.rdoproject.org/r/#/c/27550/ (Increase timeout for "periodic-tripleo-centos-7-rocky-containers-build-push") * i see this issue is with train as well and they increase the timeout here https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleoci.yaml#L349 ::: ### OSP * 16.0 retriggered blue, p3 kicked * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP16/job/multijob-phase2-osp-16.0-RHEL-8.1/9/ * 16.1 to be done tomorrow * OSP13 p1 failure BZed and CIXed: * https://trello.com/c/bGsC674i CIX * https://bugzilla.redhat.com/show_bug.cgi?id=1835828 BZ * 17 - ??? unknown who debugs it and in what state it is * IR master brekage: * breakage cause reverted seems to be: http://post-office.corp.redhat.com/archives/rhos-infrared/2020-May/msg00071.html * https://projects.engineering.redhat.com/browse/RHOSINFRA-3317 * https://code.engineering.redhat.com/gerrit/#/c/200394/ ## May 13th ### TripleO **gate** * tripleo-ci-centos-7-undercloud-upgrades jobs are failing on stable/queens : https://review.opendev.org/#/q/topic:queens-backports+(status:open+OR+status:merged) * LP Bug: https://bugs.launchpad.net/tripleo/+bug/1877031 [queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version] * Fix: https://review.opendev.org/#/c/727696/1 [Don't run tripleo-ci-centos-7-undercloud-upgrades in queens] * Update: On #rhos-ops channel there was some discussion between apevec and ykarale regarding this issue and apevec commented on LP Bug. He suggested deprecating this upgrade job. Note: Refer Comment #2 ### OSP * 16.1 jobs occupy queue * 180 jobs still after 3/4 days * 16.0 - promoting content 12.1 * getting jobs thru CI to reach some reasonable reporting state * solving situation around passed_phase2 promoted * %TODO add jira * 13 - live deployment reproduced * no CIX so far because still no clue what happens (no BZ yet) * 17 - nothing ## May 12th ### Tripleo #### openstack-periodic-24h * https://bugs.launchpad.net/tripleo/+bug/1878248 NetworkSecGroupTest failing on fs020 stein/train * https://review.opendev.org/727287 [skiplist] NetworkSecGroupTest timeout #### gate <bhagyashris|ruck> https://review.opendev.org/#/c/726993/ <bhagyashris|ruck> https://review.opendev.org/#/c/726004/ https://review.opendev.org/#/c/727113/ [linters refresh w/ afferent bugfixes] fixes this bug https://bugs.launchpad.net/tripleo/+bug/1878150 [tox-linters jobs failing with AttributeError] :::spoiler Pipeline: openstack-periodic-master https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master# * Job Failures: * periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master * panda has submitted fix for this job failure * Fix: https://review.rdoproject.org/r/#/c/26694/ , https://review.opendev.org/#/c/722677 * testproject: https://review.rdoproject.org/r/27435 Test ovn job fix on #26694 * periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master * Reported bug : https://bugs.launchpad.net/tripleo/+bug/1878190 * Note: For more details about above job failure you can check May 11th -> Pipeline: openstack-periodic-master section ::: :::spoiler Pipeline: openstack-periodic-latest-released https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released * Job Failures: * periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train * Reported Bug: https://bugs.launchpad.net/tripleo/+bug/1878197 * This is failing for long time or never worked: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train ::: ### OSP * Patience with octavia is gone * escalation: https://trello.com/c/pkHE0ay2 * ignoring until they will fix it * New OSP13 p1 failure * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase1-13_director-rhel-7.8-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/17/ * Testing of UMB triggering * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/umb-testing/97/ ## May 11th ### Tripleo :::spoiler Pipeline: Gate bang.. opened bug https://bugs.launchpad.net/tripleo/+bug/1878101 Seen this a few times today https://6d806783ed4dfdd971c5-158a33a5449e12f6f494625dd8517fb1.ssl.cf5.rackcdn.com/726374/1/gate/tripleo-ci-centos-8-undercloud-containers/2e3a947/logs/undercloud/home/zuul/undercloud_install.log https://e453f1d8808c5b6bd184-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf2.rackcdn.com/722662/1/gate/tripleo-ci-centos-8-undercloud-containers/ad8d1e6/logs/undercloud/home/zuul/undercloud_install.log TASK [AllNodesValidationConfig] ************************************************ Monday 11 May 2020 20:46:19 +0000 (0:00:01.555) 0:00:50.282 ************ fatal: [undercloud]: FAILED! => changed=true msg: non-zero return code rc: 1 stderr: '' stderr_lines: <omitted> stdout: |- Trying to ping default gateway 10.4.70.1...Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... Ping to 10.4.70.1 failed. Retrying... FAILURE 10.4.70.1 is not pingable. stdout_lines: <omitted> ::: :::spoiler Pipeline: openstack-periodic-master https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master# * Job Failures * periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master * Note: This job is not promotion blocker but it's consistently failing * Details: Tempest is failing https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/a86ab83/logs/undercloud/home/zuul/tempest/tempest.html.gz * Discussion on IRC: I have discussed this failure with <arxcruz> and chandankumar: ~~~ <bhagyashris|ruck> chandankumar, arxcruz this job -> "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master" is consistently failing https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master here <bhagyashris|ruck> chandankumar, arxcruz https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/a86ab83/logs/undercloud/home/zuul/tempest/tempest.html.gz <arxcruz> bhagyashris|ruck: i though panda|pto had fixed it <arxcruz> bhagyashris|ruck: you need to add an tempest option on that <arxcruz> bhagyashris|ruck: give me a few minutes and i'll show you <chandankumar> arxcruz, bhagyashris|ruck regarding ovn tempest, patch is coming soon ~~~ * Fix: https://review.rdoproject.org/r/#/c/26694/ https://review.opendev.org/#/c/722677/2 * periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master * Note: This job is not promotion blocker but it's consistently failing * Details: Execute tempest is failing https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/7788828/logs/undercloud/var/log/tempest/stestr_results.html.gz * Discussion on IRC: I have discussed this with arxcruz: ~~~ <bhagyashris|ruck> chandankumar, arxcruz is this know failure https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/7788828/logs/undercloud/var/log/tempest/stestr_results.html.gz <bhagyashris|ruck> chandankumar, arxcruz this job is consistently failing "periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master" <bhagyashris|ruck> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master <arxcruz> bhagyashris|ruck: seems to be random failure * derekh (~derekh@93.107.228.126) has joined <arxcruz> I would wait for a next run to double check, if continues to happen, open a bug * amoralej|off is now known as amoralej <arxcruz> actually, it's happening a lot... <arxcruz> bhagyashris|ruck: I would open a bug, and add it on the skip list <bhagyashris|ruck> arxcruz, yes, i checked last 4 failure and it's failing to execute the tempest test <bhagyashris|ruck> arxcruz, yes ack just ping once you open a bug ~~~ ::: :::spoiler Pipeline: openstack-periodic-latest-released https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released * Job Failures: * periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train * Note: this job is consistently Time out and not a promotion blocker as well. * Details: https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/extra/errors.txt.txt.gz https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train * Note: This job is consistently failing and is not promotion blocker as well. This jobs are me * Details: Execute tempest tests failing https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/job-output.txt https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/logs/undercloud/var/log/tempest/stestr_results.html.gz * Duscussion on IRC with chandan: ~~~ <bhagyashris|ruck> hey <bhagyashris|ruck> are you working on fs021 <bhagyashris|ruck> because this job is consistently failing periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train <chandankumar> bhagyashris|ruck fs021 is meant to fail <chandankumar> use ignore karo <bhagyashris|ruck> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train <chandankumar> yes <chandankumar> in every relase it's getting failed <bhagyashris|ruck> ok why <bhagyashris|ruck> ? <bhagyashris|ruck> https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/logs/undercloud/var/log/tempest/stestr_results.html.gz <bhagyashris|ruck> why? <chandankumar> bhagyashris|ruck here skipped test are running <bhagyashris|ruck> ok <bhagyashris|ruck> so should i safely ignore this ~~~ ::: :::spoiler ~~Pipeline: openstack-component-compute~~ ~~https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master&pipeline=openstack-component-compute~~ ~~* Job failures:~~ ~~* periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master~~ ~~https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master&pipeline=openstack-component-compute~~ ~~* Note: this is failing randomly~~ ~~* Details: Execute tempest tests failing because of insufficient ip adderss~~ ~~https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master/cdfe0e3/logs/undercloud/var/log/tempest/stestr_results.html.gz~~ ~~* Bug: https://bugs.launchpad.net/tripleo/+bug/1852770~~ ~~* Fix: https://review.opendev.org/#/c/722662/~~ ::: ### OSP * octavia jobs unstable * debugging * trying to get CI reports for OSP16/16.1 for RelDel ## May 8th ### TripleO :::spoiler ~~cirros image fixed tempest bug~~ ~~### tempest issues on stable branches CentOS-7 OVB jobs are RED fs001 https://bugs.launchpad.net/tripleo/+bug/1875731 https://bugs.launchpad.net/tripleo/+bug/1876972 TRAIN: GREEN except by Tempest fail in fs039 STEIN: Tempest fail ( arx is looking at it ) ROCKY: Tempest fail ( @arxcruz FYI) QUEENS: Tempest fail ( @arxcruz FYI) https://bugs.launchpad.net/tripleo/+bug/1876087 --> tempest bug FYI : @rfolco @arxcruz ~~ Added by bhagyashris: * Above bug is breaking "periodic-tripleo-ci-centos-7-ovb- 3ctlr_1comp_1supp-featureset039-train" job as well [1] * Reason: The reason it's not happening in train+:- os_tempest is used where cirros image is getting downloaded instead of using cached. In case of fs039, it's running validate-tempest instead of os-tempest, so hitting issue. So this is breaking Train as well. [1]: https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train/0d4f342/logs/undercloud/home/zuul/tempest/tempest.html.gz Reference : https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train ::: ::: spoiler ~~container build failures = mirror issue~~ ~~### container build https://bugs.launchpad.net/tripleo/+bug/1877416 >> mirror issue~~ ::: * Bhagyashris observations: * job1: periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master&pipeline=openstack-periodic-master This job is failing because of the tempest issue * Note: Here we are getting two different error logs as below This job is consistently failing on openstack-periodic-master pipeline * Error logs after 27th April: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/b26393e/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/b26393e/logs/undercloud/home/zuul/tempest/tempest.html.gz * Error logs before 27th April: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/7d32c39/logs/undercloud/home/zuul/tempest/tempest.html.gz This is issue may fixed by LP: https://bugs.launchpad.net/tripleo/+bug/1876087 :::spoiler ~~job2: periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master~~ * job2: periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master * Note: this job is recently added https://github.com/rdo-infra/review.rdoproject.org-config/commit/b83b902e6d913a02bae9337f2e6a2e95d9368cb3 and this is not in promotion criteria https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini#L33 overcloud deployment failed here: https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/7611a6d/job-output.txt Heat resource create failed (Nova compute heat resource) https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/7611a6d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz ~~* Reported ad LP bug: https://bugs.launchpad.net/tripleo/+bug/1877581~~ ~~### bring scenario10 online @TheG Please work the networking team to bring https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone online. (wes) removed (non-periodic) here https://review.opendev.org/#/c/726224/1/zuul.d/layout.yaml (rfolco) periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master: remove from promotion criteria if red~~ ### OSP See https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg?view ## May 7th ### Tripleo * master periodic container build failing: * NEW BUG: https://bugs.launchpad.net/tripleo/+bug/1877416 * rhel rdo tox job failing * https://bugs.launchpad.net/tripleo/+bug/1877299 * https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/overcloud-full.tar doesn't exist * https://bugs.launchpad.net/tripleo/+bug/1877230 * Here is the fix : https://review.opendev.org/#/c/725858/2 ### OSP * in previous doc: https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg?view ## May 6th ### Tripleo 1877031: queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version ### OSP ::: --- ## Completed On-Going :::spoiler ### undercloud containers cix https://trello.com/c/E7gL6d4b/1490-cixlp1878101tripleociproa-ping-br-ctlplane-is-failing-too-often-trying-to-ping-default-gateway lp https://bugs.launchpad.net/tripleo/+bug/1878101 logstash https://bit.ly/3bnXuwc testproject https://review.rdoproject.org/r/#/c/27453/ upstream/check https://review.opendev.org/#/c/727754/2 Fix: https://review.opendev.org/#/c/727942/ (Use /32 netmask for VIPs) ### INFRA Mirror issues NOTICE: Our CI mirrors in OVH BHS1 and GRA1 regions were offline between 12:55 and 14:35 UTC, any failures there due to unreachable mirrors can safely be rechecked * https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades&branch=stable%2Fqueens ::: --- ## History of bugs :::spoiler bugs Reported | Bugzilla | Name | status | Review | | -------- | ---- |------- | ------ | | [1873770](https://bugs.launchpad.net/tripleo/+bug/1873770) | OVB fs001 in centos8 master fails to push certificates contents to controllers | Incomplete | | | [1873892](https://bugs.launchpad.net/tripleo/+bug/1873892) | Non root login prevented on overcloud machines | Fixed Release | | |[1874019](https://bugs.launchpad.net/tripleo/+bug/1874019)| scenario009-multinode.yaml and openshift.yaml is missing | In Progress | | [1873770](https://bugs.launchpad.net/tripleo/+bug/1873770) | OVB fs001 in centos8 master fails to push certificates contents to controllers | Incomplete | | | [1875352](https://bugs.launchpad.net/tripleo/+bug/1875352)| keystone container failed to start in scenario000 | Triged | | | [1875871](https://bugs.launchpad.net/tripleo/+bug/1875871) | periodic rocky jobs failing with missing --name argument for pcs | Triged | | | [1875846](https://bugs.launchpad.net/tripleo/+bug/1875846) | Overcloud stack creation failed because of failed dependencies. | Closed | | | [1875833](https://bugs.launchpad.net/tripleo/+bug/1875833) | The WebSocket timed out before the Workflow completed in rocky/stain jobs | New | | | [1876087](https://bugs.launchpad.net/tripleo/+bug/1876087) | Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout | Triged | | | [1876096](https://bugs.launchpad.net/tripleo/+bug/1876096) | Queens: tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern tests failed | Triged | | | [1876672](https://bugs.launchpad.net/tripleo/+bug/1876672) | Python 2 - AttributeError: 'module' object has no attribute 'get_makefile_name' | Fixed Release | | | [1876893](https://bugs.launchpad.net/tripleo/+bug/1876893)| Error: error removing container - device or resource busy | In Progress | | | [1877031](https://bugs.launchpad.net/tripleo/+bug/1877031) | queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version | | ::: --- ## Handoff notes Notes from previous RR cycle

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.