Ruck and Rover notes #NovDec2020

# Ruck and Rover notes #NovDec2020 ###### tags: `ruck_rover` :::info Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ) **Ruck Rover - Unified Sprint #36 (NovDec 2020 timeframe)** Dates: Nov 11 - Jan 11 Tripleo CI team ruck|rover: marios, rlandy, wes OSP CI team ruck|rover: Previous notes: https://hackmd.io/a_PhAlijTr-KTWHzGuf3qQ ::: :::info RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0 Please try this out instead of cutting pasting bugs. Give feedback to wes please ::: [TOC] --- ## on-going issues :::danger ## TripleO ### gate ### periodic / 3rd party http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1 ## OSP ::: ### Reviews / Fixes ::: spoiler PATCHES ::: ### Bugs reported ::: spoiler PATCHES ::: --- :::info add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections. ::: ## Jan 25 - Jan 29 - Tempest volume tests filures - https://bugs.launchpad.net/tripleo/+bug/1913383 - Ara installation is failing - https://bugs.launchpad.net/tripleo/+bug/1913046 - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/772325 - Destination directory /home/zuul/workspace/logs does not exists - https://bugs.launchpad.net/tripleo/+bug/1913077 - https://review.opendev.org/c/openstack/tripleo-ci/+/772372 ## Jan 18 - Jan 22 - Mirros are down - https://bugs.launchpad.net/tripleo/+bug/1912177 - Tempest test [Victoria] FloatingIpTestCasesAdmin.test_two_vms_fips - https://bugs.launchpad.net/tripleo/+bug/1912171 - Python deps error in content provider - https://bugs.launchpad.net/tripleo/+bug/191247 - Cause: https://review.rdoproject.org/r/#/c/31573/ - ## Jan 7 - Jan 15 - Tempest tried to run on overcloud [1911696](https://bugs.launchpad.net/tripleo/+bug/1911696) - Revert OS Tempest - [770359](https://review.opendev.org/c/openstack/tripleo-quickstart/+/770359) - [770837](https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/770837) ## Jan 6th - FS037-updates stein job fails consistenly. https://bugs.launchpad.net/tripleo/+bug/1910399 Fix : https://review.opendev.org/c/openstack/tripleo-ci/+/769536 Testing : https://review.rdoproject.org/r/#/c/31487/ ## Jan 5th - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master (recent two failures) https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master test project passed - https://review.rdoproject.org/r/#/c/31424/ ## Dec 31st - https://bugs.launchpad.net/tripleo/+bug/1909750 **C7 train** Overcloud deployment is failing in C7 train with "stderr": "Get https://192.168.24.1:8787/v1/_ping: http: server gave HTTP response to HTTPS client" ## Dec 30th - **ussuri** https://bugs.launchpad.net/tripleo/+bug/1909658 periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri is failing during undercloud install with message - error checking if pulling from registry is blocked: unable to parse the registries configuration (/etc/containers/registries.conf): mixing sysregistry v1/v2 is not supported - **stein** Fixed https://bugs.launchpad.net/tripleo/+bug/1909671 Centos 7 based jobs are failing on TASK [Enable container-tools and disable rhel-modules] with error: /bin/sh: line 1: dnf: command not found ## Dec 29th - **Component pipeline** https://bugs.launchpad.net/tripleo/+bug/1909574 "periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-octavia-master/victoria/ussuri failing on tempest tests with SSHTimeout issue with error message "socket.timeout: timed out"" - **train promotion blocker** Fixed https://bugs.launchpad.net/tripleo/+bug/1909582 "Introspection failing in train branch with message: mistral.exceptions.InvalidActionException: Failed to find action [action_name=baremetal_introspection.get_status]\n', 'introspection_attempt': 0}" [Critical,Triaged] [High,Triaged] ## Dec 28th - **master:** - Fixed - https://bugs.launchpad.net/tripleo/+bug/1909445 Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n" - fixed by https://review.rdoproject.org/r/#/c/31439/ ## Dec 24th - **master:** - FIXED https://bugs.launchpad.net/tripleo/+bug/1909191 ( Error: Unable to find a match: cephadm failing on ovb-1ctlr_1comp-featureset002-master)- Patch is up https://review.opendev.org/c/openstack/tripleo-quickstart/+/768599 - **victoria:** --- ## Dec 23rd - **Gate:** - https://bugs.launchpad.net/tripleo/+bug/1909118 ( [train] b"TypeError: a bytes-like object is required, not 'MagicMock'" failing openstack-tox-py36 & openstack-tox-py37) - **master:** - https://bugs.launchpad.net/tripleo/+bug/1909105 ( ERROR openstack Stderr: 'level=debug msg="Pull Policy for pull [PullIfNewer]"\nerror building at STEP "RUN ln -s /usr/share/openstack-tripleo-common/healthcheck/swift-account failing on build-containers-ubi-8-push master ) - Notes: Because of change https://review.opendev.org/c/openstack/tripleo-common/+/747731 this issue started hitting periodic-tripleo-ci-build-containers-ubi-8-push job. So after reverting back the change https://review.opendev.org/c/openstack/tripleo-common/+/768264 it started passing the build-containers-ubi-8-push at https://review.rdoproject.org/r/#/c/31424/. - https://bugs.launchpad.net/tripleo/+bug/1908976 ( ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-<uuid> is failing on featureset001 and featureset035 master) - https://review.opendev.org/c/openstack/neutron/+/753847 (Avoid race condition when processing RowEvents) - https://review.rdoproject.org/r/31417 [DNM] Test bug/1908976 - Result: Still failing with same issue. - **victoria:** - **train:** - fs020 failed in the last run of periodic integration pipeline but it passed in testproject patch : https://review.rdoproject.org/r/#/c/31425/ hope this will help for promotion. --- ## Dec 22nd - **master:** - https://bugs.launchpad.net/tripleo/+bug/1908976 ( ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-<uuid> is failing on featureset001 and featureset035 master) - https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/768163 (DNM: Revert "Fix lower constraints") - This revert didn't work so Abandoned patch. - https://review.opendev.org/c/openstack/neutron/+/753847 checked with this fix as well but still facing same issue. - https://review.rdoproject.org/r/31417 [DNM] Test bug/1908976 - https://bugs.launchpad.net/tripleo/+bug/1907193 ( package podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 is filtered out by modular filtering) - failing featureset050-upgrades-master, multinode-oooq-container-updates-master and featureset037-updates-master - https://bugs.launchpad.net/tripleo/+bug/1909008 ( tempest.lib.exceptions.PreconditionFailed: Precondition Failed on standalone-full-tempest-api-master) - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768204 - **victoria:** - https://bugs.launchpad.net/tripleo/+bug/1908982 ( [victoria] tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.115 via SSH timed out. failing on standalone-full-tempest-scenario-victoria) - https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/768152 ## Dec 21st - **master:** - periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master and scenario000-multinode-oooq-container-updates-master is failing = https://bugs.launchpad.net/tripleo/+bug/1907193 - Error log: https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset037-updates-master/6d3f5b1/logs/undercloud/home/zuul/overcloud_update_run-Controller.log.txt.gz - Error ``` Depsolve Error occured: \n Problem: cannot install both podman-1.6.4-23.module_el8.3.0+566+4759265c.x86_64 and podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64\n - package podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64 requires podman = 2.0.5-5.module_el8.3.0+512+b3b58dca, but none of the providers can be installed ``` - ovb-1ctlr_1comp-featureset002-master is failing - https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/495856c/logs/undercloud/home/zuul/build-err.log.gz - featureset035-master is failing because of some tempest test failure - https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master/8f42da5/logs/undercloud/var/log/tempest/stestr_results.html.gz - master tempest failures .. start debugging from https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/43f38e4/logs/overcloud-controller-0/var/log/extra/errors.txt.txt.gz - **Victoria:** - standalone-full-tempest-scenario-victoria is failing = tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.105 via SSH timed out. - https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-victoria/2f22bf0/logs/undercloud/var/log/tempest/stestr_results.html.gz ## Dec 18th #### AMOL's notes - Logs: - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8f4/767679/1/check/tripleo-ci-centos-8-containers-multinode/8f43a97/job-output.txt - ssh key change - http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22REMOTE%20HOST%20IDENTIFICATION%20HAS%20CHANGED!%5C%22 - Job tripleo-ci-centos-7-standalone-upgrade-train - Do we need to keep it around? - https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-standalone-upgrade-train ## Dec 17th ## Dec 16th ## Dec 15th - Bugs - https://bugs.launchpad.net/tripleo/+bug/1908246 Victoria upgrade jobs aren't using content provider passed dlrn hash tag. [Patch](https://review.opendev.org/c/openstack/tripleo-quickstart/+/767112) - https://bugs.launchpad.net/tripleo/+bug/1908274 tox-py36-ci-config job is broken. ## Dec 14th - Bugs: - https://bugs.launchpad.net/tripleo/+bug/1908054 Ansible collect logs failing. - ### summary of status.. shitty week last week.. pypy depsolv, ubi-container in check, centos 8.2 - 8.3, content-provider patches all contributing to failures upstream atm. We're through most issues atm. OVB in check is full red, but as I said looks to be coming back to green via periodic promotions. * pypy - lower-constraints https://review.opendev.org/q/topic:%2522tripleo-lower%2522+(status:open+OR+status:merged) * ubi-8 - container-builds e.g. https://review.opendev.org/c/openstack/tripleo-ci/+/766839 * c8-train -> ussuri upgrade jobs failing on pacemaker mismatch - requires promotion to fix or method to inject change to content-providers. * content-provider ip failures https://bugs.launchpad.net/tripleo/+bug/1907657 * overcloud-full-hardened moved to nv until we promote all c8 branches https://bugs.launchpad.net/tripleo/+bug/1907457 * master appears to be back and healthy enough for a promotion. fs01/035 starting to pass again. wes to make sure master is promoted soon. * c8-train is the most import and a problem. container-builds are failing and tripleo-common needs patches to help us diagnose. See https://review.rdoproject.org/r/#/c/31336/ * need to land to fix ovb 3rd party https://review.opendev.org/c/openstack/paunch/+/766779 https://bugs.launchpad.net/tripleo/+bug/1907833 ## Dec 10th Need help landing https://review.opendev.org/c/openstack/tripleo-common/+/766516 Please review: https://review.opendev.org/c/openstack/tripleo-ci/+/766621 ## Dec 8th Please focus on unblocking: https://review.opendev.org/#/q/owner:%22amolkahat+%253Camolkahat%2540gmail.com%253E%22+branch:stable/ussuri gate - release issue https://bugs.launchpad.net/tripleo/+bug/1907122 https://review.opendev.org/q/topic:%22release-aware-gate%22+(status:open%20OR%20status:merged) Not sure if https://bugs.launchpad.net/tripleo/+bug/1907006 is related to the above issue or not ## Dec 7th new bugs in https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0 --- ## Dec 4th & earlier ongoing things #### https://bugs.launchpad.net/tripleo/+bug/1906769 periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master failing intermitently with paramiko.ssh_exception.AuthenticationException: Authentication failed #### https://bugs.launchpad.net/tripleo/+bug/1883843 tripleo-ci-centos-8-scenario007-multinode-oooq-container is having intermittently timeouts #### https://bugs.launchpad.net/tripleo/+bug/1906486 FS01 ovb job fails with failed to stat() /home/zuul/overcloud-full.raw #### https://bugs.launchpad.net/tripleo/+bug/1905386 Centos 7 train jobs failing with pymod2pkg requiring python 3.6 #### https://bugs.launchpad.net/tripleo/+bug/1903508 periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler * just waiting on train cherrypick https://review.opendev.org/c/openstack/tripleo-common/+/763721 ::: #### https://bugs.launchpad.net/tripleo/+bug/1903882 tripleo-ci-centos-8-content-provider failed on TASK [container-build : Pull failed containers from RDO registry] with error was: 'dict object' has no attribute 'split' :::spoiler * waiting on fix to merge https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/762400/ ::: --- # stop here eveything still open is duplicated above this line --- ## ~~Dec 3rd~~ * inconsistent timeout issue for updates job : (https://bugs.launchpad.net/tripleo/+bug/1883843) https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates&branch=stable%2Ftrain ## Dec 2nd * Scenario 01/04 : Fix was merged (https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764140/) * New issue reported consistent failure : https://bugs.launchpad.net/tripleo/+bug/1906486 ## Dec 1st * Random timed_out on this patch : https://review.opendev.org/c/openstack/tripleo-heat-templates/+/764140 ## Nov 30th * wes - to check gate failures.. https://zuul.openstack.org/status * FAK, all upstream jobs failing https://bugs.launchpad.net/tripleo/+bug/1906265, patch up ## Nov 27th - https://bugs.launchpad.net/tripleo/+bug/1905942 Ussuri jobs failing to lookup hiera - 13 gate failures UTC 20:00 - bug spreadsheet updated ## Nov 26th - Nothing much, a lot of stuff is broken right now - 90+ gate failures ## Nov 25th - https://bugs.launchpad.net/tripleo/+bug/1905386 - potential fix: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232 - https://bugs.launchpad.net/tripleo/+bug/1905536 = Ceph wrong role name - https://bugs.launchpad.net/tripleo/+bug/1905592 = quay.io issue - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764217 - https://review.opendev.org/c/openstack/tripleo-ci/+/764218 ## Nov 24th - https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/764055 Fix stackviz for failed tempest runs [WIP] [NEW] - https://bugs.launchpad.net/tripleo/+bug/1905460 - periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master - random failures: TestVolumeBootPattern, NetworkSecGroupTest, TestShelveInstance (dstat looks good, no indication of resource consumption problems) - testing here https://review.rdoproject.org/r/31193 - https://bugs.launchpad.net/tripleo/+bug/1905348 has a fix https://review.opendev.org/c/openstack/python-tripleoclient/+/763919 - https://bugs.launchpad.net/tripleo/+bug/1905421 - Tempest error - https://bugs.launchpad.net/tripleo/+bug/1905418 - Tempest error - Error in train jobs due python 2.7 and pymod2pkg - https://bugs.launchpad.net/tripleo/+bug/1905386 - attempt to bump venv to py3: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/764232 Switch to python3 venv by default - ~~introspection bug 1904936 fix~~ - ~~https://review.opendev.org/c/openstack/tripleo-ansible/+/763972~~ ## Nov 23th - ussuri gate jobs still failing - ussuri repo-setup - introspection bug 1904936 is still a thing - certificate tempest failure on victoria - 1903508 (waiting to merge) - cold migration bug 1903033 - waiting to merge - centos-8-scenario002-standalone ssh time out bug 1904223 (unresolved) - affects victoria promotion? - ~~ensure private network exists bug 1902048 >> FIXED~~ - ~~fs037-stein >> FIXED~~ - ~~bug 1905036 (scen10 ovn) >> FIXED~~ - ~~pulling containers from docker.io (stein) >> FIXED~~ (details below...) ## Nov 20th #### ~~new/transient/not filed yet~~ :::spoiler ~~periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently: >> fixed fs037-stein is green: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein~~ * https://logserver.rdoproject.org/25/25325/67/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/d358af8/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz * Should have been fixed by: https://review.opendev.org/#/c/762871/ * marios: added https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18 i think we need stein cherrypick @ https://review.opendev.org/#/c/763546/ & testing with rlandy testproject review https://review.rdoproject.org/r/#/c/25325/ ::: :::spoiler ussuri gate is blocked on our content provider stuff merging STILL! * https://review.opendev.org/#/c/757544/ * now blocked on https://bugs.launchpad.net/tripleo/+bug/1905034 * 16:29 <@mwhahaha> marios|ruck, weshay|ruck: sooo what are we doing about the fact that ussuri CI is completely blocked on tripleo-common and THT? ::: :::spoiler ~~scen10-ovn~~ https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect 15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/ ::: :::spoiler ~~pulling containers from docker.io~~ https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io * bug is fixed but the fix needs to go back to stein https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18 * waiting on stein cherrypick https://review.opendev.org/#/c/763546/ ::: :::spoiler ussuri repo-setup https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0 waiting to merge: https://review.opendev.org/c/openstack/tripleo-quickstart/+/763633 ::: #### https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect :::spoiler info 15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io :::spoiler info * bug is fixed but the fix needs to go back to stein https://bugs.launchpad.net/tripleo/+bug/1903581/comments/18 * waiting on stein cherrypick https://review.opendev.org/#/c/763546/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0 --- ## Nov 19th #### https://bugs.launchpad.net/tripleo/+bug/1904936 openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node #### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler info * ~~https://review.opendev.org/#/c/762497/~~ * https://review.opendev.org/#/c/763185/ - victoria cherry-pick * blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs) * victoria promotion blocker ::: #### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n' :::spoiler info * currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini ::: #### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out #### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists] :::spoiler info * temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec * waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.' #### new/transient/not filed yet :::spoiler chkumar working on rdo->vexx remaining jobs 09:32 < chandankumar> marios: sshnaidm I am tracking rdocloud nodeset removal here https://etherpad.opendev.org/p/remove-rdocloud-jobs ::: :::spoiler still chasing a queens promotion * latest buildset https://review.rdoproject.org/zuul/buildset/14a3df4f2f4544a3b6928e1326f0b19b * rlandy posted yesterday https://review.rdoproject.org/r/#/c/25325/ - some green from that * still missing fs18/20 http://38.102.83.109/promoter_logs/centos7_queens.log 2020-11-19 09:10:53,087 22394 INFO promoter Skipping promotion of centos7-queens from tripleo-ci-testing to current-tripleo, missing successful jobs: [u'periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens', u'periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset018-queens'] * posted https://review.rdoproject.org/r/31145 for missing ^^^ let's see * it is a good hash - from 18th https://trunk.rdoproject.org/centos7-queens/83/37/83378d81927e5f1e0c1b7e85cf7c4258c4e32356_7268d992/ ::: :::spoiler Merged Update rdo-openvswitch in ussuri to support ovs/ovn2.13 NFV SIG https://review.rdoproject.org/r/30774 12:07 < ykarel> marios|ruck, fyi ^ in case u see something wron gin ussuri jobs, let us know 12:07 < ykarel> we tested multiple jobs and it went good, but just in case if there is some issue ::: :::spoiler https://review.rdoproject.org/r/#/c/30776/ - network component still very red ::: :::spoiler periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently: https://logserver.rdoproject.org/25/25325/67/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/d358af8/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz Should have been fixed by: https://review.opendev.org/#/c/762871/ ::: --- ## ~~Nov 18th~~ :::info RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0 Please try this out instead of cutting pasting bugs. Give feedback to wes please ::: #### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler info * ~~https://review.opendev.org/#/c/762497/~~ * https://review.opendev.org/#/c/763185/ - victoria cherry-pick * blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs) * victoria promotion blocker ::: #### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n' :::spoiler info * currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini ::: #### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out #### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists] :::spoiler info * temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec * waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.' #### ~~https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE~~ :::spoiler info * Moved fs19 and fs18 to a single playbook. May be fixed ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source~~ :::spoiler info blocks victoria gate fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112 ::: #### new/transient/not filed yet :::spoiler ussuri *just* missed promotion * ussuri good buildset https://review.rdoproject.org/zuul/buildset/87a37f60460c4ffd9e2aee3f49256efb * missing periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri so * posted https://review.rdoproject.org/r/31132 * job is generally green https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri ::: :::spoiler Vexx mirrors outdated * http://mirror.regionone.vexxhost-nodepool-tripleo.rdoproject.org:8080/rdo/centos7-stein/tripleo-ci-testing/ * baseurl=https://trunk.rdoproject.org/centos7-stein/d7/87/d787d250cd2a93bf6368cd433bcb2c8b306c8edd_8d9f1861 (old hash) causes failures when pulling containers * https://review.opendev.org/#/c/763233/ * https://review.opendev.org/763401 * Wes requests python script to pull from trunk and retry ::: --- ## ~~Nov 17th~~ #### https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source :::spoiler info blocks victoria gate fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112 ::: #### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler info * ~~https://review.opendev.org/#/c/762497/~~ * https://review.opendev.org/#/c/763185/ - victoria cherry-pick * blocked on https://review.opendev.org/#/c/763005 (standalone/undercloud upgrade jobs) * victoria promotion blocker ::: #### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n' :::spoiler info * currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini ::: #### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out #### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists] :::spoiler info * temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec * waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE :::spoiler info * Moved fs19 and fs18 to a single playbook. May be fixed ::: #### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.' #### ~~https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]~~ :::spoiler info reopened bug ~~fix at https://review.rdoproject.org/r/#/c/31114/~~ tested at https://review.rdoproject.org/r/#/c/25325/ ~~@marios pls confirm that the upgrades/updates jobs are ok with the one playbook for multinode.~~ ::: #### new/transient/not filed yet :::spoiler ~~undercloud-upgrade & standalone-upgrade victoria problem~~ filed https://bugs.launchpad.net/tripleo/+bug/1904565 * ~~https://7085ed736f9157d6d701-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf1.rackcdn.com/762658/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/ad84735/logs/undercloud/home/zuul/undercloud_upgrade.log tripleo_common.image.exception.ImageNotFoundException: Not found image: http://23.253.57.32:5001/v2/tripleovictoria/openstack-keystone/manifests/7f974e10d7184d5fc45445a3073333bd"], "stdout": "", "stdout_lines": []}~~ ::: :::spoiler vexx infra/introspection issues https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/835c3e6/logs/undercloud/var/log/extra/baremetal_list.txt.gz https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/b99e7f1/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/2726291/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/970a3ea/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/72f523b/job-output.txt ::: #### fs050 victoria jobs are failing: :::spoiler Related fixes: https://review.opendev.org/#/c/763097/ Make victoria upgrades job non-voting until fixed ~~https://review.rdoproject.org/r/31120 Add undercloud fs050 upgrades to master/v/u periodic ~~ @marios : your thoughts here? where though? are you referring to upstream undercloud upgrade job fixed by https://review.opendev.org/#/c/763005/ ? or periodic job? can't be periodic since you're adding it with ~~https://review.rdoproject.org/r/31120~~ ::: --- ## ~~Nov 16th~~ #### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler info * https://review.opendev.org/#/c/762497/ * victoria promotion blocker ::: #### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n' :::spoiler info * currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini ::: #### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out #### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists] :::spoiler info * temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec * waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE #### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.' #### ~~https://bugs.launchpad.net/tripleo/+bug/1904418 ( IN PROGRESS) buildimage centos7 jobs are failing to fetch mirror info 404~~ :::spoiler info * should be fixed, watch https://static.opendev.org/mirror/logs/rsync-mirrors/ ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1904483 (Require attention)centos-7 overcloud image build jobs are failing on pre-install.d/03-centos-rt - rt repo no longer exists~~ :::spoiler info shoudl be fixed by https://review.opendev.org/#/c/762935 (will need cherry-pick to train, stein, rocky) Related fix on removing rt-repo: https://review.opendev.org/#/c/762926/ ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'~~ :::spoiler info https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498) ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests.~~ :::spoiler info * job is just taking too long... removing tempest after deployment https://review.opendev.org/#/c/762674/ ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762871/~~ --- ## ~~Nov 13th~~ #### https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch' :::spoiler info https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498) ::: #### https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file." :::spoiler info * https://review.opendev.org/#/c/762497/ * victoria promotion blocker ::: #### https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n' :::spoiler info * currently waived in master criteria http://38.145.34.55/config/CentOS-8/master.ini ::: #### https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out #### https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists] :::spoiler info * temp workaround merged at https://review.rdoproject.org/r/#/c/30776/1/openstack-neutron.spec * waiting for actual fix blocked in gate https://review.opendev.org/#/c/759673/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE #### https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests. :::spoiler info * job is just taking too long... removing tempest after deployment https://review.opendev.org/#/c/762674/ ::: #### https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.' #### https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762547/ Additional fix: https://review.opendev.org/#/c/762871/ #### ~~https://bugs.launchpad.net/tripleo/+bug/1903498 tripleo-ci-centos-8-undercloud-upgrade-ussuri is failing RuntimeError: Update extra packages failed: b'sudo: yum: command not found\n~~ :::spoiler info * waiting for merge https://review.opendev.org/#/c/761892/ ::: #### ~~https://bugs.launchpad.net/tripleo/+bug/1904214 centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7 * centos7 promotion blocker~~ #### ~~https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]~~ . :::spoiler ~~promotion status~~ #### victoria promotion: Last promotion 10th Nov Last buildset - https://review.rdoproject.org/zuul/buildset/f8e8812c49c14d2c8915af307d85731c * victoria promotions blocked on https://bugs.launchpad.net/tripleo/+bug/1903508/comments/6 * periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria(In criteria) & periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-victoria, Bug: https://bugs.launchpad.net/tripleo/+bug/1903508 Cixed to security team, Provided reproducer to martin today, Patch is up - https://review.opendev.org/#/c/762497 * ~~Master/Ussuri/Victoria promotion blocker https://bugs.launchpad.net/tripleo/+bug/1900949 Ovb jobs are failing on master branch, imported nodes are not transitioning to manageable state with ‘Error: Unable to establish IPMI v2 / RMCP+ session\n’~~ * rdocloud infra issue, we moved jobs to vexx host #### ussuri promotion: Last promotion 11/13 #### train promotion: * new/transient/not filed yet: * ~~master scenario2 standalone failures https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario002-standalone&branch=master * blocks https://review.opendev.org/#/c/762497/4~~ * filed @ https://bugs.launchpad.net/tripleo/+bug/1904223 * c8 should promote again today https://review.rdoproject.org/zuul/buildset/1f410357edb34516ab3ce4cadffa491f * c7 is in trouble: (rlandy investigating) https://bugs.launchpad.net/tripleo/+bug/1904214 - centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7 <amoralej> yes, 7.9.20009 was pushed yesterday to mirrors <rlandy|rover> that's probably it <amoralej> let me dig into it <jpena> amoralej: that server mounts opendev's AFS, just like the rest of the mirrors we have in opendev <jpena> looking at https://grafana.opendev.org/d/HxrNXt2Gk/afs?orgId=1, it looks like the mirror was synced 1 hour ago <jpena> we might check where does it sync from, maybe that one seed does not have 7.9 yet <jpena> so, what I see in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L51 is that the opendev AFS mirrors sync from mirror.lstn.net/centos <jpena> looking at http://mirror.lstn.net/centos/7/os/x86_64/Packages/?C=M;O=D, it looks like they don't have the 7.9 content yet Blocking Sagi's patch will solve the issue https://review.opendev.org/#/c/761892/ for https://bugs.launchpad.net/tripleo/+bug/1903498 ``` 2020-11-13 12:22:22.839674 | primary | Error: Package: glibc-2.17-307.el7.1.i686 (base) 2020-11-13 12:22:22.839814 | primary | Requires: glibc-common = 2.17-307.el7.1 2020-11-13 12:22:22.839848 | primary | Installed: glibc-common-2.17-317.el7.x86_64 (@base) 2020-11-13 12:22:22.839870 | primary | glibc-common = 2.17-317.el7 2020-11-13 12:22:22.839898 | primary | Available: glibc-common-2.17-307.el7.1.x86_64 (base) 2020-11-13 12:22:22.839920 | primary | glibc-common = 2.17-307.el7.1 2020-11-13 12:22:22.839943 | primary | Error: Package: python3-devel-3.6.8-13.el7.x86_64 (base) 2020-11-13 12:22:22.839989 | primary | Requires: python3-libs(x86-64) = 3.6.8-13.el7 2020-11-13 12:22:22.840015 | primary | Installed: python3-libs-3.6.8-17.el7.x86_64 (@base) 2020-11-13 12:22:22.840037 | primary | python3-libs(x86-64) = 3.6.8-17.el7 2020-11-13 12:22:22.840062 | primary | Available: python3-libs-3.6.8-13.el7.x86_64 (base) 2020-11-13 12:22:22.840083 | primary | python3-libs(x86-64) = 3.6.8-13.el7 2020-11-13 12:22:22.840125 | primary | You could try using --skip-broken to work around the problem 2020-11-13 12:22:23.195322 | primary | You could try running: rpm -Va --nofiles --nodigest ``` #### rocky promotion: * rekicked on vexx https://review.rdoproject.org/r/#/c/29969/16/.zuul.yaml * blocked on https://bugs.launchpad.net/tripleo/+bug/1904214 #### stein promotion: * promoted yesterday - leaving for weekend run #### queens promotion: * testproject there (rdo) https://review.rdoproject.org/r/30906 per https://bugs.launchpad.net/tripleo/+bug/1903996/comments/1 but failing so try vexx * testproject there (vexx) https://review.rdoproject.org/r/#/c/25325/ * blocked on https://bugs.launchpad.net/tripleo/+bug/1904214 ::: --- ## ~~Nov 12th~~ ### TripleO #### Headsup <amoralej> we are updating ovs/ovn 2.13 for master and victoria today. We've gated it with different jobs so i expect to go smooth but let us know if you find any issue * [Blocking check jobs]tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch' https://bugs.launchpad.net/tripleo/+bug/1903980 * Sagi patch will solve the issue https://review.opendev.org/#/c/761892/ * [Promotion blocker for c7 train/stein/rocky]Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role] https://bugs.launchpad.net/tripleo/+bug/1903961 * Hit on rerun via testproject too - Need to investigate * "Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests." https://bugs.launchpad.net/tripleo/+bug/1903993 * Need to investigate * Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE https://bugs.launchpad.net/tripleo/+bug/1903996 * Need to investigate * testproject running - https://review.rdoproject.org/r/#/c/25325/ * Ran rocky /stein promotion ... https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-stable4-5 stein should promote. rocky failed container builds again. need to look at increasing the timeout there

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.