Ruck and rover notes #27

tags: `ruck_rover`

Important links for ruck rover's ruck/rover links to help
Ruck Rover - Unified Sprint 27
Dates: May 6 - May 26

Tripleo CI team ruck|rover: Folco (rfolco) / Bhagyashris (bhagyashris)
OSP CI team ruck|rover (April 24 - May 15): Waldek (wznoinsk) / Avi (TalmoR)

Previous notes: https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg

Ruck and rover notes #27
- on-going issues
  - Launchpad Bugs Reported
- May 28th
  - Tripleo
  - OSP
- May 27th
  - Tripleo
  - OSP
- May 26th
  - Tripleo
  - OSP
- May 25th
- May 22th
  - TripleO
  - OSP
- May 21th
  - Tripleo
  - OSP
- May 20th
  - Tripleo
  - OSP
- May 19th
  - TripleO
  - OSP
- May 18th
  - Tripleo
  - OSP
- May 15th
  - TripleO
  - OSP
- May 14th
  - TripleO
  - OSP
- May 13th
  - TripleO
  - OSP
- May 12th
  - Tripleo
  - OSP
- May 11th
  - Tripleo
  - OSP
- May 8th
  - TripleO
- Completed On-Going
- History of bugs
- Handoff notes

on-going issues

TripleO

https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images) here is the fix:
https://review.opendev.org/731498 (Added image sanity check condition)

gate

RDO CI

watch next periodic runs - https://review.opendev.org/#/c/730533 merged

OSP

OSP17 still without attention (!) because of fires in OSP<16
foreign jobs still invading p1/p2 views
- not yet tested https://code.engineering.redhat.com/gerrit/#/c/198375

add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections.

Launchpad Bugs Reported

Bugzilla	Name	status	Review
1878101	ping br-ctlplane is failing too often, "Trying to ping default gateway"	Complete	727942
1878190	periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master job is consistently failing because of some tesmpest test are failing	Triged	727192
1878197	periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train is failing and time_out on tempest execute	Triged
1878150	tox-linters jobs failing with AttributeError	In Progress	727113
1878248	NetworkSecGroupTest failing on fs020 stein/train	In Progress	727287
1877031	queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version	In Progress	727696
1879267	Error: Failed to download metadata for repo 'advanced-virtualization'\n level=debug msg="error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \"centos-binary-swift-container-working-container\": error while running runtime: exit status 1 failing standlone deployment jobs	Complete	728761
1879638	[Train Only] Error: No matching repo to modify: epel failing container build push on centos-8 train	Complete	729519
1880383	ERROR! Unable to retrieve file contents, Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller. failing tripleo quickstart deployment on centos-8 (master, ussuri and train)	Complete	730533

May 28th

Tripleo

Train and Stein get promoted recently.
Master promotion is blocked because of fs001 failure.
RDO CI Failures:
- https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images) https://review.opendev.org/731498 (Added image sanity check condition)
- Master : fs001 is failing on master and blocking promotion.
  Issue : https://bugs.launchpad.net/tripleo/+bug/1879766 (master ovb jobs failing on Destination directory /etc/pki/tls/private does not exist)
- Ussuri: "periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri" is failing because of tempest test failures : https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri/903029a/logs/undercloud/var/log/tempest/stestr_results.html.gz
  "periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-ussuri" is getting time out consistently at Execute tempest test task. Submitted patch to increase the time out here : https://review.rdoproject.org/r/#/c/27811/ and testing the jobs: https://review.rdoproject.org/r/#/c/27789/
- Train C7: Currently most of the jobs are failing because of https://bugs.launchpad.net/tripleo/+bug/1881090 (virt-customize: error: libguestfs error: overcloud-full.qcow2: No such file failing ooci-build-images), once this will patche will merge https://review.opendev.org/731498 (Added image sanity check condition) it will go green.

OSP

May 27th

Tripleo

Gate:
- scenario10 on stable/ussuri check pipeline: (https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-standalone&pipeline=check#) error log: The specified regex doesn't match with anything (https://88e1de9a81e55d590d5b-26f184bb59af339cfe698349cbda4177.ssl.cf5.rackcdn.com/731036/1/check/tripleo-ci-centos-8-scenario010-standalone/e355cdc/logs/undercloud/var/log/tempest/tempest_run.log) this is failing because these two files are not in sync https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_master.yml and https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/validate-tempest/vars/tempest_skip_ussuri.yml
  ~~Here is the fix: https://review.opendev.org/#/c/731149/ (Remove octavia tempest tests from skiplist)~~

RDO CI Failures:

ussuri failing on scen010, fs001 and fs020
- scen010 –> https://bugs.launchpad.net/tripleo/+bug/1880981
- fs001/fs020 tempest tests

Stein C7: fs020 and fs035 is failing and blocking the promotion because of below issue and same jobs are passing on test patch https://review.rdoproject.org/r/#/c/27709/ . So as this is know and random failure.

2020-05-27 07:21:18 | [WARNING]: Unhandled error in Python interpreter discovery for host overcloud-
2020-05-27 07:21:18 | controller-0: Failed to connect to the host via ssh: Warning: Permanently added
2020-05-27 07:21:18 | '192.168.24.25' (ECDSA) to the list of known hosts.  Permission denied
2020-05-27 07:21:18 | (publickey,gssapi-keyex,gssapi-with-mic).
2020-05-27 07:21:18 | 
2020-05-27 07:21:18 | 
2020-05-27 07:21:18 | TASK [Gathering Facts] *********************************************************
2020-05-27 07:21:18 | fatal: [overcloud-novacompute-0]: UNREACHABLE! => {
2020-05-27 07:21:18 |     "changed": false, 
2020-05-27 07:21:18 |     "unreachable": true
2020-05-27 07:21:18 | }
2020-05-27 07:21:18 | 
2020-05-27 07:21:18 | MSG:
2020-05-27 07:21:18 | 
2020-05-27 07:21:18 | Data could not be sent to remote host "192.168.24.15". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.15' (ECDSA) to the list of known hosts.
2020-05-27 07:21:18 | Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

OSP

May 26th

Tripleo

Gate:
- On cockpit Most of the failures are from patch: https://review.opendev.org/#/c/725623/ (Add ansible hieradata file) and all the failures are random one those are listed yesterday.
RDO CI failures:
- Stein c7 : featureset020 is failing randomly , recently it failed because of below error, Below error is known and random one so will need to keep eye on next run
  Test patch: https://review.rdoproject.org/r/#/c/27709/ got SUCCESS on test patch.
```
2020-05-26 07:14:15 | TASK [Gathering Facts]     *********************************************************
2020-05-26 07:14:15 | fatal: [overcloud-novacompute-1]: UNREACHABLE! => {
2020-05-26 07:14:15 |     "changed": false, 
2020-05-26 07:14:15 |     "unreachable": true
2020-05-26 07:14:15 | }
2020-05-26 07:14:15 | 
2020-05-26 07:14:15 | MSG:
2020-05-26 07:14:15 | 
2020-05-26 07:14:15 | Data could not be sent to remote host "192.168.24.21". Make sure this host can be reached over ssh: Warning: Permanently added '192.168.24.21' (ECDSA) to the list of known hosts.
2020-05-26 07:14:15 | Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
2020-05-26 07:14:15 | 
2020-05-26 07:14:15 | fatal: [overcloud-novacompute-0]: UNREACHABLE! => {
2020-05-26 07:14:15 |     "changed": false, 
2020-05-26 07:14:15 |     "unreachable": true
2020-05-26 07:14:15 | }
```
- Train c7: featureset020 failed recenlty after long time because of tempest test failure
  https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train/2629c18/logs/undercloud/var/log/tempest/stestr_results.html.gz
  will need to keep eye on this in next run test patch: https://review.rdoproject.org/r/#/c/27657/ got SUCCESS on test patch.
- Train C8: standalone-full-tempest-api job got failed after this patch Revert "[install-deps] Use venv instead of virtualenv when possible" got merged because of tempest test failure https://logserver.rdoproject.org/53/27753/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/7ef9d7b/logs/undercloud/var/log/tempest/stestr_results.html.gz
  Test patch here : https://review.rdoproject.org/r/#/c/27753/ -> Same tempest test is failing on the patch : https://logserver.rdoproject.org/53/27753/1/check/periodic-tripleo-ci-centos-8-standalone-full-tempest-api-train/7ef9d7b/logs/undercloud/var/log/tempest/stestr_results.html.gz . Will check for the next run result on test patch. Got SUCCESS on next run. Hope will pass on openstack-periodic-24hr pipeline as well.
- Ussuri: fs001 is failing randomly with https://bugs.launchpad.net/tripleo/+bug/1879766 (ovb jobs failing on Destination directory /etc/pki/tls/private does not exist) https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-ussuri/8f8eeda/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz Here doing regression testing : https://review.rdoproject.org/r/#/c/27750/
- Master: periodic-tripleo-centos-8-buildimage-overcloud-full-master is failing on current running openstack-peridic-master pipeline but it passed after that on testproject patch https://review.rdoproject.org/r/#/c/27754/SUCCESS. Test patch got SUCCESS in regression testing as well. Hope this will pass on next run on openstack-periodic-master pipeline.

OSP

May 25th

Tripleo

gate

13 failures 05-25-2020 3:50 UTC

no pattern, random failures, examples:
- tempest.scenario.test_snapshot_pattern.TestSnapshotPattern failed on scen001 standalone (centos8)
- Unable to disable service iscsid.socket on containers multinode (stein)
- image prepare failed on containers multinode (train)
- TIME OUT on undercloud-containers, containers-multinode, scen000 (train)
- container-puppet tasks failed on centos7 standalone (train)
- Wait for containers to start for step 3 using paunch >> scen003 standalone (train)
- puppet host configuration failed on scen010 (train)
patches rechecked

RDO CI Failures

https://bugs.launchpad.net/tripleo/+bug/1880383 (ERROR! Unable to retrieve file contents, Could not find or access '/home/zuul/workspace/.quickstart/config/release/queens.yml' on the Ansible Controller. failing tripleo quickstart deployment on centos-8 (master, ussuri and train))
~~Fix: https://review.opendev.org/#/c/730533/ - Revert "[install-deps] Use venv instead of virtualenv when possible"~~
Test patch is here : https://review.rdoproject.org/r/#/c/27734/ -> SUCCESSS

undercloud-upgrade failing in gate

~~https://bugs.launchpad.net/tripleo/+bug/1876893~~

~~https://review.opendev.org/#/c/726008/2/tripleo_ansible/roles/tripleo-container-rm/tasks/tripleo_podman_container_rm.yml~~
~~https://review.opendev.org/#/c/725939/~~
- ~~Above patch is Abandoned in favor of https://review.opendev.org/#/c/726008/2 and this patch is merged~~
~~https://review.opendev.org/#/c/725944/ this patch is also merged~~

OSP

May 22th

TripleO

Gate failures:
- python-tripleoclient
  - https://bugs.launchpad.net/tripleo/+bug/1879392 [build-test-packages] ERROR:dlrn:Received exception Error in build_rpm_wrapper for python-tripleoclient
- tripleo-ci-centos-8-scenario001-standalone: this job is mostly failing becuase of tempest test failure and standalone deployment failure
  - Tempest failure: (https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5ec/725250/6/gate/tripleo-ci-centos-8-scenario001-standalone/5ec570c/logs/undercloud/var/log/tempest/stestr_results.html)
  - Standalone deployment failure: https://6e5de493453a596bf54b-e12a251c5f6363a4d35eb8aac39c4442.ssl.cf2.rackcdn.com/729508/1/gate/tripleo-ci-centos-8-scenario001-standalone/270ea66/logs/undercloud/home/zuul/standalone_deploy.log
  - timeoute increase: https://review.opendev.org/730416 Increase scenario001 standalone timeout
RDO CI failures:

train promotion failed due to:

periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train
- https://bugs.launchpad.net/tripleo/+bug/1880236
- @rfolco fyi, This job is not a promotion blocker https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/train.ini#L32
tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train
- https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train#
- TIME OUTs: https://review.rdoproject.org/r/27724 Increase timeout for fs035 train
- @rfolco fyi, If you see, it's never time out here https://review.rdoproject.org/r/#/c/27644/ (test patch on testproject), even in openstack-periodic-24hr pipeline it got SUCCESS https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train

master:

periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master: is failing consistently because of https://bugs.launchpad.net/tripleo/+bug/1879766 (master ovb jobs failing on Destination directory /etc/pki/tls/private does not exist): Not sure why this bug is marked as invalid.
periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein: is getting time out because of some tempest test are running out of time.

OSP

May 21th

Tripleo

Gate failures:
- tripleo-ci-centos-7-scenario010-standalone is failing on gate/check https://zuul.opendev.org/t/openstack/builds?pipeline=gate&job_name=tripleo-ci-centos-7-scenario010-standalone
  https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-7-scenario010-standalone because of setUpClass(octavia_tempest_plugin.tests.scenario.v2.test_load_balancer) tempest test failure https://55e769a6be92d5846621-6d67d1055c684f7187f360a048e3d766.ssl.cf5.rackcdn.com/721577/1/gate/tripleo-ci-centos-7-scenario010-standalone/5ead591/logs/undercloud/var/log/tempest/stestr_results.html
  - Bug: https://bugs.launchpad.net/tripleo/+bug/1879941 (
    tempest.lib.exceptions.NotFound: Object not found causes failure on stable/train tripleo-ci-centos-7-scenario010-standalone)
  - Fix: https://review.opendev.org/#/c/729926/ (Use tempest_tempestconf_profile_overrides for extra overrides)
  - Test patch: https://review.opendev.org/#/c/729930/ ([DNM] sc10 train gate)
RDO CI failures
- periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train job is getting timeout https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-24hr&job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train but on testproject it's working fine https://review.rdoproject.org/r/#/c/27644/

OSP

May 20th

Tripleo

rfolco notes on gate failures

msg: '[''mysql_init_bundle''] failed to start

https://47fba6cbbdf7110dae32-95a2136c445cc9f6f88a9d6c838f9ef5.ssl.cf2.rackcdn.com/728390/5/gate/tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates/fcba19d/logs/undercloud/home/zuul/overcloud_update_run_Controller.log

RUN END RESULT_TIMED_OUT

master
- fs001 https://bugs.launchpad.net/tripleo/+bug/1879766

Container build push is failing on c8 train

~~Reported Bug: https://bugs.launchpad.net/tripleo/+bug/1879638 ([Train Only] Error: No matching repo to modify: epel failing container build push on centos-8 train)~~
~~Fix: https://review.opendev.org/#/c/729519/ ([Train only] Fix for CentOS8 containers build)~~

Now build containers are failing here: https://logserver.rdoproject.org/34/27634/4/check/periodic-tripleo-centos-8-train-containers-build-push/b8bf039/logs/buildah-builds/kolla-builds/aed6abec-8abe-4cbd-883d-a26ed33d44a6/docker/cinder/cinder-base/cinder-base-build.log as the ceph-common is getting disabled because of this change https://review.opendev.org/#/c/721329/13/docker/base/ceph.repo

STEP 1: FROM             trunk.registry.rdoproject.org/tripleotraincentos8/centos-binary-openstack-base:b74e95c27fff46a3abf661fd6348b604cabf47db_34d2c7ae
STEP 2: LABEL maintainer="Kolla Project (https://launchpad.net/kolla)" name="cinder-base" build-date="20200520"
STEP 3: RUN usermod --append --home /var/lib/cinder --groups kolla cinder     && mkdir -p /var/lib/cinder     && chown -R 42407:42407 /var/lib/cinder
STEP 4: RUN dnf -y install ceph-common lvm2 cryptsetup openstack-cinder python3-automaton python3-oslo-vmware     && dnf clean all && rm -rf /var/cache/dnf
Repository centos-opstools is listed more than once in the configuration
delorean-openstack-keystone-b74e95c27fff46a3abf 3.7 MB/s | 1.2 MB     00:00    
CentOS-8 - AppStream                            7.2 MB/s | 7.0 MB     00:00    
CentOS-8 - Base                                 2.2 MB/s | 2.2 MB     00:00    
CentOS-8 - Extras                                32 kB/s | 5.9 kB     00:00    
CentOS-8 - HA                                   1.7 MB/s | 564 kB     00:00    
CentOS-8 - NFS Ganesha 2.8                       44 kB/s |  11 kB     00:00    
CentOS-8 - OpsTools - collectd                  231 kB/s | 117 kB     00:00    
CentOS-8 - PowerTools                           667 kB/s | 2.0 MB     00:03    
dlrn-train-testing                              2.4 MB/s | 976 kB     00:00    
dlrn-train-build-deps                           1.4 MB/s | 395 kB     00:00    
Advanced Virtualization mirror                  116 kB/s |  72 kB     00:00    
Messaging RabbitMQ                              257 kB/s |  80 kB     00:00    
No match for argument: ceph-common
Package lvm2-8:2.03.05-5.el8.0.1.x86_64 is already installed.
Error: Unable to find a match: ceph-common

~~We will need to do some changes here to add those repos: https://opendev.org/openstack/tripleo-repos/src/branch/master/tripleo_repos/main.py#L46~~
~~where we get ceph for train ? https://trunk.rdoproject.org/centos8-master/deps/storage/storage8-ceph-nautilus/~~
- ~~added some work arounds solution on patch https://review.opendev.org/#/c/729519/ ~~
~~https://review.opendev.org/729893 qdrouterd: ignore failure of disabling epel repos~~

OSP

May 19th

TripleO

Looks like stein fs020 hitting issue in overcloud-deploy.. waiting on repeat..
https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-stein/27d5e4f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

2020-05-19 07:08:47 | 
2020-05-19 07:08:47 | novacompute-1: Failed to connect to the host via ssh: Warning: Permanently
2020-05-19 07:08:47 | added '192.168.24.24' (ECDSA) to the list of known hosts.  Permission denied
2020-05-19 07:08:47 | (publickey,gssapi-keyex,gssapi-with-mic).

OSP

May 18th

Tripleo

https://bugs.launchpad.net/tripleo/+bug/1879392 [build-test-packages] Run DLRN: Unexpected DLRN return code
https://bugs.launchpad.net/tripleo/+bug/1879379 - OVB failures on Vexx cloud: none of the requested nodes are available and match the resource class baremetal
https://bugs.launchpad.net/tripleo/+bug/1879365 - container build issue (found on gate)
Affecting check/gate https://bugs.launchpad.net/tripleo/+bug/1879267 [Error: Failed to download metadata for repo 'advanced-virtualization'\n level=debug msg="error running [bash -x /tmp/yum_update.sh delorean-current,quickstart-centos-ceph-nautilus] in container \"centos-binary-swift-container-working-container\": error while running runtime: exit status 1 failing standlone deployment jobs]
https://review.opendev.org/728761 [Use centos proxy mirror for centos8 repos]- Patch is up
~~LP Bug: https://bugs.launchpad.net/tripleo/+bug/1879255 [openstack overcloud node introspect –all-manageable timing out and failing periodic tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master]~~
~~Temporary reschedule openstack-periodic-master:https://review.rdoproject.org/r/#/c/27586/~~
- ~~Related patches:~~
  ~~https://review.opendev.org/#/c/724822/ (Use latest version of python construct)~~
  ~~https://review.rdoproject.org/r/#/c/27549/ (Update minimal version of construct to 2.9.39)~~
  ~~https://review.rdoproject.org/r/#/c/27585/ ([DNM] promote baremetal component to unblock current-tripleo promotions)~~
LP Bug: https://bugs.launchpad.net/tripleo/+bug/1879292 [ERROR paunch [ ] Error running ['podman', 'run', '–name', 'rabbitmq_init_bundle', '–label', 'config_id=tripleo_step2', '–label', 'container_name=rabbitmq_init_bundle' failing periodic tripleo-ci-centos-8 scenario004-standalone-train and ovb-1ctlr_1comp-featureset002-train]
- Note: This is the consistent failure and not the promotion blocker.
- wes, if possible add rabbit debug commands to https://opendev.org/openstack/ansible-role-collect-logs/src/branch/master/defaults/main.yml#L174
- rabbitmqctl status

OSP

May 15th

TripleO

Pipeline: openstack-periodic-latest-released

ERROR: "Could not find or access '/home/zuul/workspace/.quickstart/vars/tempest_skip_ussuri.yml'"

Failing jobs:
1. periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri
2. periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-ussuri
3. periodic-tripleo-ci-centos-8-standalone-full-tempest-api-ussuri
4. periodic-tripleo-ci-centos-8-scenario012-standalone-ussuri
5. periodic-tripleo-ci-centos-8-scenario010-standalone-ussuri
6. periodic-tripleo-ci-centos-8-scenario007-standalone-ussuri
7. periodic-tripleo-ci-centos-8-scenario008-standalone-ussuri
8. periodic-tripleo-ci-centos-8-scenario003-standalone-ussuri
9. periodic-tripleo-ci-centos-8-scenario002-standalone-ussuri
10. periodic-tripleo-ci-centos-8-scenario001-standalone-ussuri
11. periodic-tripleo-ci-centos-8-standalone-ussuri
12. periodic-tripleo-ci-centos-8-undercloud-containers-ussuri

Error log

  2020-05-14 23:03:09.383163 | primary | TASK [Load tempest skiplist file] **********************************************
  2020-05-14 23:03:09.383171 | primary | Thursday 14 May 2020  23:03:09 +0000 (0:00:00.052)       0:44:05.425 **********
  2020-05-14 23:03:09.425360 | primary | fatal: [undercloud]: FAILED! => {
  2020-05-14 23:03:09.425423 | primary |     "ansible_facts": {},
  2020-05-14 23:03:09.425433 | primary |     "ansible_included_var_files": [],
  2020-05-14 23:03:09.425440 | primary |     "changed": false,
  2020-05-14 23:03:09.425448 | primary |     "message": "Could not find or access '/home/zuul/workspace/.quickstart/vars/tempest_skip_ussuri.yml' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"
  2020-05-14 23:03:09.425470 | primary | }

This issue will be fixed by https://review.opendev.org/#/c/728134/ and https://review.opendev.org/#/c/728133/ (add ussuri tempest skiplist)

Below jobs are failing bacuse of emit_releases_file.py: error: argument –stable-release: invalid choice: 'ussuri' (choose from 'newton', 'ocata', 'pike', 'queens', 'rocky', 'stein', 'train', 'master')

Error log

  2020-05-14 22:12:11.131318 | primary | ++(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:113): main(): basename /home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset037.yml
  2020-05-14 22:12:11.132520 | primary | +(/home/zuul/src/opendev.org/openstack/tripleo-ci/toci_gate_test.sh:113): main(): python3 /home/zuul/src/opendev.org/openstack/tripleo-ci/scripts/emit_releases_file/emit_releases_file.py --stable-release ussuri --featureset-file /home/zuul/src/opendev.org/openstack/tripleo-quickstart/config/general_config/featureset037.yml --output-file /home/zuul/workspace/logs/releases.sh --log-file /home/zuul/workspace/logs/emit_releases_file.log --distro-name centos --distro-version 8 --is-periodic
  2020-05-14 22:12:11.316668 | primary | usage: emit_releases_file.py [-h] --stable-release
  2020-05-14 22:12:11.316747 | primary |                              {newton,ocata,pike,queens,rocky,stein,train,master}
  2020-05-14 22:12:11.316756 | primary |                              --distro-name {centos} --distro-version {7,8}
  2020-05-14 22:12:11.316763 | primary |                              --featureset-file FEATURESET_FILE
  2020-05-14 22:12:11.316770 | primary |                              [--output-file OUTPUT_FILE] [--log-file LOG_FILE]
  2020-05-14 22:12:11.316776 | primary |                              [--upgrade-from] [--is-periodic]
  2020-05-14 22:12:11.317979 | primary | emit_releases_file.py: error: argument --stable-release: invalid choice: 'ussuri' (choose from 'newton', 'ocata', 'pike', 'queens', 'rocky', 'stein', 'train', 'master')
  2020-05-14 22:12:12.165646 | primary | ERROR
  2020-05-14 22:12:12.165914 | primary | {
  2020-05-14 22:12:12.165958 | primary |   "delta": "0:00:03.616096",
  2020-05-14 22:12:12.165991 | primary |   "end": "2020-05-14 22:12:11.334283",
  2020-05-14 22:12:12.166019 | primary |   "msg": "non-zero return code",
  2020-05-14 22:12:12.166078 | primary |   "rc": 2,
  2020-05-14 22:12:12.166111 | primary |   "start": "2020-05-14 22:12:07.718187"
  2020-05-14 22:12:12.166140 | primary | }
  2020-05-14 22:12:12.214538 | 
  2020-05-14 22:12:12.214686 | PLAY RECAP
  2020-05-14 22:12:12.214748 | primary | ok: 8 changed: 5 unreachable: 0 failed: 1 skipped: 11 rescued: 0 ignored: 0
  2020-05-14 22:12:12.214779 | 
  2020-05-14 22:12:12.454582 | RUN END RESULT_NORMAL: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-ci/run-v3.yaml@master]
  2020-05-14 22:12:12.454795 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]

Failed job list:

  1. periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-ussuri
  2. periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-ussuri

Fix: https://review.opendev.org/#/c/723905/ (Add ussuri support for emit_releases_file.py

)

OSP

16.0 - latest content went trhu (20200513.n.1) + phase3 ran
16.1 - 20200511 p2 almost complete - p3 run today
- 1 octavia Tempest failure TBD
  - https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_2comp-ipv4-geneve/49/
    - needs investigation and escalation
13.0 - as below (y-day)+
- work on CIX continues
  - latest BZ update: https://bugzilla.redhat.com/show_bug.cgi?id=1835828#c3
on-going long-term isues:
- OSP17 still without attention (!) because of fires in OSP<16
- foreign jobs still invading p1/p2 views
  not yet tested https://code.engineering.redhat.com/gerrit/#/c/198375

May 14th

TripleO

Pipeline: periodic-tripleo-centos-7-rocky-containers-build-push is getting time out from TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/build.log]

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-rocky-containers-build-push#

Here this job is getting time out from this task:
```
2020-05-13 08:27:23.219557 | TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/build.log]
2020-05-13 10:21:06.985995 | RUN END RESULT_TIMED_OUT: [untrusted : opendev.org/openstack/tripleo-ci/playbooks/tripleo-buildcontainers/run.yaml@master]
2020-05-13 10:21:06.986377 | POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/post.yaml@master]
```
- if you see here the containers are not build successfully: https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-7-rocky-containers-build-push/cbf360b/logs/build.log
- I have increase the time out here https://review.rdoproject.org/r/#/c/27542/ ([DNM] Test periodic-tripleo-centos-7-rocky-containers-build-push) and tested this job -> it's pass with increase timeout.
- So one of the process is taking to much of time to push the build container that i can't fin till now (where exactly and which container is taking more time to build and push)
- Work around: https://review.rdoproject.org/r/#/c/27550/ (Increase timeout for "periodic-tripleo-centos-7-rocky-containers-build-push")
- i see this issue is with train as well and they increase the timeout here https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/tripleoci.yaml#L349

OSP

16.0 retriggered blue, p3 kicked
- https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP16/job/multijob-phase2-osp-16.0-RHEL-8.1/9/
16.1 to be done tomorrow
OSP13 p1 failure BZed and CIXed:
- https://trello.com/c/bGsC674i CIX
- https://bugzilla.redhat.com/show_bug.cgi?id=1835828 BZ
17 - ??? unknown who debugs it and in what state it is
IR master brekage:
- breakage cause reverted seems to be: http://post-office.corp.redhat.com/archives/rhos-infrared/2020-May/msg00071.html
  - https://projects.engineering.redhat.com/browse/RHOSINFRA-3317
    - https://code.engineering.redhat.com/gerrit/#/c/200394/

May 13th

TripleO

gate

tripleo-ci-centos-7-undercloud-upgrades jobs are failing on stable/queens : https://review.opendev.org/#/q/topic:queens-backports+(status:open+OR+status:merged)
- LP Bug: https://bugs.launchpad.net/tripleo/+bug/1877031 [queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version]
- Fix: https://review.opendev.org/#/c/727696/1 [Don't run tripleo-ci-centos-7-undercloud-upgrades in queens]
  - Update:
    On #rhos-ops channel there was some discussion between apevec and ykarale regarding this issue and apevec commented on LP Bug.
    He suggested deprecating this upgrade job.
    Note: Refer Comment #2

OSP

16.1 jobs occupy queue
- 180 jobs still after 3/4 days
16.0 - promoting content 12.1
- getting jobs thru CI to reach some reasonable reporting state
- solving situation around passed_phase2 promoted
  - %TODO add jira
13 - live deployment reproduced
- no CIX so far because still no clue what happens (no BZ yet)
17 - nothing

May 12th

Tripleo

openstack-periodic-24h

https://bugs.launchpad.net/tripleo/+bug/1878248 NetworkSecGroupTest failing on fs020 stein/train
https://review.opendev.org/727287 [skiplist] NetworkSecGroupTest timeout

gate

<bhagyashris|ruck> https://review.opendev.org/#/c/726993/
<bhagyashris|ruck> https://review.opendev.org/#/c/726004/

https://review.opendev.org/#/c/727113/ [linters refresh w/ afferent bugfixes] fixes this bug https://bugs.launchpad.net/tripleo/+bug/1878150 [tox-linters jobs failing with AttributeError]

Pipeline: openstack-periodic-master

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master#

Job Failures:
- periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master
  - panda has submitted fix for this job failure
    - Fix: https://review.rdoproject.org/r/#/c/26694/ , https://review.opendev.org/#/c/722677
    - testproject: https://review.rdoproject.org/r/27435 Test ovn job fix on #26694
- periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
  - Reported bug : https://bugs.launchpad.net/tripleo/+bug/1878190
- Note: For more details about above job failure you can check May 11th -> Pipeline: openstack-periodic-master section

Pipeline: openstack-periodic-latest-released

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released

Job Failures:
- periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train
  - Reported Bug: https://bugs.launchpad.net/tripleo/+bug/1878197
  - This is failing for long time or never worked: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train

OSP

Patience with octavia is gone
- escalation: https://trello.com/c/pkHE0ay2
- ignoring until they will fix it
New OSP13 p1 failure
- https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP13/job/phase1-13_director-rhel-7.8-virthost-1cont_1comp_1ceph-ipv4-vxlan-ceph-containers/17/
Testing of UMB triggering
- https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/umb-testing/97/

May 11th

Tripleo

Pipeline: Gate

bang.. opened bug https://bugs.launchpad.net/tripleo/+bug/1878101

Seen this a few times today
https://6d806783ed4dfdd971c5-158a33a5449e12f6f494625dd8517fb1.ssl.cf5.rackcdn.com/726374/1/gate/tripleo-ci-centos-8-undercloud-containers/2e3a947/logs/undercloud/home/zuul/undercloud_install.log

https://e453f1d8808c5b6bd184-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf2.rackcdn.com/722662/1/gate/tripleo-ci-centos-8-undercloud-containers/ad8d1e6/logs/undercloud/home/zuul/undercloud_install.log

TASK [AllNodesValidationConfig] ************************************************
Monday 11 May 2020 20:46:19 +0000 (0:00:01.555) 0:00:50.282 ************
fatal: [undercloud]: FAILED! => changed=true
msg: non-zero return code
rc: 1
stderr: ''
stderr_lines: <omitted>
stdout: |-
Trying to ping default gateway 10.4.70.1…Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
Ping to 10.4.70.1 failed. Retrying…
FAILURE
10.4.70.1 is not pingable.
stdout_lines: <omitted>

Pipeline: openstack-periodic-master

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master#

Job Failures

periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master

Note: This job is not promotion blocker but it's consistently failing
Details: Tempest is failing
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/a86ab83/logs/undercloud/home/zuul/tempest/tempest.html.gz

Discussion on IRC:
I have discussed this failure with <arxcruz> and chandankumar:

<bhagyashris|ruck> chandankumar, arxcruz this job -> "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master" is consistently failing https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master here
    <bhagyashris|ruck> chandankumar, arxcruz https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/a86ab83/logs/undercloud/home/zuul/tempest/tempest.html.gz
    <arxcruz> bhagyashris|ruck: i though panda|pto had fixed it
    <arxcruz> bhagyashris|ruck: you need to add an tempest option on that
    <arxcruz> bhagyashris|ruck: give me a few minutes and i'll show you
    <chandankumar> arxcruz, bhagyashris|ruck regarding ovn tempest, patch is coming soon

Fix: https://review.rdoproject.org/r/#/c/26694/
https://review.opendev.org/#/c/722677/2

periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master

Note: This job is not promotion blocker but it's consistently failing
Details: Execute tempest is failing
https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/7788828/logs/undercloud/var/log/tempest/stestr_results.html.gz

Discussion on IRC:
I have discussed this with arxcruz:

<bhagyashris|ruck> chandankumar, arxcruz is this know failure https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/7788828/logs/undercloud/var/log/tempest/stestr_results.html.gz
<bhagyashris|ruck> chandankumar, arxcruz this job is consistently failing "periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master"
<bhagyashris|ruck> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-master&job_name=periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master
<arxcruz> bhagyashris|ruck: seems to be random failure
* derekh (~derekh@93.107.228.126) has joined
<arxcruz> I would wait for a next run to double check, if continues to happen, open a bug
* amoralej|off is now known as amoralej
<arxcruz> actually, it's happening a lot...
<arxcruz> bhagyashris|ruck: I would open a bug, and add it on the skip list
<bhagyashris|ruck> arxcruz, yes, i checked last 4 failure and it's failing to execute the tempest test
<bhagyashris|ruck> arxcruz, yes ack just ping once you open a bug

Pipeline: openstack-periodic-latest-released

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released

Job Failures:
periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train
- Note: this job is consistently Time out and not a promotion blocker as well.
- Details:
  https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/extra/errors.txt.txt.gz
  https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/containers/ironic/ironic-conductor.log.txt.gz
  https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train/2bcfe3c/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz

periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train
https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train

Note: This job is consistently failing and is not promotion blocker as well. This jobs are me
Details: Execute tempest tests failing
https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/job-output.txt
https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/logs/undercloud/var/log/tempest/stestr_results.html.gz

Duscussion on IRC with chandan:

<bhagyashris|ruck> hey
<bhagyashris|ruck> are you working on fs021
<bhagyashris|ruck> because this job is consistently failing periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train
<chandankumar> bhagyashris|ruck fs021 is meant to fail
<chandankumar> use ignore karo
<bhagyashris|ruck> https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released&job_name=periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train
<chandankumar> yes
<chandankumar> in every relase it's getting failed
<bhagyashris|ruck> ok why
<bhagyashris|ruck> ?
<bhagyashris|ruck> https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train/5e3cae2/logs/undercloud/var/log/tempest/stestr_results.html.gz
<bhagyashris|ruck> why?
<chandankumar> bhagyashris|ruck here skipped test are running
<bhagyashris|ruck> ok
<bhagyashris|ruck> so should i safely ignore this

~~Pipeline: openstack-component-compute~~

~~https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master&pipeline=openstack-component-compute~~

* Job failures:
* periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master
~~https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master&pipeline=openstack-component-compute~~
* Note: this is failing randomly
* Details: Execute tempest tests failing because of insufficient ip adderss
https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-compute-master/cdfe0e3/logs/undercloud/var/log/tempest/stestr_results.html.gz
* Bug: https://bugs.launchpad.net/tripleo/+bug/1852770
* Fix: https://review.opendev.org/#/c/722662/

OSP

octavia jobs unstable
- debugging
trying to get CI reports for OSP16/16.1 for RelDel

May 8th

TripleO

~~cirros image fixed tempest bug~~

~~### tempest issues on stable branches
CentOS-7 OVB jobs are RED fs001
https://bugs.launchpad.net/tripleo/+bug/1875731
https://bugs.launchpad.net/tripleo/+bug/1876972
TRAIN: GREEN except by Tempest fail in fs039
STEIN: Tempest fail ( arx is looking at it )
ROCKY: Tempest fail ( @arxcruz FYI)
QUEENS: Tempest fail ( @arxcruz FYI)
https://bugs.launchpad.net/tripleo/+bug/1876087 –> tempest bug
FYI : @rfolco @arxcruz ~~

Added by bhagyashris:

Above bug is breaking "periodic-tripleo-ci-centos-7-ovb- 3ctlr_1comp_1supp-featureset039-train" job as well 1
Reason: The reason it's not happening in train+:- os_tempest is used where cirros image is getting downloaded instead of using cached. In case of fs039, it's running validate-tempest instead of os-tempest, so hitting issue.

So this is breaking Train as well.

Reference : https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train

~~container build failures = mirror issue~~

### container build
https://bugs.launchpad.net/tripleo/+bug/1877416 >> mirror issue

Bhagyashris observations:
job1: periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master&pipeline=openstack-periodic-master

This job is failing because of the tempest issue
- Note:
  Here we are getting two different error logs as below
  This job is consistently failing on openstack-periodic-master pipeline
- Error logs after 27th April:
  
  https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/b26393e/logs/undercloud/home/zuul/tempest/tempest.log.txt.gz
  
  https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/b26393e/logs/undercloud/home/zuul/tempest/tempest.html.gz
- Error logs before 27th April:
  https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/7d32c39/logs/undercloud/home/zuul/tempest/tempest.html.gz
This is issue may fixed by LP: https://bugs.launchpad.net/tripleo/+bug/1876087

~~job2: periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master~~

job2: periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master
Note: this job is recently added https://github.com/rdo-infra/review.rdoproject.org-config/commit/b83b902e6d913a02bae9337f2e6a2e95d9368cb3 and this is not in promotion criteria https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-8/master.ini#L33

overcloud deployment failed here:

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/7611a6d/job-output.txt

Heat resource create failed (Nova compute heat resource)

https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/7611a6d/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

* Reported ad LP bug: https://bugs.launchpad.net/tripleo/+bug/1877581

### bring scenario10 online
@TheG Please work the networking team to bring https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone online.
(wes) removed (non-periodic) here https://review.opendev.org/#/c/726224/1/zuul.d/layout.yaml
(rfolco) periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master: remove from promotion criteria if red

OSP

See https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg?view

May 7th

Tripleo

master periodic container build failing:
- NEW BUG: https://bugs.launchpad.net/tripleo/+bug/1877416
rhel rdo tox job failing
- https://bugs.launchpad.net/tripleo/+bug/1877299
https://images.rdoproject.org/centos8/master/rdo_trunk/tripleo-ci-testing/overcloud-full.tar doesn't exist
- https://bugs.launchpad.net/tripleo/+bug/1877230
- Here is the fix : https://review.opendev.org/#/c/725858/2

OSP

in previous doc: https://hackmd.io/1pY-KQB_QwOe-a-5oEXTRg?view

May 6th

Tripleo

1877031: queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version

OSP

Completed On-Going

undercloud containers

cix https://trello.com/c/E7gL6d4b/1490-cixlp1878101tripleociproa-ping-br-ctlplane-is-failing-too-often-trying-to-ping-default-gateway
lp https://bugs.launchpad.net/tripleo/+bug/1878101
logstash https://bit.ly/3bnXuwc
testproject https://review.rdoproject.org/r/#/c/27453/
upstream/check https://review.opendev.org/#/c/727754/2
Fix: https://review.opendev.org/#/c/727942/ (Use /32 netmask for VIPs)

INFRA Mirror issues

NOTICE: Our CI mirrors in OVH BHS1 and GRA1 regions were offline between 12:55 and 14:35 UTC, any failures there due to unreachable mirrors can safely be rechecked

https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-undercloud-upgrades&branch=stable%2Fqueens

History of bugs

bugs Reported

Bugzilla	Name	status
1873770	OVB fs001 in centos8 master fails to push certificates contents to controllers	Incomplete
1873892	Non root login prevented on overcloud machines	Fixed Release
1874019	scenario009-multinode.yaml and openshift.yaml is missing	In Progress
1875352	keystone container failed to start in scenario000	Triged
1875871	periodic rocky jobs failing with missing –name argument for pcs	Triged
1875846	Overcloud stack creation failed because of failed dependencies.	Closed
1875833	The WebSocket timed out before the Workflow completed in rocky/stain jobs	New
1876087	Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout	Triged
1876096	Queens: tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern tests failed	Triged
1876672	Python 2 - AttributeError: 'module' object has no attribute 'get_makefile_name'	Fixed Release
1876893	Error: error removing container - device or resource busy	In Progress
1877031	queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version

Handoff notes

Notes from previous RR cycle

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck and rover notes #27

tags: ruck_rover

on-going issues

TripleO

gate

RDO CI

OSP

Launchpad Bugs Reported

May 28th

Tripleo

OSP

May 27th

Tripleo

OSP

May 26th

Tripleo

OSP

May 25th

Tripleo

gate

RDO CI Failures

undercloud-upgrade failing in gate

OSP

May 22th

TripleO

OSP

May 21th

Tripleo

OSP

May 20th

Tripleo

msg: '[''mysql_init_bundle''] failed to start

RUN END RESULT_TIMED_OUT

async task did not complete within the requested time - 5700s

Unable to start service tripleo_rabbitmq.service

FAILED - RETRYING: Wait for puppet host configuration to finish

tempest (network_basic_ops)

OSP

May 19th

TripleO

OSP

May 18th

Tripleo

OSP

May 15th

TripleO

OSP

May 14th

TripleO

OSP

May 13th

TripleO

OSP

May 12th

Tripleo

openstack-periodic-24h

gate

OSP

May 11th

Tripleo

OSP

May 8th

TripleO

OSP

May 7th

Tripleo

OSP

May 6th

Tripleo

OSP

Completed On-Going

undercloud containers

INFRA Mirror issues

History of bugs

Handoff notes

tags: `ruck_rover`