Ruck and Rover notes #40

# Ruck and Rover notes #40 ###### tags: `ruck_rover` :::info Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ) **Ruck Rover - Unified Sprint #<fix>** Dates: Feb 4 - Feb 25 Tripleo CI team ruck|rover: arxcruz , ysandeep OSP CI team ruck|rover: <fix>Names</fix> Previous notes: [link](https://hackmd.io/_XvcCzQlQ1-A9ygYMlmvKA) ::: [TOC] ### Issues to track on-going put these issues in the spoiler. :::danger #### tripleo check/Gate: Stein branch check/Gate jobs are failing because of missing container images, Error - ImageNotFoundException https://bugs.launchpad.net/tripleo/+bug/1915921 promotions: **Master: 17th Feb** (Yellow) Jobs are green, we haven't got promotion recentrly because earlier promoter was stopped and we have also stopped triggering master line for c8 stream work : https://review.rdoproject.org/r/#/c/32064/ * Last run from 22nd: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main sc01/02 only failed once, passed in testproject We have a bug for fs39 for master, fixed now.. fix need to hit integration line. * [ Master Promotion blocker] Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500" https://bugs.launchpad.net/tripleo/+bug/1916445 * Fix merged https://review.opendev.org/c/openstack/tripleo-common/+/776870 We also need to talk with security dfg about fs039 - need to drop/migrate this job **Victoria - Green - last promoted yesterday 23rd Feb** * Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged] https://bugs.launchpad.net/tripleo/+bug/1916742 **Ussuri - Green - last promoted yesterday 23rd Feb** **c8 train- yellow - Last promoted on 17th Feb, 2021** Waiting for promoter server to be ready for c8 train, last run was completely green https://review.rdoproject.org/zuul/buildset/0c074bf21d4e463d82d51e42baf7400b ** c7 train - Red - 23rd Jan** [CIX][LP:1915519][tripleoci][proa][Train][CentOS7][scenario004] Failing with Error: 'ip-192.168.24.3' already exists. Too many tries" https://bugs.launchpad.net/tripleo/+bug/1915519 * Stein - Green - Promoted on 24th Feb * Rocky - Red periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network" https://bugs.launchpad.net/tripleo/+bug/1916695 * Queens - Promoted on 11th Feb ::: :::info add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections. ::: ## Feb 25th ### Tripleo ## Feb 24th ### Tripleo * Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged] https://bugs.launchpad.net/tripleo/+bug/1916742 * https://review.opendev.org/c/openstack/tripleo-ci/+/777378 * Rocky - periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network" https://bugs.launchpad.net/tripleo/+bug/1916695 * Upgrade jobs looking for wrong hash https://bugs.launchpad.net/tripleo/+bug/1916689 Promotions: Victoria: below jobs failed in last run, Monitor today's run ~~~ periodic-tripleo-ci-centos-8-scenario010-standalone-victoria periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria ~~~ Ussuri: below jobs failed yesterday, Cleared in today's run ~~~ periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri ~~~ ## Feb 23rd ### Tripleo * [Fixed] Promotion blocker - https://bugs.launchpad.net/tripleo/+bug/1916561 Launchpad bug 1916561 in tripleo "tripleo-ci-centos-7-containers-multinode does not build with patches" [High,Triaged] * https://review.opendev.org/c/openstack/tripleo-quickstart/+/777051 ## Feb 22nd ### Tripleo * [Promotion blocker] Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500" https://bugs.launchpad.net/tripleo/+bug/1916445 * https://review.opendev.org/c/openstack/tripleo-common/+/776870 will fix job. * We also need to talk with security dfg about fs039 - need to drop/migrate this job * tripleo-ci-centos-7-containers-multinode-queens failing on TASK [tripleo-bootstrap : Set 'dns=none' in /etc/NetworkManager/NetworkManager.conf] with "error while evaluating conditional (ansible_facts.services['NetworkManager.service']['status'] == 'enabled'): 'dict object' has no attribute 'status" https://bugs.launchpad.net/tripleo/+bug/1916459 https://review.opendev.org/c/openstack/tripleo-common/+/776925 fix merged today * periodic-tripleo-ci-centos-8-scenario010-standalone-master failed twice on tempest - ran successfully with testproject * periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria failed in last run - will monitor today's run * periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri failed twice - rerunning via testproject https://review.rdoproject.org/r/#/c/28446/ ## Feb 19th ### Tripleo * Promotions:- * Master - Green * Victoria - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria failed yesterday, will monitor today's run * Ussuri - periodic-tripleo-ci-centos-8-standalone-full-tempest-api-ussuri & periodic-tripleo-ci-centos-8-scenario002-standalone-ussuri failed in today's run, Cleared in rerun: https://review.rdoproject.org/r/#/c/28446 * c8 train: All green today ## Feb 18th ### Tripleo * Promotions:- * Master - Good * Victoria: periodic-tripleo-centos-8-buildimage-overcloud-full-victoria - failed, other ovb jobs skiped, Failed job rerun with https://review.rdoproject.org/r/#/c/28446/, awaiting results * Ussuri - periodic-tripleo-ci-centos-8-standalone-ussuri failed in last run, cleared in rerun https://review.rdoproject.org/r/#/c/31612/ * c8 - Train: periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-train - Failed in tempest, Next scheduled run is ongoing ## Feb 17th ### Tripleo * [Fixed][Promotion blocker]Launchpad bug 1915932 in tripleo "Centos7 Stein Jobs failed with error - sudo: pip3: command not found" [Critical,Triaged] https://bugs.launchpad.net/tripleo/+bug/1915932 * https://review.opendev.org/c/openstack/tripleo-quickstart/+/776391 * [Promotion blocker]Launchpad bug 1915921 in tripleo "Stein branch check/Gate jobs are failing because of missing container images, Error - ImageNotFoundException" [Critical,Triaged] https://bugs.launchpad.net/tripleo/+bug/1915921 * good news all c8 line promoted today ## Feb 16th ### Tripleo * Victoria branch Scenario10 logs collection not collecting all the directory and files. https://bugs.launchpad.net/tripleo/+bug/1915778 * https://review.opendev.org/c/openstack/tripleo-quickstart/+/771593 ## Feb 15th ### Tripleo * Good news - master/v/u/c8 train all promoted yesterday ## Feb 12th ### Tripleo * "[Fixed]Issue with Upstream Mirror - error: Status code: 403 " https://bugs.launchpad.net/tripleo/+bug/1915487 * Workaround:- Infra team added afs server - afsdb01 and afsdb02 to emergency disable list and added back missing public UDP ports in firewall rules. Permanent solution:- https://review.opendev.org/c/opendev/system-config/+/775311 ## Feb 11th ### TripleO * Promotion status * Master - 10th Feb - Green * Victoria - 05th Feb (06 days old now) - Red * Ussuri - 05th Feb (06 days old now) - Red * C8 train - 07th Feb (04 days old now) - Yellow **Let's wait for today's run** * Seeing some node failures in rdo - https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE * Ping on internal #rhos-ops channel, dpawlik updated the ticket with vexxhost * NB planned outage in VexxH today http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020347.html * Infra issue with some ovb jobs:- https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/aa5e30e/job-output.txt ~~~ 2021-02-11 05:58:08.405522 | TASK [ovb-manage : Attach instance to public OVB network] 2021-02-11 05:58:28.797485 | primary | Failed to attach network adapter device to bcb82723-5001-42fd-95fa-6ccde37417a2 (HTTP 500) (Request-ID: req-095b7c67-e1b1-479a-882e-875a7e3b88e1) ~~~ * Reported to #rhos-ops * Check/Gate patches failures * https://review.opendev.org/c/openstack/tripleo-heat-templates/+/774779/ , Ussuru Upgrade job failed - Looks like network failure between node and content provider job - posted recheck * https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/774892- 500 Server Error: Internal Server Error for url: https://opendev.org/openstack/requirements/raw/branch/stable/ussuri/upper-constraints.txt - Posted recheck ## Feb 10th ### TripleO * Promotion - Master * periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master and periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master failed on tempest - Will keep an eye on next scheduled run. * [Fixed]Promotion blocker [Train] [CentOS7] Undercloud jobs puppet task ertmonger_certificate[haproxy-external-cert] fails with Unrecognized parameter or wrong value type https://bugs.launchpad.net/tripleo/+bug/1915242 ## Feb 09th ### TripleO * [Fixed][promotion-blocker] Number of jobs fail with RETRY_LIMIT, No module named 'setuptools_rust https://bugs.launchpad.net/tripleo/+bug/1915101 Patch https://review.opendev.org/c/openstack/requirements/+/774593 We need new pip:- https://review.opendev.org/c/openstack/tripleo-ci/+/774603 ## Feb 08th ### TripleO * Good news - **Master promoted - 08th feb** * sc-10 still timing out, removed from criteria(I think wes removed it from criteria) * https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774419 - Skipping some tests to avoid timeout - **train promoted - 07th feb** * periodic-tripleo-ci-centos-8-scenario004-standalone-master failed in last periodic run, Cleared in testproject rerun * periodic-tripleo-ci-centos-8-scenario004-standalone-master https://review.rdoproject.org/zuul/build/1735f67f22014c35afec550efc257772 : SUCCESS in 1h 29m 45s * [Fixed][Promotion blocker]Victoria/Ussuri fs020 failed with Error: The --deployed-server cannot be used without the --disable-validations https://bugs.launchpad.net/tripleo/+bug/1914982 * https://review.opendev.org/c/openstack/tripleo-quickstart/+/774416 * Testing here - https://review.rdoproject.org/r/#/c/28446/ * [Not observed again]Again observing issues with limestone mirror - https://bugs.launchpad.net/tripleo/+bug/1914585 * Chatting with #opendev infra * [Fixed]Jobs fails on stein and older branches with ERROR! the role 'tripleo_validations' was not found (https://bugs.launchpad.net/tripleo/+bug/1914993) * Patch - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/774445 ## Feb 05th ### TripleO * [Fixed]Promotion blocker [Master featureset020/fs039 failed while trying to upload image, HttpException: 500: Server Error for url: https://192.168.24.2:13292/v2/images/b898eee8-66c3-4a27-bbad-e6ca19d1f8d8/file, Internal Server Error](https://bugs.launchpad.net/tripleo/+bug/1914735) * Patches are up for fs39, testproject is running for fs020 * periodic-tripleo-ci-centos-8-scenario010-standalone-network-master is timing out * We need network component promotion for new python-ovn-octavia-provider patch - https://review.opendev.org/c/openstack/ovn-octavia-provider/+/771889/ * Context: Brian comment on https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774079 * periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/victoria failed on tempest * Victoria(No logs) * ussuri failed on TrafficOperationsScenarioTest https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/9773373/logs/undercloud/var/log/tempest/stestr_results.html.gz ## Feb 04th ### TripleO Check:- * [Bug#1914585 Content provider jobs failed after failing to connect to mirrors, Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host Edit](https://bugs.launchpad.net/tripleo/+bug/1914585) * Pinged on #opendev for someone from infra to look * [Bug#1914600 "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing on tempest test with Exception: Server 192.168.24.122 on port 60092/8085 did not begin passing traffic within the timeout period."](https://bugs.launchpad.net/tripleo/+bug/1914600) * Failing tempest test was not running on last green run, this test was removed from skiplist recently * https://opendev.org/openstack/openstack-tempest-skiplist/commit/ecb8c966af1fbc544fd69d3b7b185ea807713a91 * https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774079 - we added the test back in skiplist Promotions:- * Master promotion line - Fs001/35 failed on tempest * Cleared with testproject rerun https://review.rdoproject.org/r/#/c/28458/ ~~~ periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master https://review.rdoproject.org/zuul/build/3903c34aa7cf425ba4b59153d6f5ba9c : SUCCESS in 3h 38m 04s periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master https://review.rdoproject.org/zuul/build/23bb024a00664f3cb50bfdd86f61af0f : SUCCESS in 3h 52m 02s ~~~ * Ussuri promotion line - periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri failed * Rerunning with testproject - https://review.rdoproject.org/r/#/c/23626/ * Train promotion line - Some jobs were skipped due to node_failure on 1 job * Rerunning with testproject - https://review.rdoproject.org/r/#/c/28537/ ### OSP

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.