# Ruck and Rover notes #40
###### tags: `ruck_rover`

:::info
Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ)
**Ruck Rover - Unified Sprint #<fix>**
Dates: Feb 4 - Feb 25

Tripleo CI team ruck|rover: arxcruz , ysandeep
OSP CI team ruck|rover: <fix>Names</fix>

Previous notes: [link](https://hackmd.io/_XvcCzQlQ1-A9ygYMlmvKA)
:::

[TOC]

### Issues to track on-going
put these issues in the spoiler.
:::danger
#### tripleo

check/Gate:

Stein branch check/Gate jobs are failing because of missing container images, Error - ImageNotFoundException
https://bugs.launchpad.net/tripleo/+bug/1915921

promotions:

**Master: 17th Feb** (Yellow)

Jobs are green, we haven't got promotion recentrly because earlier promoter was stopped and we have also stopped triggering master line for c8 stream work : https://review.rdoproject.org/r/#/c/32064/

* Last run from 22nd: https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main

sc01/02 only failed once, passed in testproject

We have a bug for fs39 for master, fixed now.. fix need to hit integration line.
* [ Master Promotion blocker] Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500"
https://bugs.launchpad.net/tripleo/+bug/1916445
* Fix merged https://review.opendev.org/c/openstack/tripleo-common/+/776870

We also need to talk with security dfg about fs039 - need to drop/migrate this job

**Victoria - Green - last promoted yesterday 23rd Feb**
* Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged]
https://bugs.launchpad.net/tripleo/+bug/1916742

**Ussuri - Green - last promoted yesterday 23rd Feb**

**c8 train- yellow - Last promoted on 17th Feb, 2021**

Waiting for promoter server to be ready for c8 train, last run was completely green https://review.rdoproject.org/zuul/buildset/0c074bf21d4e463d82d51e42baf7400b

** c7 train - Red - 23rd Jan**
[CIX][LP:1915519][tripleoci][proa][Train][CentOS7][scenario004] Failing with Error: 'ip-192.168.24.3' already exists. Too many tries"
https://bugs.launchpad.net/tripleo/+bug/1915519

* Stein - Green - Promoted on 24th Feb
* Rocky - Red
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network"
https://bugs.launchpad.net/tripleo/+bug/1916695
* Queens - Promoted on 11th Feb

:::

:::info
add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections.
:::

## Feb 25th
### Tripleo

## Feb 24th
### Tripleo

* Launchpad bug 1916742 in tripleo "Victoria Container build job is failing, buildah deprecations warning is added as a first item in the list instead of container name" [Critical,Triaged]
https://bugs.launchpad.net/tripleo/+bug/1916742
* https://review.opendev.org/c/openstack/tripleo-ci/+/777378

* Rocky - periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky is failing with NeutronError: "Invalid input for operation: segmentation_id requires physical_network for VLAN provider network"
https://bugs.launchpad.net/tripleo/+bug/1916695

* Upgrade jobs looking for wrong hash
https://bugs.launchpad.net/tripleo/+bug/1916689

Promotions:

Victoria: below jobs failed in last run, Monitor today's run
~~~
periodic-tripleo-ci-centos-8-scenario010-standalone-victoria
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria
periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-victoria
~~~

Ussuri: below jobs failed yesterday, Cleared in today's run
~~~
periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri
~~~

## Feb 23rd
### Tripleo

* [Fixed] Promotion blocker - https://bugs.launchpad.net/tripleo/+bug/1916561
Launchpad bug 1916561 in tripleo "tripleo-ci-centos-7-containers-multinode does not build with patches" [High,Triaged]
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/777051

## Feb 22nd
### Tripleo

* [Promotion blocker] Heat stack creation failing for featureset039 in Master branch with "ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: MessagingTimeout, Code: 500"
https://bugs.launchpad.net/tripleo/+bug/1916445

* https://review.opendev.org/c/openstack/tripleo-common/+/776870 will fix job.
* We also need to talk with security dfg about fs039 - need to drop/migrate this job

* tripleo-ci-centos-7-containers-multinode-queens failing on TASK [tripleo-bootstrap : Set 'dns=none' in /etc/NetworkManager/NetworkManager.conf] with "error while evaluating conditional (ansible_facts.services['NetworkManager.service']['status'] == 'enabled'): 'dict object' has no attribute 'status"
https://bugs.launchpad.net/tripleo/+bug/1916459
https://review.opendev.org/c/openstack/tripleo-common/+/776925 fix merged today

* periodic-tripleo-ci-centos-8-scenario010-standalone-master failed twice on tempest - ran successfully with testproject

* periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria failed in last run - will monitor today's run
* periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri failed twice - rerunning via testproject https://review.rdoproject.org/r/#/c/28446/

## Feb 19th
### Tripleo

* Promotions:-
* Master - Green
* Victoria - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria failed yesterday, will monitor today's run
* Ussuri - periodic-tripleo-ci-centos-8-standalone-full-tempest-api-ussuri & periodic-tripleo-ci-centos-8-scenario002-standalone-ussuri failed in today's run, Cleared in rerun: https://review.rdoproject.org/r/#/c/28446
* c8 train: All green today

## Feb 18th
### Tripleo

* Promotions:-
* Master - Good
* Victoria: periodic-tripleo-centos-8-buildimage-overcloud-full-victoria - failed, other ovb jobs skiped, Failed job rerun with https://review.rdoproject.org/r/#/c/28446/, awaiting results
* Ussuri - periodic-tripleo-ci-centos-8-standalone-ussuri failed in last run, cleared in rerun https://review.rdoproject.org/r/#/c/31612/
* c8 - Train: periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-train - Failed in tempest, Next scheduled run is ongoing

## Feb 17th
### Tripleo

* [Fixed][Promotion blocker]Launchpad bug 1915932 in tripleo "Centos7 Stein Jobs failed with error - sudo: pip3: command not found" [Critical,Triaged]
https://bugs.launchpad.net/tripleo/+bug/1915932
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/776391

* [Promotion blocker]Launchpad bug 1915921 in tripleo "Stein branch check/Gate jobs are failing because of missing container images, Error - ImageNotFoundException" [Critical,Triaged]
https://bugs.launchpad.net/tripleo/+bug/1915921

* good news all c8 line promoted today

## Feb 16th
### Tripleo
* Victoria branch Scenario10 logs collection not collecting all the directory and files.
https://bugs.launchpad.net/tripleo/+bug/1915778
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/771593

## Feb 15th
### Tripleo

* Good news - master/v/u/c8 train all promoted yesterday

## Feb 12th
### Tripleo

* "[Fixed]Issue with Upstream Mirror - error: Status code: 403 "
https://bugs.launchpad.net/tripleo/+bug/1915487
* Workaround:-

Infra team added afs server - afsdb01 and afsdb02 to emergency disable list and added back missing public UDP ports in firewall rules.
Permanent solution:-
https://review.opendev.org/c/opendev/system-config/+/775311

## Feb 11th
### TripleO

* Promotion status
* Master - 10th Feb - Green
* Victoria - 05th Feb (06 days old now) - Red
* Ussuri - 05th Feb (06 days old now) - Red
* C8 train - 07th Feb (04 days old now) - Yellow
**Let's wait for today's run**

* Seeing some node failures in rdo - https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE
* Ping on internal #rhos-ops channel, dpawlik updated the ticket with vexxhost
* NB planned outage in VexxH today http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020347.html

* Infra issue with some ovb jobs:-
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-ussuri/aa5e30e/job-output.txt
~~~
2021-02-11 05:58:08.405522 | TASK [ovb-manage : Attach instance to public OVB network]
2021-02-11 05:58:28.797485 | primary | Failed to attach network adapter device to bcb82723-5001-42fd-95fa-6ccde37417a2 (HTTP 500) (Request-ID: req-095b7c67-e1b1-479a-882e-875a7e3b88e1)
~~~
* Reported to #rhos-ops

* Check/Gate patches failures
* https://review.opendev.org/c/openstack/tripleo-heat-templates/+/774779/ , Ussuru Upgrade job failed - Looks like network failure between node and content provider job - posted recheck
* https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/774892- 500 Server Error: Internal Server Error for url: https://opendev.org/openstack/requirements/raw/branch/stable/ussuri/upper-constraints.txt - Posted recheck

## Feb 10th
### TripleO

* Promotion - Master
* periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master and periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master failed on tempest - Will keep an eye on next scheduled run.

* [Fixed]Promotion blocker [Train] [CentOS7] Undercloud jobs puppet task ertmonger_certificate[haproxy-external-cert] fails with Unrecognized parameter or wrong value type
https://bugs.launchpad.net/tripleo/+bug/1915242

## Feb 09th
### TripleO

* [Fixed][promotion-blocker] Number of jobs fail with RETRY_LIMIT, No module named 'setuptools_rust
https://bugs.launchpad.net/tripleo/+bug/1915101
Patch
https://review.opendev.org/c/openstack/requirements/+/774593
We need new pip:-
https://review.opendev.org/c/openstack/tripleo-ci/+/774603

## Feb 08th
### TripleO

* Good news
- **Master promoted - 08th feb**
* sc-10 still timing out, removed from criteria(I think wes removed it from criteria)
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774419 - Skipping some tests to avoid timeout
- **train promoted - 07th feb**

* periodic-tripleo-ci-centos-8-scenario004-standalone-master failed in last periodic run, Cleared in testproject rerun
* periodic-tripleo-ci-centos-8-scenario004-standalone-master https://review.rdoproject.org/zuul/build/1735f67f22014c35afec550efc257772 : SUCCESS in 1h 29m 45s

* [Fixed][Promotion blocker]Victoria/Ussuri fs020 failed with Error: The --deployed-server cannot be used without the --disable-validations https://bugs.launchpad.net/tripleo/+bug/1914982
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/774416
* Testing here - https://review.rdoproject.org/r/#/c/28446/

* [Not observed again]Again observing issues with limestone mirror - https://bugs.launchpad.net/tripleo/+bug/1914585
* Chatting with #opendev infra

* [Fixed]Jobs fails on stein and older branches with ERROR! the role 'tripleo_validations' was not found (https://bugs.launchpad.net/tripleo/+bug/1914993)
* Patch - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/774445

## Feb 05th
### TripleO

* [Fixed]Promotion blocker [Master featureset020/fs039 failed while trying to upload image, HttpException: 500: Server Error for url: https://192.168.24.2:13292/v2/images/b898eee8-66c3-4a27-bbad-e6ca19d1f8d8/file, Internal Server Error](https://bugs.launchpad.net/tripleo/+bug/1914735)
* Patches are up for fs39, testproject is running for fs020

* periodic-tripleo-ci-centos-8-scenario010-standalone-network-master is timing out
* We need network component promotion for new python-ovn-octavia-provider patch - https://review.opendev.org/c/openstack/ovn-octavia-provider/+/771889/
* Context: Brian comment on https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774079

* periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/victoria failed on tempest
* Victoria(No logs)
* ussuri failed on TrafficOperationsScenarioTest
https://logserver.rdoproject.org/openstack-periodic-integration-stable2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-ussuri/9773373/logs/undercloud/var/log/tempest/stestr_results.html.gz

## Feb 04th
### TripleO

Check:-

* [Bug#1914585 Content provider jobs failed after failing to connect to mirrors, Failed to connect to mirror.regionone.limestone.opendev.org port 8080: No route to host Edit](https://bugs.launchpad.net/tripleo/+bug/1914585)
* Pinged on #opendev for someone from infra to look

* [Bug#1914600 "periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master is failing on tempest test with Exception: Server 192.168.24.122 on port 60092/8085 did not begin passing traffic within the timeout period."](https://bugs.launchpad.net/tripleo/+bug/1914600)
* Failing tempest test was not running on last green run, this test was removed from skiplist recently
* https://opendev.org/openstack/openstack-tempest-skiplist/commit/ecb8c966af1fbc544fd69d3b7b185ea807713a91
* https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/774079 - we added the test back in skiplist

Promotions:-

* Master promotion line - Fs001/35 failed on tempest
* Cleared with testproject rerun https://review.rdoproject.org/r/#/c/28458/
~~~
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master https://review.rdoproject.org/zuul/build/3903c34aa7cf425ba4b59153d6f5ba9c : SUCCESS in 3h 38m 04s
periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master https://review.rdoproject.org/zuul/build/23bb024a00664f3cb50bfdd86f61af0f : SUCCESS in 3h 52m 02s
~~~
* Ussuri promotion line - periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-ussuri failed
* Rerunning with testproject - https://review.rdoproject.org/r/#/c/23626/

* Train promotion line - Some jobs were skipped due to node_failure on 1 job
* Rerunning with testproject - https://review.rdoproject.org/r/#/c/28537/

### OSP