# Ruck Rover 2022-08-12 to 2022-08-18

###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/5zEei6JoTSmZ9mXlN5bxow

[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

[Downstream cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[OpenStack Program Meeting 2022](
https://docs.google.com/document/d/1n6ArkMh68R9zivjlyGbpedkggk1wMwEIcrMZSN2uIjc/edit)

[Downstream promoter](http://10.0.110.143/promoter_logs/)

---
Ongoing Bugs

Blocking:

Wallaby Check/Gate
* https://bugs.launchpad.net/tripleo/+bug/1986960
Wallaby Ceph based scenarios(SC01/SC04/SC010) are failing with "ERROR: Container release quincy != cephadm release pacific

* https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
* Xek proposed: https://review.opendev.org/853478
* ipa job running in check itself - need to work with security team to continue debug/fixing.

Degraded:

https://bugzilla.redhat.com/show_bug.cgi?id=2119349 -Component and Integration line jobs intermittently failing with retry_limit

* https://bugs.launchpad.net/tripleo/+bug/1983817 - Jobs failing with retry
* Still happening - Need to capture a node for debug, but job is failing at early stage - hard to hold a node because issue is random + our keys not pushed till that point.(so even if we hold the node - without infra can't enter node and infra is out today because of recharge day)

* https://bugs.launchpad.net/tripleo/+bug/1986708 - opendev.org ssh failure
* Seems to be affecting all jobs randomly. Could be helpful to enable ansible debug but it's tricky to catch a failing job.
* Doesn't seem to be happening anymore. https://review.opendev.org/c/openstack/project-config/+/853536 proposed by opendev folks to mitigate.

* https://trello.com/c/3p8i2YdZ/2639-cixlp1982874tripleociproa-testcreateobjectwithtransferencoding-is-failing-on-tripleo-jobs
* Christian is out and will be after next week, we will share node with him once he is back.

---

### New bug:
* https://bugzilla.redhat.com/show_bug.cgi?id=2119349
Component and Integration line jobs intermittently failing with retry_limit

* https://bugs.launchpad.net/tripleo/+bug/1986960
Wallaby Ceph based scenarios(SC01/SC04/SC010) are failing with "ERROR: Container release quincy != cephadm release pacific

#### 18th Aug

Check/Gate

following in recheck

* https://review.opendev.org/c/openstack/tripleo-common/+/848597/
* https://review.opendev.org/c/openstack/python-tripleoclient/+/852645/

ssh error seen in one more job:-
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_649/852532/5/gate/tripleo-ci-centos-9-content-provider/649da27/job-output.txt
~~~
2022-08-18 01:14:06.982440 | primary | failed: [undercloud] (item=quay.io/prometheus/node-exporter:v1.3.1) => {"ansible_loop_var": "item", "item": "quay.io/prometheus/node-exporter:v1.3.1", "msg": "Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host\r\nConnection closed by 127.0.0.2 port 22", "unreachable": true}
~~~

---
#### 17th Aug

#### New Bugs
https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.

* Xek proposed: https://review.opendev.org/853478
* ipa job running in check itself - need to work with security team to continue debug/fixing.

#### Check/gate

Below patches in gate failed with https://bugs.launchpad.net/tripleo/+bug/1986708 - random ssh issue , All below patches are in recheck.

https://review.opendev.org/c/openstack/python-tripleoclient/+/852645/
https://review.opendev.org/c/openstack/tripleo-ansible/+/852513/
https://review.opendev.org/c/openstack/tripleo-common/+/848597/

#### Master - last promotion - 17th (Green)

Tracking today's hash here: https://review.rdoproject.org/r/c/testproject/+/38348

#### Master - component
* security - fs039/ipa - bug reported - https://bugs.launchpad.net/tripleo/+bug/1986755

#### Wallaby c9 - last promotion - 17th

Tracking today's hash here:
https://review.rdoproject.org/r/c/testproject/+/39357

#### Wallaby c9 components
* security - ipa/fs039 - bug reported - https://bugs.launchpad.net/tripleo/+bug/1986755

rerunning here: https://review.rdoproject.org/r/c/testproject/+/44516

#### Wallaby c8 - last promotion - 15th - Green
No new hash when I checked in morning.

#### Wallaby c8 components - All green

#### Train c8 - last promotion - 16th - Green
No new hash when I checked in morning.

#### Train c8 components - All green

#### 16.2/8 - promoted 17th Aug - All green

#### 16.2/8 - component - Green now

#### 17/9 - promoted 17th Aug - All green

#### 17/9 - component - All green now

#### 17.1/9 - promoted 17th Aug - All green

#### 17/8
Rerunning failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/403445

#### 17/8 component line

* cloudops
* tripleo - ovb

Rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/424969

---
16th Aug

#### Master - last promotion - 12th

* Tracking here: fs39 remaining
* https://review.rdoproject.org/r/c/testproject/+/38348
* ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/201382~~

#### Master - component
* baremetal - fs001
* security - many jobs failing(line running right now)

Didn't debug, blank recheck first here: https://review.rdoproject.org/r/c/testproject/+/28446

#### Wallaby c9 - last promotion - 12th
* Tracking here: fs035 left
* https://review.rdoproject.org/r/c/testproject/+/44516
* ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/209874~~

#### Wallaby c9 - component
* baremetal - ovb failing
* tripleo - sc04/ovb failing
* security - ipa/fs039 failing

Didn't debug, blank recheck first here:
https://review.rdoproject.org/r/c/testproject/+/42657

#### Wallaby c8 - last promotion - 15th(Green)
#### Wallaby c8 component line - green

#### Train C8 - Last promotion - 16th(Green)
#### Train c8 component line - green

#### Downstream

We changed private network - wohoo passed earlier metadata issue.

https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/424881

#### 17/9
* bm fs001 failed
* https://code.engineering.redhat.com/gerrit/c/testproject/+/424969

#### 16.2/8
* No new hash for integration line

#### 16.2/8 components
* baremetal - ovb
https://code.engineering.redhat.com/gerrit/c/testproject/+/209874
---

15th Aug

## New bugs

~~https://bugs.launchpad.net/tripleo/+bug/1986502~~ - master/wallaby/train lines
are impacted with node_failure, started on 13th Aug, 2022

~~https://bugs.launchpad.net/tripleo/+bug/1985981~~ - standalone job failing with Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

## Check/Gate

### Promotion status

## Master - 12th Aug
Blocked on https://bugs.launchpad.net/tripleo/+bug/1985981 and https://bugs.launchpad.net/tripleo/+bug/1986502

trying to fix catatonit issue via https://review.rdoproject.org/r/c/testproject/+/44527

## Wallaby C9 - 12th Aug

https://bugs.launchpad.net/tripleo/+bug/1986502 - master/wallaby/train lines
are impacted with node_failure, started on 13th Aug, 2022

## Wallaby C8 - 15th Aug

~~chasing promotion here: https://review.rdoproject.org/r/c/testproject/+/44522~~

## Train - 13th Aug

chasing promotion here: https://review.rdoproject.org/r/c/testproject/+/44526

## Downstream
Explored possibility of enabling config-drive in downstream but need some info from infra first
* Left comment here: https://trello.com/c/6ZWMl7pM/2662-cixbz2116287osp162osp17no-promotions-occurs-due-to-nodefailure-at-downstream-on-weekend-since-7th-august

12th Aug

## New bugs

https://bugs.launchpad.net/tripleo/+bug/1985981 - Sc010 kvm internal job failing with Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

Gate blocker

* ~~https://bugs.launchpad.net/tripleo/+bug/1984175/~~
* Mirror came back in sync:-
* https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html

Master promotion blocker
~~https://bugs.launchpad.net/tripleo/+bug/1984184~~
* Fix: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790 merged
* fs035 passed with above fix - https://review.rdoproject.org/r/c/testproject/+/36254

* ~~https://bugs.launchpad.net/tripleo/+bug/1984453~~
* ~~Same mirror issue - rekicked fs002 via testproject to confirm green run https://review.rdoproject.org/r/c/testproject/+/28537~~

## Check/Gate

* https://5e6d48fe25fecf1b2eda-cc994f454b3da94a07b55a097da7db60.ssl.cf5.rackcdn.com/852880/3/check/openstack-tox-linters/0abde65/job-output.txt

~~~
2022-08-11 15:29:01.824649 | ubuntu-focal | TypeError: 'NoneType' object is not iterable
2022-08-11 15:29:01.824656 | ubuntu-focal | error: Support for editable installs via PEP 660 was recently introduced
2022-08-11 15:29:01.824668 | ubuntu-focal | in `setuptools`. If you are seeing this error, please report to:
2022-08-11 15:29:01.824676 | ubuntu-focal |
2022-08-11 15:29:01.824683 | ubuntu-focal | https://github.com/pypa/setuptools/issues
2022-08-11 15:29:01.824690 | ubuntu-focal |
2022-08-11 15:29:01.824697 | ubuntu-focal | Meanwhile you can try the legacy behavior by setting an
2022-08-11 15:29:01.824704 | ubuntu-focal | environment variable and trying to install again:
2022-08-11 15:29:01.824711 | ubuntu-focal |
2022-08-11 15:29:01.824718 | ubuntu-focal | SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
2022-08-11 15:29:01.824725 | ubuntu-focal | [end of output]
~~~

Seen last night - rlandy updated tox.ini in undercloud non-voting patch, but tox-py jobs were getting timedout.

ysandeep have removed tox.ini changes and linters passed in current run.(not seeing the issue anymore)

### Promotion status

# Master - 01st Aug (11 days old)
* blocked till tht merges: https://review.rdoproject.org/r/c/testproject/+/36254 (+wed)
* fs035 passed with this patch in depends-on

* tracking two hashes with depends on above patch
* 4e352c3ada5a2e91b161ff220dc42d85 : https://review.rdoproject.org/r/c/testproject/+/38348
* Need to skip internal sc10 kvm job - bug reported(as same job passing in vexx I think safe to skip and promote if 035/34 passes)
* 120b06809f1abde369d06cef81ded6ab: https://review.rdoproject.org/r/c/testproject/+/36254
* just waiting for 002

# Wallaby C9 - 09th Aug (03 days old)
* 120b06809f1abde369d06cef81ded6ab: fs20 failed
* rerunning here https://review.rdoproject.org/r/c/testproject/+/39357

# Wallaby C8 - 11th Aug (Green)
# Train - 09th Aug
* tracking here: https://review.rdoproject.org/r/c/testproject/+/44516

# Downstream
Explored possibility of enabling config-drive in downstream but need some info from infra first
* Left comment here: https://trello.com/c/6ZWMl7pM/2662-cixbz2116287osp162osp17no-promotions-occurs-due-to-nodefailure-at-downstream-on-weekend-since-7th-august