Ruck Rover 2022-08-12 to 2022-08-18

tags: `ruck_rover`

Ongoing Bugs

Blocking:

Wallaby Check/Gate

https://bugs.launchpad.net/tripleo/+bug/1986960 Wallaby Ceph based scenarios(SC01/SC04/SC010) are failing with "ERROR: Container release quincy != cephadm release pacific
https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.
- Xek proposed: https://review.opendev.org/853478
- ipa job running in check itself - need to work with security team to continue debug/fixing.

Degraded:

https://bugzilla.redhat.com/show_bug.cgi?id=2119349 -Component and Integration line jobs intermittently failing with retry_limit

https://bugs.launchpad.net/tripleo/+bug/1983817 - Jobs failing with retry
- Still happening - Need to capture a node for debug, but job is failing at early stage - hard to hold a node because issue is random + our keys not pushed till that point.(so even if we hold the node - without infra can't enter node and infra is out today because of recharge day)
https://bugs.launchpad.net/tripleo/+bug/1986708 - opendev.org ssh failure
- Seems to be affecting all jobs randomly. Could be helpful to enable ansible debug but it's tricky to catch a failing job.
- Doesn't seem to be happening anymore. https://review.opendev.org/c/openstack/project-config/+/853536 proposed by opendev folks to mitigate.
https://trello.com/c/3p8i2YdZ/2639-cixlp1982874tripleociproa-testcreateobjectwithtransferencoding-is-failing-on-tripleo-jobs
- Christian is out and will be after next week, we will share node with him once he is back.

New bug:

https://bugzilla.redhat.com/show_bug.cgi?id=2119349 Component and Integration line jobs intermittently failing with retry_limit
https://bugs.launchpad.net/tripleo/+bug/1986960 Wallaby Ceph based scenarios(SC01/SC04/SC010) are failing with "ERROR: Container release quincy != cephadm release pacific

18th Aug

Check/Gate

following in recheck

ssh error seen in one more job:- https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_649/852532/5/gate/tripleo-ci-centos-9-content-provider/649da27/job-output.txt

2022-08-18 01:14:06.982440 | primary | failed: [undercloud] (item=quay.io/prometheus/node-exporter:v1.3.1) => {"ansible_loop_var": "item", "item": "quay.io/prometheus/node-exporter:v1.3.1", "msg": "Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host\r\nConnection closed by 127.0.0.2 port 22", "unreachable": true}

17th Aug

New Bugs

https://bugs.launchpad.net/tripleo/+bug/1986755 Master/wallaby Security component "Standalone with ipa" and fs039 job are failing with ERROR! couldn't resolve module/action 'freeipa.ansible_freeipa.ipahost'. This often indicates a misspelling, missing collection, or incorrect module path.

Xek proposed: https://review.opendev.org/853478
ipa job running in check itself - need to work with security team to continue debug/fixing.

Check/gate

Below patches in gate failed with https://bugs.launchpad.net/tripleo/+bug/1986708 - random ssh issue , All below patches are in recheck.

https://review.opendev.org/c/openstack/python-tripleoclient/+/852645/ https://review.opendev.org/c/openstack/tripleo-ansible/+/852513/ https://review.opendev.org/c/openstack/tripleo-common/+/848597/

Wallaby c8 - last promotion - 15th - Green

No new hash when I checked in morning.

Wallaby c8 components - All green

Train c8 - last promotion - 16th - Green

No new hash when I checked in morning.

Train c8 components - All green

16.2/8 - promoted 17th Aug - All green

16.2/8 - component - Green now

17/9 - promoted 17th Aug - All green

17/9 - component - All green now

17.1/9 - promoted 17th Aug - All green

17/8

Rerunning failing jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/403445

17/8 component line

cloudops
tripleo - ovb

Rerunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/424969

16th Aug

Master - last promotion - 12th

Tracking here: fs39 remaining
- https://review.rdoproject.org/r/c/testproject/+/38348
- ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/201382~~

Master - component

baremetal - fs001
security - many jobs failing(line running right now)

Didn't debug, blank recheck first here: https://review.rdoproject.org/r/c/testproject/+/28446

Wallaby c9 - last promotion - 12th

Tracking here: fs035 left
- https://review.rdoproject.org/r/c/testproject/+/44516
- ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/209874~~

Wallaby c9 - component

baremetal - ovb failing
tripleo - sc04/ovb failing
security - ipa/fs039 failing

Didn't debug, blank recheck first here: https://review.rdoproject.org/r/c/testproject/+/42657

Wallaby c8 - last promotion - 15th(Green)

Wallaby c8 component line - green

Train C8 - Last promotion - 16th(Green)

Train c8 component line - green

Downstream

We changed private network - wohoo passed earlier metadata issue.

https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/424881

17/9

bm fs001 failed
- https://code.engineering.redhat.com/gerrit/c/testproject/+/424969

16.2/8

No new hash for integration line

16.2/8 components

baremetal - ovb https://code.engineering.redhat.com/gerrit/c/testproject/+/209874

15th Aug

New bugs

~~https://bugs.launchpad.net/tripleo/+bug/1986502~~ - master/wallaby/train lines are impacted with node_failure, started on 13th Aug, 2022

~~https://bugs.launchpad.net/tripleo/+bug/1985981~~ - standalone job failing with Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

Check/Gate

Promotion status

Master - 12th Aug

Blocked on https://bugs.launchpad.net/tripleo/+bug/1985981 and https://bugs.launchpad.net/tripleo/+bug/1986502

trying to fix catatonit issue via https://review.rdoproject.org/r/c/testproject/+/44527

Wallaby C9 - 12th Aug

https://bugs.launchpad.net/tripleo/+bug/1986502 - master/wallaby/train lines are impacted with node_failure, started on 13th Aug, 2022

Wallaby C8 - 15th Aug

~~chasing promotion here: https://review.rdoproject.org/r/c/testproject/+/44522~~

Train - 13th Aug

chasing promotion here: https://review.rdoproject.org/r/c/testproject/+/44526

Downstream

Explored possibility of enabling config-drive in downstream but need some info from infra first

Left comment here: https://trello.com/c/6ZWMl7pM/2662-cixbz2116287osp162osp17no-promotions-occurs-due-to-nodefailure-at-downstream-on-weekend-since-7th-august

12th Aug

New bugs

https://bugs.launchpad.net/tripleo/+bug/1985981 - Sc010 kvm internal job failing with Error: container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

Gate blocker

~~https://bugs.launchpad.net/tripleo/+bug/1984175/~~
- Mirror came back in sync:-
  - https://lists.centos.org/pipermail/centos-devel/2022-August/120525.html

Master promotion blocker ~~https://bugs.launchpad.net/tripleo/+bug/1984184~~

Fix: https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852790 merged * fs035 passed with above fix - https://review.rdoproject.org/r/c/testproject/+/36254
~~https://bugs.launchpad.net/tripleo/+bug/1984453~~
- ~~Same mirror issue - rekicked fs002 via testproject to confirm green run https://review.rdoproject.org/r/c/testproject/+/28537~~

Check/Gate

https://5e6d48fe25fecf1b2eda-cc994f454b3da94a07b55a097da7db60.ssl.cf5.rackcdn.com/852880/3/check/openstack-tox-linters/0abde65/job-output.txt

2022-08-11 15:29:01.824649 | ubuntu-focal |       TypeError: 'NoneType' object is not iterable
2022-08-11 15:29:01.824656 | ubuntu-focal |       error: Support for editable installs via PEP 660 was recently introduced
2022-08-11 15:29:01.824668 | ubuntu-focal |       in `setuptools`. If you are seeing this error, please report to:
2022-08-11 15:29:01.824676 | ubuntu-focal |
2022-08-11 15:29:01.824683 | ubuntu-focal |       https://github.com/pypa/setuptools/issues
2022-08-11 15:29:01.824690 | ubuntu-focal |
2022-08-11 15:29:01.824697 | ubuntu-focal |       Meanwhile you can try the legacy behavior by setting an
2022-08-11 15:29:01.824704 | ubuntu-focal |       environment variable and trying to install again:
2022-08-11 15:29:01.824711 | ubuntu-focal |
2022-08-11 15:29:01.824718 | ubuntu-focal |       SETUPTOOLS_ENABLE_FEATURES="legacy-editable"
2022-08-11 15:29:01.824725 | ubuntu-focal |       [end of output]

Seen last night - rlandy updated tox.ini in undercloud non-voting patch, but tox-py jobs were getting timedout.

ysandeep have removed tox.ini changes and linters passed in current run.(not seeing the issue anymore)

Promotion status

Master - 01st Aug (11 days old)

blocked till tht merges: https://review.rdoproject.org/r/c/testproject/+/36254 (+wed)
- fs035 passed with this patch in depends-on
tracking two hashes with depends on above patch
- 4e352c3ada5a2e91b161ff220dc42d85 : https://review.rdoproject.org/r/c/testproject/+/38348
  - Need to skip internal sc10 kvm job - bug reported(as same job passing in vexx I think safe to skip and promote if 035/34 passes)
- 120b06809f1abde369d06cef81ded6ab: https://review.rdoproject.org/r/c/testproject/+/36254
  - just waiting for 002

Wallaby C9 - 09th Aug (03 days old)

120b06809f1abde369d06cef81ded6ab: fs20 failed
- rerunning here https://review.rdoproject.org/r/c/testproject/+/39357

Wallaby C8 - 11th Aug (Green)

Train - 09th Aug

* tracking here: https://review.rdoproject.org/r/c/testproject/+/44516

Downstream

Explored possibility of enabling config-drive in downstream but need some info from infra first

Left comment here: https://trello.com/c/6ZWMl7pM/2662-cixbz2116287osp162osp17no-promotions-occurs-due-to-nodefailure-at-downstream-on-weekend-since-7th-august

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2022-08-12 to 2022-08-18

tags: ruck_rover

Previous RR notes: https://hackmd.io/5zEei6JoTSmZ9mXlN5bxow

New bug:

18th Aug

17th Aug

New Bugs

Check/gate

Master - last promotion - 17th (Green)

Master - component

Wallaby c9 - last promotion - 17th

Wallaby c9 components

Wallaby c8 - last promotion - 15th - Green

Wallaby c8 components - All green

Train c8 - last promotion - 16th - Green

Train c8 components - All green

16.2/8 - promoted 17th Aug - All green

16.2/8 - component - Green now

17/9 - promoted 17th Aug - All green

17/9 - component - All green now

17.1/9 - promoted 17th Aug - All green

17/8

17/8 component line

Master - last promotion - 12th

Master - component

Wallaby c9 - last promotion - 12th

Wallaby c9 - component

Wallaby c8 - last promotion - 15th(Green)

Wallaby c8 component line - green

Train C8 - Last promotion - 16th(Green)

Train c8 component line - green

Downstream

17/9

16.2/8

16.2/8 components

New bugs

Check/Gate

Promotion status

Master - 12th Aug

Wallaby C9 - 12th Aug

Wallaby C8 - 15th Aug

Train - 13th Aug

Downstream

New bugs

Check/Gate

Promotion status

Master - 01st Aug (11 days old)

Wallaby C9 - 09th Aug (03 days old)

Wallaby C8 - 11th Aug (Green)

Train - 09th Aug

Downstream

tags: `ruck_rover`