Ruck Rover 2023-01-20 - 2023-01-26

RDO Cockpit / RHOS Cockpit RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

Review List

tags: `ruck_rover`

Previous RR notes: https://hackmd.io/juRN31FtQkG8UYwR_bZHSg

ruck & rover: Bhagyashris and Tengu

NOTE: Ongoing bugs

Check/Gate blockers

Intergation line blockers

https://bugs.launchpad.net/tripleo/+bug/2003865 mixed OS wallaby cs8/9 fails tempest network_basic_ops, security_groups
https://bugs.launchpad.net/tripleo/+bug/2003882 unable to configure haproxy container with puppet (probably linked to an update in puppetlabs-haproxy)
https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops
https://bugs.launchpad.net/tripleo/+bug/2000061 : CS-9 OVB fs001 tripleo master is unstable
~~https://bugs.launchpad.net/tripleo/+bug/2003658 playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~

Downstream blockers:

https://bugzilla.redhat.com/show_bug.cgi?id=2154179 periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy

26 Jan

Upstream lines

Promotions

Master: 2023-01-26 (check bellow)
Zed: 2023-01-25
Wallaby (cs9): 2023-01-24
Wallaby (cs8): 2023-01-24
Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5

Issues

Wallaby

tripleo-ci-centos-8-9-multinode-mixed-os started failing today
- https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
- https://bugs.launchpad.net/tripleo/+bug/2003865
- patch is being worked on (check LP)

Master

Network component on Master is apparently broken for ~1 or 2 days:
- https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
- https://bugs.launchpad.net/tripleo/+bug/2003884
- Seems to be due to pyroute2
TripleO component is hosed as well
- https://bugs.launchpad.net/tripleo/+bug/2003882
- Workaround: https://review.rdoproject.org/r/c/rdoinfo/+/46709
  - doesn't seem to work - investigating if the pin patch was correct or if anything more is needed. CIX card up-to-date.
- Proper fix: https://review.opendev.org/c/openstack/puppet-tripleo/+/871807
- testproject: https://review.rdoproject.org/r/c/testproject/+/46701
  - checks if pin is applied (for now)

Downstream lines

25 Jan

Upstream lines

Promotions

Master: 2023-01-24
Zed: 2023-01-23
Wallaby (cs9): 2023-01-24
Wallaby (cs8): 2023-01-24
Train: 2023-01-20 (no new content) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5

Issues

tripleo-ci-centos-8-9-multinode-mixed-os started failing today
- https://zuul.opendev.org/t/openstack/builds?job_name=tripleo-ci-centos-8-9-multinode-mixed-os
- https://bugs.launchpad.net/tripleo/+bug/2003865
- patch is being worked on (check LP)
Network component on Master is apparently broken for ~1 or 2 days:
- https://review.rdoproject.org/r/c/testproject/+/46701 (attempted re-kick)
- http://dashboard-ci.tripleo.org/d/WWV51gO4k/component-pipeline?orgId=1&viewPanel=478
TripleO component is hosed as well
- https://bugs.launchpad.net/tripleo/+bug/2003882
- probably due to puppetlabs-haproxy patch (check LP comments)

Downstream promotion

RHOS-18 RH9 - 2023-01-24
RHOS-17.1 RH9 - 2023-01-15
- running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803 [DNM] Test rhos17.1 on rhel9 jobs
- jenkins job failure:
  - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
  - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
    - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
RHOS-17.1 RH8 - 2023-01-24
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
RHOS-17 RH9 - 2023-01-23
- running failing job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431169 [DNM] Test rhos17 on rhel9 failing job
RHOS-16-2 RH8 - 2023-01-25

24 Jan

Upstream lines

Promotions

Master: 2023-01-23
Zed: 2023-01-23
Wallaby (cs9): ~~2023-01-19~~ 2023-01-24
Wallaby (cs8): 2023-01-24
Train: 2023-01-20 (no new content ) https://trunk.rdoproject.org/centos8-train/tripleo-ci-testing/delorean.repo.md5 vs https://trunk.rdoproject.org/centos8-train/current-tripleo/delorean.repo.md5

Issues/blockers

re-kicked missing promotion job for wallaby on cs9 (5 days old):
- https://review.rdoproject.org/r/c/testproject/+/46698
- Should promote shortly. DONE
master promotion: 2 jobs are failing on tempest, apparently every time. Need investigations.
- periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master: issue with TestNetworkBasicOps, leading to an API timeout on actions
- periodic-tripleo-ci-centos-9-ovb-1ctlr_2comp-featureset020-master: cinder snapshot test, hitting some content format issue (?)
- Re-kicked failed jobs:
  - https://review.rdoproject.org/r/c/testproject/+/46701

Downstream promotion

RHOS-18 RH9 - 2023-01-21
RHOS-17.1 RH9 - 2023-01-15
- jenkins job failure:
  - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
  - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
    - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
RHOS-17.1 RH8 - 2023-01-22
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
RHOS-17 RH9 - 2023-01-23
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
RHOS-16-2 RH8 - 2023-01-21
- No new content

23 Jan

Upstream lines

Promotions

Master: 20/01/2023
Zed: 20/01/2023
Wallaby (cs9): 19/01/2023
Wallaby (cs8): 21/01/2023
Train: 20/01/2022

Issues/blockers

~~all upstream hosed, https://bugs.launchpad.net/tripleo/+bug/2003658 playbooks/tripleo-rdo-base/container-login.yaml is failing on all RDO jobs~~
- Re-kicked master pipeline (openstack-periodic-integration-main) in order to ensure the above issue is really solved
  - we're good! Promotion should happen shortly.

Downstream promotion

RHOS-18 RH9 - 2023-01-21
RHOS-17.1 RH9 - 2023-01-15
- jenkins job failure:
  - pinged Attila on rhos-ops <bhagyashris> afazekas, hey jenkins integration rhos17.1 on rhel9 jobs are failing which are blocking the promotions. could you plz check
  - https://bugzilla.redhat.com/show_bug.cgi?id=2162653
    - https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438871 Temp remove jenkins jobs from criteria for BZ fix
RHOS-17.1 RH8 - 2023-01-22
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
RHOS-17 RH9 - 2023-01-23
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN
RHOS-16-2 RH8 - 2023-01-21
- run failed jobs on testproject here https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/7#message-a2caa6efd850df89090db1a610ece3a5c0d29109 GREEN

20 Jan

Upstream promotion

Master CS9 - 2023-01-19
Zed - 2023-01-18
Wallaby c9 - 2023-01-18
Wallaby c8 - 2023-01-19
Train C8 - 2023-01-18
- 10:54 < Tengu|rover> marios, bhagyashris|ruck https://review.rdoproject.org/r/c/testproject/+/46686 train re-run

Downstream promotion

RHOS-17.1 RH9 - 2023-01-15
- periodic-tripleo-rhel-9-upload-job-trigger-rhos-17.1 have passed
- blocked on https://bugzilla.redhat.com/show_bug.cgi?id=2161501
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
RHOS-17.1 RH8 - 2023-01-14
- https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/438865 Temp remove mixed rhel from criteria for BZ fix
RHOS-17 RH9 - 2023-01-19
RHOS-16-2 RH8 - 2023-01-19
- https://code.engineering.redhat.com/gerrit/c/testproject/+/431140

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2023-01-20 - 2023-01-26

tags: ruck_rover

Previous RR notes: https://hackmd.io/juRN31FtQkG8UYwR_bZHSg

ruck & rover: Bhagyashris and Tengu

NOTE: Ongoing bugs

26 Jan

Upstream lines

Promotions

Issues

Wallaby

Master

Downstream lines

25 Jan

Upstream lines

Promotions

Issues

Downstream promotion

24 Jan

Upstream lines

Promotions

Issues/blockers

Downstream promotion

23 Jan

Upstream lines

Promotions

Issues/blockers

Downstream promotion

20 Jan

Upstream promotion

Downstream promotion

tags: `ruck_rover`