Ruck Rover 2023-01-06 - 2023-01-12

RDO Cockpit / RHOS Cockpit RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

Review List

tags: `ruck_rover`

Previous RR notes: https://hackmd.io/CeFztoYvSqO-nwjxNYtKhQ

ruck & rover: Marios and Sandeep

NOTE: Ongoing bugs

Check/Gate blockers

~~https://bugs.launchpad.net/tripleo/+bug/2002469~~ Master: Node provisioning failure with "ironic.common.exception.ConfigInvalid: Auth plugin requires parameters which were not given: auth_url"
~~https://bugs.launchpad.net/tripleo/+bug/2002112~~ openstack-tox-cover/tox-sanity-py39 job failure in openstack/tripleo-repos
~~https://bugs.launchpad.net/tripleo/+bug/2001602~~ tripleo-ansible openstack-tox-linters fails with NoInterpreter
~~https://bugs.launchpad.net/tripleo/+bug/2002035~~ openstack-tox-* are failing in opendev with "env name py38 conflicting with base python python3
https://bugs.launchpad.net/tripleo/+bug/2002526 openstack-tox-linters failure against tripleo-repos

Intergation line blockers

https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops
~~https://bugs.launchpad.net/tripleo/+bug/2001916 periodic standalone sc10 wallaby broken~~
- ~~Fix: https://review.rdoproject.org/r/c/rdo-jobs/+/46536~~
~~https://bugs.launchpad.net/tripleo/+bug/2000226~~ : xfs_growfs: /dev/mapper/vg-lv_srv is not a mounted XFS filesystem
- Workaround in place
https://bugs.launchpad.net/tripleo/+bug/2000061 : CS-9 OVB fs001 tripleo master is unstable

Downstream blockers:

https://bugzilla.redhat.com/show_bug.cgi?id=2154179 periodic-tripleo-rhel-9-8-rhos-17.1-mixed-os-compute-internal failing consistently on task Run container image prepare for compute deploy

12th Jan

New Bug:

https://bugzilla.redhat.com/show_bug.cgi?id=2160392 OSP17.1/Rhel-9 Ceph based standalone jobs failing on Ceph Deploy with error "FATAL | Wait for expected number of osds to be running" cephadm latest package corresponds to 5.3, container tag is still 5.2 - mismatch is causing issue - Ceph team checking internally about right tag for 5.3.

Gate failures: https://review.opendev.org/c/openstack/tripleo-quickstart/+/869820/ - tempest failure and sc000 install failure - under recheck https://review.opendev.org/c/openstack/tripleo-ansible/+/868277/ - content-provier - "Failed to pull following containers: ['quay.io/prometheus/prometheus:v2.18.1', 'quay.io/prometheus/node-exporter:v0.18.1']"} - under recheck

internal jobs getting aborted e.g. https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-kvm-internal-standalone-network-train
- e.g uploaded v119 and then before zuul reported, uploaded v120 https://review.rdoproject.org/r/c/tripleo-downstream-trigger-nested-virt/+/46510/
- might need to tweak the time or have different patch for each branch
  - posted https://review.rdoproject.org/r/c/config/+/46584

Upstream Integration line promotion status

Master CS9 - 2023-01-12
Zed - 2023-01-12
Wallaby c9 - 2023-01-12
Wallaby c8 - 2023-01-12 - Green
Train C8 - 2023-01-12 - Green

Upstream Component line promotion status

Master CS9
- compute
- network
- octavia
- tempest
Zed CS9
- network
  - https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-zed/da346df/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
Wallaby CS9
- network
  - https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby/7bcbfd0/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
  - https://logserver.rdoproject.org/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby/b1555ed/logs/undercloud/var/log/tempest/failing_tests.log.txt.gz
- compute
Wallaby CS8 - All green
Train CS8 - All green
Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/39357
Downstream Integration line promotion status
- RHOS-17.1 RH9 - 2023-01-10 - Green
  - newer hash available
  - Rerrunning here: https://code.engineering.redhat.com/gerrit/c/testproject/+/398540
- RHOS-17.1 RH8 - 2023-01-11 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-11 - Green
  - Jobs running right now, will check back in few hours
Downstream Component line promotion status
- RHOS-17.1 RH9
  - cinder - - sc01/sc04 - failure could be legit - rerunning before opening a bug
    - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario001-standalone-cinder-rhos-17.1/10459c9/job-output.txt
    - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cinder/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario004-standalone-cinder-rhos-17.1/f45d8ac/job-output.txt
  - network - ovb
    - introspection failure: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-network/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-network-rhos-17.1/8df040a/job-output.txt
- RHOS-17.1 RH8 - Green - no component jobs
- RHOS-17 RH9
- RHOS-16-2 RH8 - All green
- Rerunning failed jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/211643

11th Jan

gate

https://review.opendev.org/c/openstack/tripleo-ansible/+/869360/ - tox-py310 failure - ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/v2/collections/community/crypto/versions/2.3.2/': The read operation timed out - rechecked -> passed on recheck

please check https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-linters&project=openstack/tripleo-quickstart
- we have 2 there (otherwise/other projects green) lets see if recheck still hits https://review.opendev.org/c/openstack/tripleo-quickstart/+/865238
- reported: https://bugs.launchpad.net/tripleo/+bug/2002526

New Bugs

https://bugs.launchpad.net/tripleo/+bug/2002469 Master: Node provisioning failure with "ironic.common.exception.ConfigInvalid: Auth plugin requires parameters which were not given: auth_url"
https://bugs.launchpad.net/tripleo/+bug/2002478 centos9 wallaby scenario 10 fails octavia tempest test_traffic_ops
https://bugs.launchpad.net/tripleo/+bug/2002526 openstack-tox-linters failure against tripleo-repos

Upstream Integration line promotion status

Master CS9 - 2023-01-10 - Green
- New hash available
  - All ovb jobs failing - reported https://bugs.launchpad.net/tripleo/+bug/2002469
  - sc07/ovn-provider - rerunning here to confirm if consistent failure: https://review.rdoproject.org/r/c/testproject/+/42657
Zed CS9 - 2023-01-11 - Green
Wallaby CS9 - 2023-01-11 - Green
- Intermittent failure for sc10
  - ovn-provider: tempest failure - same test failed - Bug reported: https://bugs.launchpad.net/tripleo/+bug/2002478
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
Wallaby CS8 - 2023-01-11 - Green
Train CS8 - 2023-01-09 - Green
- Newer hash available
  - fs035/fs039
    - wrong fs type, bad option, bad superblock on /dev/nbd0, missing codepage or helper program, or other error.",
  - ~~full-tempest-api : tempest failure~~
  - Rerunning here: https://review.rdoproject.org/r/c/testproject/+/28537

Upstream Component line promotion status

Master CS9
- baremetal - same ovb bug as integration line
- network - same ovb bug as integration line
- tripleo - same ovb bug as integration line
Zed CS9 - All green
Wallaby CS9 - All green
Wallaby CS8 - All green
- Train CS8 - All green
Downstream Integration line promotion status
- RHOS-17.1 RH9 - 2023-01-10 - Green
- RHOS-17.1 RH8 - 2023-01-10 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-11 - Green
Downstream Component line promotion status
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - Green - no component jobs
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All green

10th Jan

Gate

https://review.opendev.org/c/openstack/tripleo-ansible/+/865620/ - random ubi image pull issue - in recheck

Upstream Integration line promotion status

Master CS9 - 2023-01-10 - Green
Zed CS9 - 2023-01-09 - Green
Wallaby CS9 - 2023-01-09 - Green
Wallaby CS8 - 2023-01-10 - Green
Train CS8 - 2023-01-09 - Green

Upstream Component line promotion status

Master:
- Baremetal
Zed: All Green
Wallaby C9: All Green
Wallaby C8 - All Green
Train C8
- network - internal kvm job - mirror issue
- Why we are using Vexxhost mirror for job running in internal? Need to debug further
Downstream Integration line promotion status
- RHOS-17.1 RH9 - 2023-01-10 - Green - ~~ovb fs-20/fs-20-rbac/fs-35 failed with retry_limit~~
  - ~~Changed the network we use for ovb: https://review.rdoproject.org/r/c/config/+/46549~~
- RHOS-17.1 RH8 - 2023-01-07 - Green
- RHOS-17 RH9 - 2023-01-09 - Green
- RHOS-16-2 RH8 - 2023-01-09 - Green
Downstream Component line promotion status
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - we don't have component jobs in this line.
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All Green

09th Jan

Gate failures

https://review.opendev.org/c/openstack/tripleo-quickstart/+/864167/ - tripleo-tox-molecule - timed out - rechecked as single failure

Upstream Integration line promotion status

Master CS9 - 2023-01-08 - Green
Zed CS9 - 2023-01-08 - newer hash available
- fs039 - tempest failure
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/42657
Wallaby CS9 - 2023-01-08 - Green
Wallaby CS8 - 2023-01-07 - Green
Train CS8 - 2023-01-06- Yellow
- fs001
- fs039
- Rerunning here: https://review.rdoproject.org/r/c/testproject/+/39420
- also need c-8-scenario010-kvm-internal-standalone-train 4defb2bf4d2d5225896e23af590… https://code.engineering.redhat.com/gerrit/c/testproject/+/438475

Upstream Component line promotion status

Master: All green
Zed - All green
Wallaby C9 - All green
Wallaby C8 - All green
Train C8 - All green
Downstream Integration line promotion status
- RHOS-17.1 RH9 - 2023-01-08 - Green
- RHOS-17.1 RH8 - 2023-01-07 - Green
- RHOS-17 RH9 - 2023-01-04 - Yellow
  - Bm job - consistent failure on Node provisioning -
    - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/6ed8de8/logs/undercloud/home/zuul/overcloud_node_provision.log
    - https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17/3f09f34/logs/undercloud/home/zuul/overcloud_node_provision.log
    - https://privatebin.corp.redhat.com/?64c5d4cfdd88a141#C6WUL9dqhSH6bTRUKuhG7Fo97CgCHecWPwYSdxKchx44
    - 1 node failed to provision - Failed command: WRITE DMA EXT - Looks like issue with Hard disk.
    - created cix directly no BZ https://trello.com/c/O7jqHzjK
- RHOS-16-2 RH8 - 2023-01-08 - Green
  - Newer hash running right now - Bm job still pending - Will check back in few hours
- Downstream Component line promotion status
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - We don't have component jobs anymore
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 - All green

06th Jan

New bugs:

https://bugs.launchpad.net/tripleo/+bug/2002112 openstack-tox-cover/tox-sanity-py39 job failure in openstack/tripleo-repos

Gate

https://review.opendev.org/c/openstack/tripleo-ansible/+/865136/ tox failures in wallaby
- also tht/wallaby https://zuul.opendev.org/t/openstack/build/253f20d226f948a29f18e7a8cd02f84a rabi fixing with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/868464

Upstream Integration line promotion status

Master CS9 - 2023-01-06 - Green
Zed CS9 - 2023-01-06 - Green
Wallaby CS9 - 2023-01-06 - Green
Wallaby CS8 - 2023-01-06 - Green
Train CS8 - 2023-01-05- Green

Upstream Component line promotion status

Master: All green
Zed
- network
  - ovb
Wallaby c9 : All green
Wallaby c8 - all green
Train c8 - all green

Rerunning failed jobs here: https://review.rdoproject.org/r/c/testproject/+/28537

Downstream Integration line promotion status
- RHOS-17.1 RH9 - 2023-01-0506 (Green)
- RHOS-17.1 RH8 - 2023-01-0406
  - mixed os job failing (NODE_FAILURE https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-8-multinode-mixed-os-rhos-17.1&project=openstack/tripleo-ci)
  - rerunning ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/438405~~
- RHOS-17 RH9 - 2023-01-04 (Green no new content)
- RHOS-16-2 RH8 - ~~2022-12-25 (Yellow)~~ 2022-01-06
  - Line running right now with new content but lot of node failures
  - Need to ping on #rhos-ops again when more folks are around, pinged once already
  - re-running NODE_FAILURE jobs ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/438408~~
Downstream Component line promotion status
- RHOS-17.1 RH9 - All green
- RHOS-17.1 RH8 - All green - we don't have jobs in component
- RHOS-17 RH9 - All green
- RHOS-16-2 RH8 : All green

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2023-01-06 - 2023-01-12

tags: ruck_rover

Previous RR notes: https://hackmd.io/CeFztoYvSqO-nwjxNYtKhQ

ruck & rover: Marios and Sandeep

NOTE: Ongoing bugs

12th Jan

New Bug:

Upstream Integration line promotion status

Upstream Component line promotion status

11th Jan

gate

New Bugs

Upstream Integration line promotion status

Upstream Component line promotion status

10th Jan

Gate

Upstream Integration line promotion status

Upstream Component line promotion status

09th Jan

Gate failures

Upstream Integration line promotion status

Upstream Component line promotion status

06th Jan

New bugs:

Gate

Upstream Integration line promotion status

Upstream Component line promotion status

tags: `ruck_rover`