Ruck Rover 2022-09-01 to 2022-09-01

tags: `ruck_rover`

Next RR notes: https://hackmd.io/0RguTgCAQsiZz_SnF5209A

Previous RR notes: https://hackmd.io/7qAKWCiCQA6IEdE9WXYn4Q

ruck: Pooja, rover: Jakob

RDO Cockpit / RHOS Cockpit

RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

2022-09-01

Reruns and Investigations:

(crossed out := known bugs recorded above)

NOTE: Watch for running testproject jobs on https://review.rdoproject.org/zuul/status and https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

Promotions
- C9 Master
  - last_promotion=2022-08-31
  - periodic-tripleo-ci-build-containers-ubi-9-quay-push-master
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64, might be LP#1988397 or related to LP#1985981
  - periodic-tripleo-ci-build-containers-centos-9-push-master
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64, might be LP#1988397 or related to LP#1985981
    - <ysandeep> fyi.. there was a bug related to catatonit few days back: https://bugs.launchpad.net/tripleo/+bug/1985981 <ysandeep> https://review.opendev.org/c/openstack/tripleo-quickstart/+/853142 was temp workaround
  - reruns
- C9 Wallaby
  - last_promotion=2022-08-29
  - periodic-tripleo-ci-build-containers-ubi-9-quay-push-wallaby
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64, might be LP#1988397 or related to LP#1985981
  - periodic-tripleo-ci-build-containers-centos-9-push-wallaby
    - package podman-2:4.2.0-3.el9.x86_64 conflicts with catatonit provided by catatonit-3:0.1.7-7.el9.x86_64, might be LP#1988397 or related to LP#1985981
  - yesterdays reruns still in progress
- C8 Wallaby
  - last_promotion=2022-08-31
  - check-rdo/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-wallaby:
    - overcloud inspection failed with FATAL | ... No nodes are manageable at this time
- C8 Train
  - last_promotion=2022-08-30
  - missing fs001 to promote - rerun in https://review.rdoproject.org/r/c/testproject/+/44662
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-train
    - overcloud deploy failed with FATAL | Wait for containers to start for step 3 using paunch
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train
    - overcloud deploy failed with FATAL | Wait for containers to start for step 3 using paunch
    - LP#1938987
  - periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-train
    - tempest failures
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
    - tempest failures
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-internal-train
    - TASK [modify-image : Mount image]
  - reruns
- OSP 17.1 2022-08-29 (2 days out)
  - reruns
- OSP 17 2022-08-28 (3 days out)
  - periodic-tripleo-ci-rhel-9-bm_envB-3ctlr_1comp-featureset001-rhos-17 No space left on device
  - Blocked on OVBs Instance unable to reach metadata
    - Sandeep's Ticket
- OSP 16.2 : 2022-08-29 (2 days out)
  - Bring back to prodchain blocked CIX card
    - rerun
  - Blocked on OVBs Instance unable to reach metadata
    - Sandeep's Ticket
C9 Master:
- clients
  - last_promotion=2022-08-31
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-clients-master
    - FATAL | Create containers managed by Podman for ...
    - FATAL | Create containers managed by Podman for ...
- common
  - last_promotion=2022-08-31
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-common-master
    - probably LP#1988397
- network
  - last_promotion=2022-08-31
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-undercloud-upgrade-network-master
    - probably LP#1988397
- security
  - last_promotion=2022-08-30
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master
    - LP#1988347
- reruns
C9 Wallaby:
- common
  - last_promotion=2022-08-10
  - no new content?
  - periodic-tripleo-ci-centos-8-9-multinode-mixed-os-common-wallaby
    - TODO
  - periodic-tripleo-ci-centos-9-8-multinode-mixed-os-common-wallaby
    - TODO
- network
  - last_promotion=2022-08-30
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-standalone-full-tempest-api-network-wallaby
    - tempest failures
  - periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-network-wallaby
    - other tempest failures
- tripleo
  - last_promotion=2022-08-31
  - tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-tripleo-wallaby
    - FATAL | Create containers managed by Podman for ...
- reruns
C8 Wallaby:
- network
  - last_promotion=2022-08-30
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-network-wallaby
    - overcloud deploy failed with ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: ...deploy_steps_playbook.yaml ... timeout, Return Code: 254
- reruns
C8 Train:
- tripleo
  - last_promotion=2022-08-26
  - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-tripleo-train
    - tempest failures
- reruns
RHEL9 OSP17:
- tripleo component lagging with :
  - periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-tripleo-rhos-17 Find out UUID of instance with metadata URL
  - Rest all components are up to date
RHEL8 OSP16.2:
- network component is lagging with :
  - periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-network-rhos-16.2
  - Rest all components are up to date

Known Bugs

https://bugs.launchpad.net/tripleo/+bug/1960355
- Tempest tests test_oauth1_tokens.OAUTH1TokensTest are failing on tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-(master|wallaby)
- affecting OSP17.1 as well
https://bugs.launchpad.net/tripleo/+bug/1987092
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
BLOCKER
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
- Pacemaker performance causes intermittent galera issues in loaded CI env
- jobs fail on Check Keystone public endpoint status or in other places
- zuul/builds
- affecting non-validation jobs such as periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby
https://bugs.launchpad.net/tripleo/+bug/1987632
- cs9 fs01 check job failing on node_provisioning with msg - ""msg": "timed out waiting for ping module test: Data could not be sent to remote host"
- zuul/builds
- lot of passes on wednesday and earlier
- atm only affecting check jobs
- might be an issue with host virt environments, i.e. the cloud provider
https://bugzilla.redhat.com/2105408
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
BLOCKER
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
- RHOSP16.2 - Octavia tempest test "test_load_balancer_ipv4_CRUD" fails due to a kernel stacktrace (in a nested KVM setup)
- reproducible only on RHEL 8.2 and RHEL 8.4 … RHEL 8.6 is unaffected. So, please update the affected CI systems to RHEL 8.6.
- latest status is tracked in cix card
- blocking integration lines of c9 master (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-master) and c9 wallaby (periodic-tripleo-ci-centos-9-scenario010-kvm-internal-standalone-wallaby)
- workaround (specifying cpu_models in libvirt) is currently being tested
https://bugs.launchpad.net/tripleo/+bug/1988347
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
BLOCKER
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
- fails on c9 master and c9 wallaby
- fs064, fs39 and others
- intermittent failure aka when rerunning hard enough it will pass eventually!
- FATAL | try modifying forward dns record | undercloud | error={"changed": false, "msg": "'arecord' not found."}
- FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master
- FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset064-master
- FAILURE in periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master
- FAILURE in periodic-tripleo-ci-centos-9-standalone-on-multinode-ipa-tripleo-master
- FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master
- FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-master
- Ansible pretends that commands arecord and ptrrecord are while trying to run modules in role tripleo_ipa_dns
- history of job periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-master shows e.g. a failure on 2022-08-22 but subsequent passes
https://bugs.launchpad.net/tripleo/+bug/1987323
- FIPS + Manila + Ceph Quincy
- … known issue on the ceph side, … already merged in master/quincy … 17.2.4 should solve this issue, so we'll check during the next promotion.
- On CIX board in lane Waiting on external dependencies
- Not a blocker

https://bugs.launchpad.net/tripleo/+bug/1988397

tripleo-ci-centos-9-undercloud-upgrade failing consistently while updating all packages

<ysandeep> we still have mirror issues:
<ysandeep> job have .40 version installed
<ysandeep> https://32f730b73845cbdfd0fc-4cf4bb096d4774f141f32d30757edf40.ssl.cf1.rackcdn.com/853411/2/check/tripleo-ci-centos-9-undercloud-upgrade/b6a6d75/logs/undercloud/var/log/extra/package-list-installed.txt
<ysandeep> ~~~
<ysandeep> NetworkManager.x86_64                            1:1.40.0-1.el9                        @baseos                       
<ysandeep> NetworkManager-libnm.x86_64                      1:1.40.0-1.el9                        @baseos 
<ysandeep> ~~~
<ysandeep> I don't see .40 rpms for NetworkManager on rackspace mirror: http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/
<ykarel> facebook mirror looks good
<ykarel> rackspace one not updated since 10 hours
<ykarel> http://dfw.mirror.rackspace.com/centos-stream/timestamp.txt
<jm1> ysandeep, ykarel: and neither on http://mirror.mtl01.iweb.opendev.org/centos-stream/9-stream/AppStream/x86_64/os/Packages/
<ykarel> yes ^ syncs from mirror which is rax currently so should be same 
<ykarel> http://mirror.mtl01.iweb.opendev.org/centos-stream/timestamp.txt
<ykarel> 2022-09-01T06:08:45,769043757+00:00

no CIX card

Intermittent Failures

https://bugs.launchpad.net/tripleo/+bug/1987616
- cs9 ovb fs01 clients wallaby is failing with error - 'neutron.agent.dhcp.agent oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply'
- no CIX card
https://bugs.launchpad.net/tripleo/+bug/1938987
- periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- ERROR keystone.server.flask.request_processing.middleware.auth_context oslo_db.exception.DBConnectionError: (pymysql.err.OperationalError) (1047, 'WSREP has not yet prepared node for application use')
- FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train
- FAILURE in periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-train
TASK [freeipa-setup : Run the tripleo-ipa preparation script]
Connection failure: The read operation timed out in TASK [Get current autoholds]
- POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-wallaby
Create containers managed by Podman for ... Failed containers: rabbitmq_bootstrap (or others)
FATAL | Wait for puppet host configuration to finish | undercloud | error…
- TASK [tripleo.operator.tripleo_undercloud_install : undercloud install] in job-output.txt
- puppet-user: Error: Execution of '/bin/dnf -d 0 -e 1 -y install net-snmp' returned 1: Error: Error downloading packages: in logs/undercloud/home/zuul/undercloud_install.log.txt.gz
- POST_FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-baremetal-wallaby
Exception: Command '['ipmitool', ..., 'status']' timed out after 60 seconds" during overcloud deploy
- FAILURE in periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby

Fixed

https://bugs.launchpad.net/tripleo/+bug/1988053
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
BLOCKER
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
- Puppet fails with "Error while evaluating a Resource Statement" on CentOS 9 master during undercloud install
- affecting c9 master tripleo component
- Takashi Kajinami is working on fixes
- c9 master tripleo component promoted today, bug is fixed

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2022-09-01 to 2022-09-01

tags: ruck_rover

Next RR notes: https://hackmd.io/0RguTgCAQsiZz_SnF5209A

Previous RR notes: https://hackmd.io/7qAKWCiCQA6IEdE9WXYn4Q

ruck: Pooja, rover: Jakob

2022-09-01

Reruns and Investigations:

Known Bugs

Intermittent Failures

Fixed

tags: `ruck_rover`