changed 3 years ago  
Linked with GitHub

CS9 fs035 downstream failure

Overcloud Deploy failure https://sf.hosted.upshift.rdu2.redhat.com/logs/69/43269/1/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby/e5d3013/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz

FATAL | Check all networks Gateway availability | overcloud-controller-1 | item=2001:db8:fd00:1000::1 | error={"ansible_loop_var": "gateway_ip", "changed": false, "cmd": ["ping6", "-w", "10", "-c", "1", "2001:db8:fd00:1000::1"], "delta": "0:00:03.082224", "end": "2022-05-30 19:53:25.382560", "gateway_ip": "2001:db8:fd00:1000::1", "msg": "non-zero return code", "rc": 1, "start": "2022-05-30 19:53:22.300336", "stderr": "", "stderr_lines": [], "stdout": "PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes\nFrom 2001:db8:fd00:1000::3ea icmp_seq=1 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::3ea icmp_seq=2 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::3ea icmp_seq=3 Destination unreachable: Address unreachable\n\n--- 2001:db8:fd00:1000::1 ping statistics ---\n4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3078ms", "stdout_lines": ["PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes", "From 2001:db8:fd00:1000::3ea icmp_seq=1 Destination unreachable: Address unreachable", "From 2001:db8:fd00:1000::3ea icmp_seq=2 Destination unreachable: Address unreachable", "From 2001:db8:fd00:1000::3ea icmp_seq=3 Destination unreachable: Address unreachable", "", "--- 2001:db8:fd00:1000::1 ping statistics ---", "4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3078ms"]}
2022-05-30 23:53:25 | 2022-05-30 23:53:25.705021 | fa163e0c-9118-b86f-ecb7-0000000016f7 |     TIMING | tripleo_nodes_validation : Check all networks Gateway availability | overcloud-controller-1 | 0:01:38.453005 | 3.64s
2022-05-30 23:53:25 | 2022-05-30 23:53:25.708062 | fa163e0c-9118-b86f-ecb7-0000000016f7 |     TIMING | tripleo_nodes_validation : Check all networks Gateway availability | overcloud-controller-1 | 0:01:38.456059 | 3.65s
2022-05-30 23:53:26 | 2022-05-30 23:53:26.147145 | fa163e0c-9118-b86f-ecb7-0000000016f7 |      FATAL | Check all networks Gateway availability | overcloud-controller-2 | item=2001:db8:fd00:1000::1 | error={"ansible_loop_var": "gateway_ip", "changed": false, "cmd": ["ping6", "-w", "10", "-c", "1", "2001:db8:fd00:1000::1"], "delta": "0:00:03.060947", "end": "2022-05-30 19:53:25.351949", "gateway_ip": "2001:db8:fd00:1000::1", "msg": "non-zero return code", "rc": 1, "start": "2022-05-30 19:53:22.291002", "stderr": "", "stderr_lines": [], "stdout": "PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes\nFrom 2001:db8:fd00:1000::267 icmp_seq=1 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::267 icmp_seq=2 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::267 icmp_seq=3 Destination unreachable: Address unreachable\n\n--- 2001:db8:fd00:1000::1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2031ms", "stdout_lines": ["PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes", "From 2001:db8:fd00:1000::267 icmp_seq=1 Destination unreachable: Address unreachable", "From 2001:db8:fd00:1000::267 icmp_seq=2 Destination unreachable: Address unreachable", "From 2001:db8:fd00:1000::267 icmp_seq=3 Destination unreachable: Address unreachable", "", "--- 2001:db8:fd00:1000::1 ping statistics ---", "3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2031ms"]}
2022-05-30 23:53:26 | 2022-05-30 23:53:26.152511 | fa163e0c-9118-b86f-ecb7-0000000016f7 |     TIMING | tripleo_nodes_validation : Check all networks Gateway availability | overcloud-controller-2 | 0:01:38.900501 | 3.76s
2022-05-30 23:53:26 | 2022-05-30 23:53:26.157142 | fa163e0c-9118-b86f-ecb7-0000000016f7 |     TIMING | tripleo_nodes_validation : Check all networks Gateway availability | overcloud-controller-2 | 0:01:38.905138 | 3.77s
2022-05-30 23:53:26 | 2022-05-30 23:53:26.525030 | fa163e0c-9118-b86f-ecb7-0000000016f7 |      FATAL | Check all networks Gateway availability | overcloud-controller-0 | item=2001:db8:fd00:1000::1 | error={"ansible_loop_var": "gateway_ip", "changed": false, "cmd": ["ping6", "-w", "10", "-c", "1", "2001:db8:fd00:1000::1"], "delta": "0:00:03.087338", "end": "2022-05-30 19:53:26.173289", "gateway_ip": "2001:db8:fd00:1000::1", "msg": "non-zero return code", "rc": 1, "start": "2022-05-30 19:53:23.085951", "stderr": "", "stderr_lines": [], "stdout": "PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes\nFrom 2001:db8:fd00:1000::320 icmp_seq=1 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::320 icmp_seq=2 Destination unreachable: Address unreachable\nFrom 2001:db8:fd00:1000::320 icmp_seq=3 Destination unreachable: Address unreachable\n\n--- 2001:db8:fd00:1000::1 ping statistics ---\n3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2057ms", "stdout_lines": ["PING 2001:db8:fd00:1000::1(2001:db8:fd00:1000::1) 56 data bytes", "From 2001:db8:fd00:1000::320 icmp_seq=1 Destination unreachable: Address unreachable", "From 2001:db8:fd00:1000::320 icmp_seq=2 Destination unreachable: Address unreachable", "From

By compairing console logs with RDO passing one https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/daccda2/logs/baremetal_0-console.log
and

https://sf.hosted.upshift.rdu2.redhat.com/logs/69/43269/1/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby/e5d3013/logs/baremetal_1_99351_0-console.log

[H[J[1;1Herror: ../../grub-core/disk/diskfilter.c:916:diskfilter writes are not
supported.
[?25l[m[H[J[1;1H[20;7H[mUse the ^ and v keys to change the selection.                       
      Press 'e' to edit the selected item, or 'c' for a command prompt.   [4;80H [7m[4;1H      CentOS Stream (5.14.0-96.el9.x86_64) 9                                   [m[4;79H[m[m[5;1H      CentOS Stream (5.14.0-80.el9.x86_64) 9                                   [m[5;79H[m[m[6;1H                                                                               [m[6;79H[m[m[7;1H                                                                               [m[7;79H[m[m[8;1H                                                                               [m[8;79H[m[m[9;1H                                                                               [m[9;79H[m[m[10;1H                                                                               [m[10;79H[m[m[11;1H                                                                               [m[11;79H[m[m[12;1H                                                                               [m[12;79H[m[m[13;1H                                                                               [m[13;79H[m[m[14;1H                                                                               [m[14;79H[m[m[15;1H                                                                               [m[15;79H[m[m[16;1H                                                                               [m[16;79H[m[m[17;1H                                                                               [m[17;79H[m[m[18;1H                                                                               [m[18;79H[m[18;80H [4;79H[22;1H   The selected entry will be started automatically in 5s.

Journal errors passing vs failing:
https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-wallaby/daccda2/logs/overcloud-controller-0/var/log/extra/journal_errors.txt.gz
and
https://sf.hosted.upshift.rdu2.redhat.com/logs/69/43269/1/check-rdo/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-internal-wallaby/e5d3013/logs/overcloud-controller-0/var/log/extra/journal_errors.txt.gz

2022-05-30T19:45:04-0400 localhost kernel: audit: kauditd hold queue overflow
2022-05-30T19:45:04-0400 localhost kernel: audit: kauditd hold queue overflow
2022-05-30T19:45:04-0400 localhost kernel: audit: kauditd hold queue overflow
2022-05-30T19:45:10-0400 overcloud-controller-0 ovs-vsctl[1003]: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
2022-05-30T19:46:10-0400 overcloud-controller-0 systemd[1]: Failed to start Network Manager Wait Online.
2022-05-30T19:59:15-0400 overcloud-controller-0 /bin/sealert[28967]: failed to retrieve rpm info for /var/lib/selinux/targeted/active/modules/400/openvswitch-custom
Select a repo