# Ruck Rover 2022-09-16 to 2022-09-22

###### tags: `ruck_rover`
###### Next RR notes: TODO
###### Previous RR notes: https://hackmd.io/dKeK6zo9R66heikGyCb4NA

##### ruck: Amol, rover: Ronelle

[RDO Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1) / [RHOS Cockpit](http://tripleo-cockpit.lab4.eng.bos.redhat.com)

[RDO Promoter](http://promoter.rdoproject.org/promoter_logs/) / [RHOS Promoter](http://10.0.110.143/promoter_logs/)

[OpenStack Program Meeting 2022](
https://docs.engineering.redhat.com/pages/viewpage.action?spaceKey=PRODCHAIN&title=Meeting+notes)

Zuul Status:
* [opendev.org:openstack](https://zuul.opendev.org/t/openstack/status/)
* [rdoproject.org:rdoproject.org](https://review.rdoproject.org/zuul/status)
* [redhat.com:tripleo-ci-internal](https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status)

---
## RR Handoff

### Active CIX bugs
* https://bugzilla.redhat.com/2127111 Stderr: 'error reading multiple stages: ARG requires exactly one argument definition\n'
* https://bugs.launchpad.net/tripleo/+bug/1990137 - ddt has no attribute named_data
* Fs20-Kernel does not provide mount namespace: No such file or directory
- https://bugs.launchpad.net/tripleo/+bug/1990359
* Failure running exec 'keystone_bootstrap' - "Lost connection to MySQL server during query"
- https://bugs.launchpad.net/tripleo/+bug/1990415
* Tempest test test_create_update_port_with_dns_domain failure KeyError: 'dns_domain'
- https://bugs.launchpad.net/tripleo/+bug/1990480
* Access denied to the swift resource
- https://bugzilla.redhat.com/show_bug.cgi?id=2129026
- Tosky marked this bug as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2123335

### Promotions
- OSP17 - RHEL-9: Promoted today, http://10.0.110.143/promoter_logs/redhat9_osp17_2022-09-22T11:04.log
- OSP17.1 - RHEL-9:
---
## 2022-09-22

### New Bugs
- Tempest test test_create_update_port_with_dns_domain failure KeyError: 'dns_domain'
- https://bugs.launchpad.net/tripleo/+bug/1990480
- Access denied to the swift resource
- https://bugzilla.redhat.com/show_bug.cgi?id=2129026
- Tosky marked this bug as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=2123335

### Reviews
- Add internal fs35 job in criteria
- https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45215
- Testing internal fs35 job here
- https://code.engineering.redhat.com/gerrit/c/testproject/+/429379

### Reruns And Investigation
#### Integration Lines

RHOS-16.2 Component jobs rerun
- https://code.engineering.redhat.com/gerrit/c/testproject/+/428925
- https://code.engineering.redhat.com/gerrit/c/testproject/+/300913

Upstream Integration jobs
- https://review.rdoproject.org/r/c/testproject/+/41469

Upstream component jobs
- https://review.rdoproject.org/r/c/testproject/+/41465

### Promotions
- OSP17 - RHEL-9: Promoted today, http://10.0.110.143/promoter_logs/redhat9_osp17_2022-09-22T11:04.log

####
---
## 2022-09-21

New Bug
-----

* Fs20-Kernel does not provide mount namespace: No such file or directory
- https://bugs.launchpad.net/tripleo/+bug/1990359
* FreeIPA failed to install - fs39-master
- https://bugs.launchpad.net/tripleo/+bug/1990371
* Failure running exec 'keystone_bootstrap' - "Lost connection to MySQL server during query"
- https://bugs.launchpad.net/tripleo/+bug/1990415

### Reruns and Investigation

#### Integration Lines

* master c9 - **promoted 09/20**
seeing https://bugs.launchpad.net/tripleo/+bug/1990415 happen a lot

* wallaby c9 - **promoted 09/19**

* wallaby c8 - **promoted 09/21**
* https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
* https://bugs.launchpad.net/tripleo/+bug/1990012
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45089

* train c8 - **promoted 09/20**
chasing train fs035

- 17.1 / RHEL-9: **Promoted today**: http://10.0.110.143/promoter_logs/redhat9_osp17-1_2022-09-21T13:52.log
- ~~https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/429155~~ Abandoned
- 17.0 Job rerun: https://code.engineering.redhat.com/gerrit/c/testproject/+/300913
- 16-2 / RHEL-8: **No new hasesh to promote.**
- 17.1 - RHEL-8: **Promoted today**: http://10.0.110.143/promoter_logs/redhat8_osp17-1_2022-09-21T11:52.log
still waiting on components - code.eng was down

---
## 2022-09-20

### Known Bugs

[CIX](https://trello.com/b/j4IcIomh/production-chain-escalation)

* ~~https://bugs.launchpad.net/tripleo/+bug/1990269~~ NODE_FAILURES when running tripleo-ci-centos-9-scenario010-standalone on opendev

* https://bugs.launchpad.net/tripleo/+bug/1989452 multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host
* Set configure_mirrors_components_9_stream to true https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858209

* ~~https://bugs.launchpad.net/tripleo/+bug/1990045
periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train is failing - No such file or directory: '/home/zuul/tempest/etc/tempest_includelist.txt' (asking Arx - he responded he will check there)~~

* ~~https://bugs.launchpad.net/tripleo/+bug/1990046~~ periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby is failing standalone deploy starting container collectd - container state improper

* ~~https://bugs.launchpad.net/tripleo/+bug/1989341 Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch~~

* ~~https://bugs.launchpad.net/tripleo/+bug/1990086~~ fs064 and fs039 are unstable - investigating deploy_freeipa

* https://bugzilla.redhat.com/2127111 Stderr: 'error reading multiple stages: ARG requires exactly one argument definition\n'

* ~~https://bugs.launchpad.net/tripleo/+bug/1989197~~ Tempest test "neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON" failing on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

* ~~https://bugs.launchpad.net/tripleo/+bug/1989606~~
container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use

* ~~https://bugs.launchpad.net/tripleo/+bug/1987632~~ cs9 fs01 check job failing on node_provisioning

* ~~https://bugzilla.redhat.com/show_bug.cgi?id=2127828 - https://download.devel.redhat.com certificate is expired.~~

* https://bugs.launchpad.net/tripleo/+bug/1990137 - ddt has no attribute named_data

### Reruns and Investigations:

#### Integration Lines

* master c9 - **promoted 09/20**
chasing two hashes
* **fs064 fs039 - Deploy IPA - install of supplemental failing - possible DNS** Investigating these two jobs
https://bugs.launchpad.net/tripleo/+bug/1990086

* wallaby c9 - **promoted 09/19**

* wallaby c8 - **promoted 09/20**
* https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
* https://bugs.launchpad.net/tripleo/+bug/1990012
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45089

* train c8 - **promoted 09/20**

* [rhel-9 osp17-1](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline)
* **QE jobs are missing to get this promotion - pls check with attila in the morning if they have not run**

* [rhel-9 osp17](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline)
* **promoted 09/20**

* rhel-8 osp17-1

* [rhel-8 osp16-2](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline)
* Last promoted on 2022-09-16: http://10.0.110.143/promoter_logs/redhat8_osp16-2_2022-09-16T11:38.log-20220917
* **no new content - QE jobs are blocking promotions here**
* will need to raise at program call if not fixed in the morning

* d/stream promotions chasers -> https://code.engineering.redhat.com/gerrit/c/testproject/+/428925 https://code.engineering.redhat.com/gerrit/c/testproject/+/428940

* Upstream promotions:
* Integration: https://review.rdoproject.org/r/c/testproject/+/41469
* Component: https://review.rdoproject.org/r/c/testproject/+/41465

---
## 2022-09-19

### Known Bugs

[CIX](https://trello.com/b/j4IcIomh/production-chain-escalation)

* https://bugs.launchpad.net/tripleo/+bug/1989452 multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host
* Set configure_mirrors_components_9_stream to true https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858209

* ~~https://bugs.launchpad.net/tripleo/+bug/1990045
periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train is failing - No such file or directory: '/home/zuul/tempest/etc/tempest_includelist.txt' (asking Arx - he responded he will check there)~~

* https://bugs.launchpad.net/tripleo/+bug/1990046 periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby is failing standalone deploy starting container collectd - container state improper

* ~~https://bugs.launchpad.net/tripleo/+bug/1989341 Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch~~

* https://bugs.launchpad.net/tripleo/+bug/1990086 fs064 and fs039 are unstable - investigating deploy_freeipa

* https://bugzilla.redhat.com/2127111 Stderr: 'error reading multiple stages: ARG requires exactly one argument definition\n'

* https://bugs.launchpad.net/tripleo/+bug/1989197 Tempest test "neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON" failing on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

* https://bugs.launchpad.net/tripleo/+bug/1989606
container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use

* https://bugs.launchpad.net/tripleo/+bug/1987632 cs9 fs01 check job failing on node_provisioning

* ~~https://bugs.launchpad.net/tripleo/+bug/1989795
periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby is failing deploy - Failed containers: designate_db_sync~~

* https://bugzilla.redhat.com/show_bug.cgi?id=2127828 - https://download.devel.redhat.com certificate is expired.
* https://bugs.launchpad.net/tripleo/+bug/1990137 - ddt has no attribute named_data

### Reruns and Investigations:

#### Integration Lines

* master c9 - **promoted 09/18**
chasing two hashes
* **fs064 fs039 - Deploy IPA - install of supplemental failing - possible DNS** Investigating these two jobs
https://bugs.launchpad.net/tripleo/+bug/1990086

* wallaby c9 - **promoted 09/19**
* waiting on fs001 to get a promotion
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/858030/ revert in gate

* wallaby c8 - **promoted 09/19**
* https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
* https://bugs.launchpad.net/tripleo/+bug/1990012
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45089

* train c8 - **promoted 09/19**
* ~~waiting on kvm job to get a promotion~~
* Train promoted: http://promoter.rdoproject.org/promoter_logs/centos8_train_2022-09-19T13:26.log

* [rhel-9 osp17-1](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline)
* Running Component jobs: https://code.engineering.redhat.com/gerrit/c/testproject/+/428925

* [rhel-9 osp17](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline)
https://code.engineering.redhat.com/gerrit/c/testproject/+/428951 Job rerun

* rhel-8 osp17-1

* [rhel-8 osp16-2](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline)
* Last promoted on 2022-09-16: http://10.0.110.143/promoter_logs/redhat8_osp16-2_2022-09-16T11:38.log-20220917

#### Component Lines

* master - manilla -
* need a bug: https://bugs.launchpad.net/tripleo/+bug/1990137: ddt has no attribute named_data # TODO akahat create cix.
* wallaby - security
* should promote today

---
## 2022-09-16

### Known Bugs

[CIX](https://trello.com/b/j4IcIomh/production-chain-escalation)

* https://bugs.launchpad.net/tripleo/+bug/1989452 multiple periodic integration jobs fail configure-mirrors - Failed to connect to mirrors.centos.org port 443: No route to host
* Set configure_mirrors_components_9_stream to true https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/858209

* https://bugs.launchpad.net/tripleo/+bug/1990045
periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train is failing - No such file or directory: '/home/zuul/tempest/etc/tempest_includelist.txt' (asking Arx - he responded he will check there)

* https://bugs.launchpad.net/tripleo/+bug/1990046 periodic-tripleo-ci-centos-9-scenario001-standalone-wallaby is failing standalone deploy starting container collectd - container state improper

* ~~https://bugs.launchpad.net/tripleo/+bug/1989341 Tempest tests "tempest.api.compute.security_groups.test_security_groups.SecurityGroupsTestJSON" etc. fail on {periodic-,}tripleo-ci-centos-8-9-multinode-mixed-os due to ovn controller/north version mismatch~~

* https://bugs.launchpad.net/tripleo/+bug/1990086 fs064 and fs039 are unstable - investigating deploy_freeipa

* https://bugzilla.redhat.com/2127111 Stderr: 'error reading multiple stages: ARG requires exactly one argument definition\n'

* https://bugs.launchpad.net/tripleo/+bug/1989197 Tempest test "neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTestJSON" failing on periodic-tripleo-ci-centos-9-standalone-full-tempest-api-master

* https://bugs.launchpad.net/tripleo/+bug/1989606
container creation during overcloud deploy fails on c9/c8 master fs1 with "You have to remove that container to be able to reuse that name.: that name is already in use

* https://bugs.launchpad.net/tripleo/+bug/1987632 cs9 fs01 check job failing on node_provisioning

* ~~https://bugs.launchpad.net/tripleo/+bug/1989795
periodic-tripleo-ci-centos-9-scenario003-standalone-wallaby is failing deploy - Failed containers: designate_db_sync~~

* https://bugzilla.redhat.com/show_bug.cgi?id=2127828 - https://download.devel.redhat.com certificate is expired.
### Reruns and Investigations:

**NOTE:** Watch for running `testproject` jobs on
https://review.rdoproject.org/zuul/status and
https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status.

#### Integration Lines

* master c9 - **promoted 09/18**
chasing two hashes
* **fs064 fs039 - Deploy IPA - install of supplemental failing - possible DNS** Investigating these two jobs
https://bugs.launchpad.net/tripleo/+bug/1990086

* wallaby c9 - real bug there
* https://review.opendev.org/c/openstack/tripleo-quickstart/+/858030/ revert in gate

* wallaby c8 - **promoted 09/17**
* https://review.rdoproject.org/zuul/buildset/db137665da154a91a62ac59bd7415a12 - mixed rhel is reporting to the wrong hash
* https://bugs.launchpad.net/tripleo/+bug/1990012
* https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45089

* train c8 - **promoted 09/15**

* [rhel-9 osp17-1](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/bSwsg0WVz/rhel9-rhos17-1-full-component-pipeline)

* [rhel-9 osp17](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/lF7RUpsnk/rhel9-rhos17-full-component-pipeline)

* rhel-8 osp17-1

* [rhel-8 osp16-2](
http://tripleo-cockpit.lab4.eng.bos.redhat.com/d/KyHCwLHMk/rhos-16-2-full-component-pipeline)

#### Component Lines

### Downstream:
* rhos16.2 on rhel8:
* Promotion: 16-sept-2022
* ovb jobs were failing due to retry_limit and node_failure so re-run the jobs here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421970/9#message-811e338ca663e2833ff5286b481df1c4bfa7626b only one job failied re-running that here: https://code.engineering.redhat.com/gerrit/c/testproject/+/428361

* rhos17.1 on rhel8
* Promotion: 16-sept-2022
*
* rhos17 on rhel9:
* Promotion: 11-Sept-2022
* Note: we don't have new content to promote.
* rhos17.1 on rhel9:
* Promoted: 15-sept-2022
* ovb jobs were failing due to retry_limit and node_failure so re-run the jobs here : https://code.engineering.redhat.com/gerrit/c/testproject/+/421970/9#message-811e338ca663e2833ff5286b481df1c4bfa7626b only one job failied re-running that here: https://code.engineering.redhat.com/gerrit/c/testproject/+/428361
* "pipeline_integration-pcci-17.1_dlrn-rhel-9.0-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph" job is also failing - started a re-run
* **TO DO: if re-run fails then will need to ping atila**

### Downstream component:

* rhos16.2 on rhel8:
* network and manila jenkins jobs are failing : hit the re-build waiting for the result.
* **TO DO: If re-run fails then will need to ping atila to look at them**
* rhos17.1 on rhel9:
* sc004 failing on common, cinder, glance, manila and tripleo component : abregman reported a bug https://bugzilla.redhat.com/show_bug.cgi?id=2126064
---