# Ruck Rover - 01 Apr 2022 - 07 Apr 2022
###### tags: `ruck_rover`
###### Previous RR notes: https://hackmd.io/mXb6zsGsTNWWO6Y6UpUNAg?view
[Cockpit](http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1)

## Apr 7 2022
- https://bugs.launchpad.net/tripleo/+bug/1968184, validations rpm not getting build
- https://bugzilla.redhat.com/show_bug.cgi?id=2073227 - Bug 2073227 - [RHOS 16.2] scenario010 standalone tempest conf is failing with "Unhandled error: config_tempest.services.base.ServiceError"

## Apr 6 2022
* ~~new issue stable/wallaby tripleo-heat-templates~~
~~* ImportError: cannot import name 'OrderedDict' from 'typing_extensions' (/home/zuul/src/opendev.org/openstack/tripleo-heat-templates/.tox/pep8/lib/python3.8/site-packages/typing_extensions.py https://zuul.opendev.org/t/openstack/builds?job_name=openstack-tox-pep8&project=openstack%2Ftripleo-heat-templates&branch=stable%2Fwallaby&skip=0~~
* https://bugs.launchpad.net/tripleo/+bug/1968065 openstack-tox-pep8 failing on stable/wallaby tripleo-heat-templates gate blocker
* c7 train content provider: https://bugs.launchpad.net/tripleo/+bug/1968066 - think this cleared?
* 16.2 promoted - yay
* https://bugzilla.redhat.com/show_bug.cgi?id=2072665 - for network 17 blocker
* wallaby c9 promoted
* testprojects running to promote victoria

## Apr 05, 2022
### Promotion blocker
* ~~C8 Wallaby https://bugs.launchpad.net/tripleo/+bug/1967943~~
* ~~C9 Master|Wallaby: https://bugs.launchpad.net/tripleo/+bug/1967945~~

It might be connected to: https://status.python.org/incidents/mxgkk3xxr9v7?u=v8pzlr5n28h8
Rechecking jobs to see if the issue is resolved (https://review.rdoproject.org/r/c/testproject/+/41370)

reference: https://lists.openstack.org/pipermail/openstack-discuss/2022-April/028037.html

The problem seems to be still present. I asked vexxhost for a confirmation wrt missing cache proxy.

FTR: The problem was connected to outdated pypi cache
> 05:57 <dpawlik> ProxyPass "/pypi/" "https://pypi.org/" ttl=120 keepalive=On retry=0
> 05:57 <dpawlik> ProxyPassReverse "/pypi/" "https://pypi.org/
> 05:58 <dpawlik> and pypifiles
> 05:58 <bhjf> Title: PyPI · The Python Package Index (at pypi.org)
> 05:58 <dpawlik> CacheEnable disk "/pypifiles"
> 05:58 <dpawlik> ProxyPass "/pypifiles/" "https://files.pythonhosted.org/" ttl=120 keepalive=On retry=0
> 05:58 <dpawlik> ProxyPassReverse "/pypifiles/" "https://files.pythonhosted.org/"

> 06:18 <dpawlik> #status log flushink cache on afs mirror: htcacheclean -v -t -p /var/cache/httpd/ -l4096M
06:18 <sfstatusbot> dpawlik: finished logging
06:19 <dpawlik> #status log flushing cache on afs mirror: htcacheclean -v -t -p /var/cache/httpd/ -l4096M
06:19 <sfstatusbot> dpawlik: finished logging

Registry pruning issue:
- https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/41574
CentOS-9-{master,wallaby} api endpoints were missing from the whitelist.

### Other
* c7 train is red: https://trunk.rdoproject.org/api-centos-train/api/civotes_detail.html?commit_hash=4c3dfec59da9dbd40c4f6e7aa9d44aa56e4f45a4&distro_hash=0ad01be232c86e32e4c397420feb6d280ae9582f
* c8, wallaby: https://review.rdoproject.org/r/c/testproject/+/41370
* victoria rerun:https://review.rdoproject.org/r/c/testproject/+/41469
* rhos-16.2 containers are not pushing - pls check into this
* New [Service now](https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=sc_request&sys_id=a83925e387a24550949fbbbf8bbb35cd) ticket
* c7 is not promoting - pls check the promoter
* train c8: https://review.rdoproject.org/r/c/testproject/+/39960
* Promoted today
* rhos17 rhel8 clients and tripleo component : https://code.engineering.redhat.com/gerrit/c/testproject/+/317875
* c8s wallaby tripleo network component testproject https://review.rdoproject.org/r/c/testproject/+/40642

## Apr 04, 2022
> 11:49:45 amoralej | yoga is the last release of rdo on centos8

* Downstream - lot sof stuck lines - need to be cleaned and see if downstream is working again. I cleared some line - pls get infra to clear the rest
* Some long running lines in RDO zuul as well - may need clean up there
* RDO - Pipelines were stuck cleard and may be need to reschedule those lines:
* openstack-periodic-integration-stable1
* openstack-periodic-integration-stable4
* openstack-component-tempest
* openstack-promote-component
* Because of the zuul update lot's of jobs got failed with error: [Zuul] Log Stream did not terminate
* **PROMOTION BLOCKER**: https://bugs.launchpad.net/tripleo/+bug/1967743

### Upstream
* cs7 train
* rerun, https://review.rdoproject.org/r/c/testproject/+/41367
> Hash under test: https://trunk.rdoproject.org/api-centos8-victoria/api/civotes_agg_detail.html?ref_hash=5996b91e558f07113a06bfec6c8cfa9e
> last_promotion=2022-04-01 22:10:41
* cs8 victoria
> last_promotion=2022-04-01 22:10:41
* ~~cs9 wallaby~~
* rerun, https://review.rdoproject.org/r/c/testproject/+/41370
> last_promotion=2022-03-30 21:29:03

### Downstream failing components
* periodic-tripleo-rhel-8-rhos-17-component-clients-promote-to-promoted-components
* periodic-tripleo-rhel-8-rhos-17-component-network-promote-to-promoted-components
* periodic-tripleo-rhel-8-rhos-17-component-tripleo-promote-to-promoted-components

* ~~periodic-tripleo-rhel-9-rhos-17-component-clients-promote-to-promoted-components~~

* ~~periodic-tripleo-rhel-8-rhos-16.2-component-network-promote-to-promoted-components~~

## Apr 01, 2022
* cs9 master: https://review.rdoproject.org/r/c/testproject/+/41278
* ~~sc10 cs9 wallaby: https://code.engineering.redhat.com/gerrit/c/testproject/+/301027~~ -- happy run -- check for promotion
* Amol/Dasm keep follow up on https://bugs.launchpad.net/tripleo/+bug/1966625 components line is blocked because of this bug.
* ~~cs8 victoria, rerun: https://review.rdoproject.org/r/c/testproject/+/41367~~ -- should be promoting. It passed
* cs9 wallaby, rerun: https://review.rdoproject.org/r/c/testproject/+/41370
* ~~16.2, network: https://code.engineering.redhat.com/gerrit/c/testproject/+/398718~~ SUCCEDED
* 17.0 -- tripleo component is too old. It doesn't contain https://opendev.org/openstack/tripleo-common/src/branch/master/container-images/kolla/base/uid_gid_manage.sh#L66
* tempest component on wallaby needs some love next week pls

## Mar 31, 2022
### Prod Blocker

#### [bz:2068055]
[cix:2069171] - downstream zuul still unavailable

* ~~https://bugs.launchpad.net/tripleo/+bug/1967430 - Ronelle sent email to ML. Revert https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/835977 proposed.~~

### CI Degraded
~~[lp:1965540]~~
~~revert created [835974], waiting for a merge of [835475]. After it's merged, we can close [cix:1965540]~~

[lp:1965540]: https://bugs.launchpad.net/tripleo/+bug/1965540
[835974]: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/835974
[835475]: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/835475
[cix:1965540]: https://trello.com/c/FE77D5P9/2414-cixlp1965540tripleociproa-tempest-test-testimagewebdownloadimport-failing-with-uri-for-web-download-does-not-pass-filtering
[bz:2068055]: https://bugzilla.redhat.com/show_bug.cgi?id=2068055
[cix:2069171]: https://trello.com/c/VNJwJhmd/2426-cixbz2069171osp16-2rhel8rhosp16-2tempestscenariotestnetworkbasicopstestnetworkbasicops-failing-on-tempest-tests-with-sshtimeout