Ruck and rover notes #24

tags: `ruck_rover`

PCCI Ruck Rover Sprint #23 - 5 Mar to 26 Mar 2020

Tripleo CI team ruck|rover: Wes (weshay) && Sandeep ysandeep
OSP CI team ruck|rover: Waldek (wznoinsk) && Vadim (vgriner)

Useful information

Sprint 23

New / Transient / No bug yet:

@raukadah hey hey@w

11:12 < marios> weshay|ruck: fyi 11:09 < dpawlik> if this one will be merged, https://review.opendev.org/#/c/713177 jobs will fail if
they are running on f29/f28
[OSP17][handover] jobs failing with "The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'ctlplane'" - https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/QE/view/OSP17/job/phase1-17_director-rhel-8.1-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/10/artifact/.sh/05-ooo-overcloud.log/view/ , any fixes to infrared should be submitted to gerrit with topic of 'osp17p1' (i.e.: ir_patches_topic in https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/QE/view/OSP17/job/phase1-17_director-rhel-8.1-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/10/parameters/)
[OSP10][handover] phase2-10-rhel-7.7-openstack-all-in-one-neutron-rabbitmq failing on 'yum clean all' command, running again (to confirm) in https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP10/job/phase2-10-rhel-7.7-openstack-all-in-one-neutron-rabbitmq/73/ - if fails as well then it needs investiation, contact: migarcia
[OSP13] PSI upshift quota exceeded - phase2-13-rhel-7.8-openstack-all-in-one-neutron-rabbitmq failing (I think vgriner was fixing this)
[OSP][handover] Work to get first osp16.1 jobs ongoing - https://projects.engineering.redhat.com/browse/RHOSINFRA-3118
found bug: ~~osp16.1, rhel8.2: ipmitool commands via vbmc (virtualbmc) take too long and cause overcloud introspection to fail - https://bugzilla.redhat.com/show_bug.cgi?id=1813889~~
* https://bugzilla.redhat.com/show_bug.cgi?id=1814616 - after extended team troubleshooted the issue on Mar 25th we've found a bogus nftables entries (which came in rhel8.2 only) blocking our dhcp/udp traffic, see the bugzilla for updates on the resolution
[OSP] All IPv6 jobs are broken. - https://projects.engineering.redhat.com/browse/RHOSINFRA-3104 (reverted = resolved, they'll work on a better patch)
[OSP] osp16.0 jobs back working on the puddle from 2020-03-11
OSP15 update job fail on No such property: overcloud_container_images_urls
https://projects.engineering.redhat.com/browse/RHOSINFRA-3091
this would affect other osp versions as well - it's now fixed (as of ~9am GMT today)
OSP16 - phase1 fails
- https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase1-16_director-rhel-8.1-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/105/artifact/.sh/04-ooo-undercloud.log
  *~~ https://bugzilla.redhat.com/show_bug.cgi?id=1809998 duplicate~~
  - Main bug: https://bugzilla.redhat.com/1809939
    - Esca: https://trello.com/c/z0LYn4Rq

Earlier unclosed things tracked here (first is new):

OSP ISSUES

Thu 05 March 2020 R&R tansfer:

OSP16
1)https://trello.com/c/z0LYn4Rq - main p1 blocker
2)^ also forked into https://trello.com/c/Dkvbl5Kb - memcached container issue

OSP13
techdebt - nova-scheduler workaround - https://trello.com/c/pVkLagqH/1337-cixbz1803150ospphase2osp13nova-scheduler-hint-for-nova-scheduler-seems-to-be-ignored - fhubik will deal with that ouside of R&R and sprint

Also

Need to construct email to pntops tlv ( check w/ Amnon )
Express a SLA to rhos-dev re: RFE: https://projects.engineering.redhat.com/browse/RHOSINFRA-3075?filter=34183
details from fhubik on that^ topic: https://trello.com/c/yBgjnUEn
also we need to document the processes behind the PCCI_INFRA_SUPPORT queue (how long of inactivity before we can close/decr. priority etc.)

TripleO ISSUES

train

container build failed on collectd ( 1 time )
https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-train-containers-build-push-vexxhost/2b2c26c/logs/containers-failed-to-build.log

other

Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
https://bugs.launchpad.net/tripleo/+bug/1867744 Pacemaker cluster fails to start in CI jobs
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
https://bugs.launchpad.net/tripleo/+bug/1868079 puppet-neutron container failed
https://bugs.launchpad.net/tripleo/+bug/1867744 Mistral failed command manually execute the following script: /var/lib/mistral/overcloud/ansible-playbook-command.sh
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
https://bugs.launchpad.net/tripleo/+bug/1867602 overcloud deploy failed with Systemd start for pcsd failed
https://bugs.launchpad.net/tripleo/+bug/1867599 overcloud deploy failing on fs030 and fs016 while pulling mariadb container from undercloud registry - FIX IS HERE –> https://review.opendev.org/#/c/712013/6/config/general_config/featureset030.yml BLOCKED BY MOLECULE FAIL
https://bugs.launchpad.net/tripleo/+bug/1866621 Can't run container mistral_db_sync

Arx Cruz2020-03-09 Adriano is taking a look ATM
Arx Cruz2020-03-09 According Emilien this should be fixed by https://review.rdoproject.org/r/25780
Arx Cruz2020-03-16 I'm no longer seeing this happening, but the patch from Emilien isn't merged yet. Moving to done.
Arx Cruz2020-03-16 According yatin, that's no longer happening because mistral is pinned, moving back to open issue
https://review.opendev.org/#/c/713873/ fix merged here testing with mistral unpinning
https://bugs.launchpad.net/tripleo/+bug/1867323 Launchpad bug 1867323 in tripleo " standalone deploy failed at Error: error checking path "/run/libvirt": stat /run/libvirt: no such file or directory"
https://bugs.launchpad.net/tripleo/+bug/1864953 (intermittent/race?) Image prepare failed: [Errno 17] File exists
- ~~https://review.opendev.org/#/c/710836/~~
Arx Cruz2020/03/09 This patch was abandoned, I'll poke around in the CIX meeting to check the status
OSP
- OSP 16 - https://bugzilla.redhat.com/show_bug.cgi?id=1807375#c10 verified
- OSP10
  - Packtack jobs still broken https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP10/job/phase2-10-rhel-7.7-openstack-all-in-one-neutron-rabbitmq/59/
    - problem between ci slaves and github fetch?
      - resolved itself, maybe temp network issue?

Closed things tracked here:

Tripleo Closed

TripleO issues

https://bugs.launchpad.net/tripleo/+bug/1868439 All stable branches OVB job fail because "No module named 'keystoneauth1'"
RHEL vs CentOS 8 for Train
- <wes> FIXED merge conflict</wes>
- (merge conflict on kolla patch https://review.opendev.org/#/c/693159/) try rhel8 container build without kolla patches posted https://review.opendev.org/711198 & testproject https://review.rdoproject.org/r/25663
- nop fails cos no yum-config-manager
  https://review.opendev.org/#/c/693159/5/docker/base/Dockerfile.j2 https://logserver.rdoproject.org/63/25663/1/check/periodic-tripleo-rhel-8-train-containers-build-push/785736a/logs/buildah-builds/kolla-v4kkjxy6/docker/base/base-build.log
- But really we want centos8 train. we have the job but not the pipeline yet
- train phase 1 green build yesterday problem reporting https://ci.centos.org/view/rdo/view/promotion-pipeline/job/rdo_trunk-promote-train-current-tripleo/130/
https://bugs.launchpad.net/tripleo/+bug/1865754 tripleo-ci-centos-8-scenario001-standalone tempest-conf fails 500 PUT http://192.168.24.1:9292/v2/images/ RADOS invalid argument
https://bugs.launchpad.net/tripleo/+bug/1865574 centos-8 multinode and undercloud jobs are hanging on the undercloud install
https://bugs.launchpad.net/tripleo/+bug/1866031 periodic centos7 fs2 upload/fs1 master fails overcloud deploy pcs "create constraint failed"
https://bugs.launchpad.net/tripleo/+bug/1867332 Mistrial tests getting skiped in undercloud deployment

Arx Cruz2020-03-13 akahat is working on this
https://bugs.launchpad.net/tripleo/+bug/1867023 ImportError: cannot import name suppress
https://bugs.launchpad.net/tripleo/+bug/1866543 All CentOS-8 jobs fail on missing various packages from component repos
- https://review.rdoproject.org/r/#/c/25770/ Fix purge script
https://bugs.launchpad.net/tripleo/+bug/1866687 tempest-conf error: Setting [volume-feature-enabled] multi_backend = True; TypeError: option values must be strings
- https://review.opendev.org/#/c/711976/
https://bugs.launchpad.net/tripleo/+bug/1866965 Duplicate declaration: Package[collectd-python] is already declared at puppet-collectd and puppet-tripleo
- https://review.opendev.org/#/c/712289/
Gate fails: No route to host
2020-03-05 02:55:49 | TASK [Gathering Facts] ********************************************************* 2020-03-05 02:55:49 | Thursday 05 March 2020 02:55:49 +0000 (0:00:00.093) 0:00:07.537 ******** 2020-03-05 02:55:53 | ok: [localhost] 2020-03-05 02:55:53 | ok: [192.168.24.14] 2020-03-05 02:56:20 | [WARNING]: Unhandled error in Python interpreter discovery for host 2020-03-05 02:56:20 | 192.168.24.20: Failed to connect to the host via ssh: ssh: connect to host 2020-03-05 02:56:20 | 192.168.24.20 port 22: No route to host 2020-03-05 02:56:35 | fatal: [192.168.24.20]: UNREACHABLE! => changed=false 2020-03-05 02:56:35 | msg: |- 2020-03-05 02:56:35 | Data could not be sent to remote host "192.168.24.20". Make sure this host can be reached over ssh: ssh: connect to host 192.168.24.20 port 22: No route to host 2020-03-05 02:56:35 | unreachable: true 2020-03-05 02:56:35 | 2020-03-05 02:56:35 | NO MORE HOSTS LEFT *************************************************************

Arx Cruz2020/03/06 - Wasn't able to confirm yet if this is a real issue because other failures is happening before we reach this step.
Arx Cruz2020/03/12 - Moving to done because jobs are running fine these days and we haven't see this error.
Gate fails: RuntimeError: Ansible execution failed. playbook: /var/lib/mistral/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2
Gate fails: RuntimeError: Ansible execution failed. playbook: /var/lib/mistral/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2 * https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_265/710811/2/gate/tripleo-ci-centos-7-containers-multinode/2651e0b/logs/undercloud/home/zuul/overcloud_deploy.log
- https://bugs.launchpad.net/tripleo/+bug/1866031 periodic centos7 fs2 upload/fs1 master fails overcloud deploy pcs "create constraint failed"
Arx Cruz2020/03/06 - Wasn't able to confirm yet if this is fixed because other failures is happening before we reach this step.
https://bugs.launchpad.net/tripleo/+bug/1867035 AttributeError: 'MoleculeItem' object has no attribute 'funcargs' in molecule jobs
- https://review.opendev.org/712527
- https://review.opendev.org/712526
Image Not Showing Possible Reasons
- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
Learn More →
https://bugs.launchpad.net/tripleo/+bug/1866316 ara < 1.0.0 is failing on overcloud-prep-image
- https://review.opendev.org/#/c/711597/
Arx Cruz2020/03/06 - I don't know the best approach here if is to update Ara, or if it is to get rid of pip2 and use only pip3 in the overcloud-prep-image role
https://bugs.launchpad.net/tripleo/+bug/1866184 centos-8 multinode overcloud deploy failed with heat_resource_tree_params = heat_resource_tree['parameters'] KeyError: 'parameters'
- https://review.opendev.org/711480
Master fs2 upload issue seems legit:
- Error: /Stage[main]/Tripleo::Profile::Pacemaker::Ovn_dbs_bundle/Pacemaker::Constraint::Order[ip-192.168.24.12-with-ovndb_servers]/Pcmk_constraint[order-ovn-dbs-bundle-ip-192.168.24.12]/ensure: change from 'absent' to 'present' failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200303-9-1fr9x0o create constraint failed: Error: Resource 'ip-192.168.24.12' does not exist~~
- https://logserver.rdoproject.org/40/25640/1/check/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/f29d8ce/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz~~
- https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-master-upload/e7bbed3/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz~~
- filed https://bugs.launchpad.net/tripleo/+bug/1866031
https://bugs.launchpad.net/tripleo/+bug/1865832 tox py3 virtualenv jobs failing ImportError cannot import name 'ContextManager'

OSP Closed

OSP issues

shark13 finally resolved - https://projects.engineering.redhat.com/browse/RHOSINFRA-3019

UNPINNING NOTES / pipeline:

Directions for the unpinning event:

Watch the reviews in [1] for merge, there may be some packages that fail to build and do not move from current -> consistent. The RDO team will handle any of those issues.
The new content will then enter the component pipeline [2], known issues are tracked here [3], note that centos-8-scenario001 has been fixed w/ [4]. Please remember the component pipeline triggers the first component @ 12am UTC and the following 14 components 1.5 hours later. Use test-project jobs if you want to speed things up, try not to starve nodepool though.

If you find a packaging or infra related issue, please alert the EOD for #rhos-ops
If you find a new issue, please create a launchpad bug and mark it a promotion blocker and owned by the associated DFG to the component ( this should be much more clear and easy to determine the owner )
If you find a CI issue please open a launch pad and mark it alert.
If you are not sure what you have found leave notes in the ruck/rover hackmd [5]

To find which packages are building or have FTBFS
https://review.rdoproject.org/r/#/q/topic:rdo-FTBFS
https://trunk.rdoproject.org/centos8-master/queue.html

I'm really proud of your effort to get us here so quickly. Thank you!!
[1] https://review.rdoproject.org/r/#/c/25612/ https://review.rdoproject.org/r/#/c/25727/
[2] http://dashboard-ci.tripleo.org/d/UDA4H3aZk/component-pipeline?orgId=1
[3] https://hackmd.io/HrQd03c9SxOMtFPFrq50tg#Failing-component-pipeline-tests
[4] https://review.opendev.org/#/c/712289/[5] https://hackmd.io/7MBqFHurTA2e5H8kYRwgag#UNPINNING-NOTES–Component-pipeline

Leave notes here on what you find from the recent unpinning

Divide and conquer

Marios + Chandan EURO time
Ronelle + Wes NA time
Arx POC for tempest :)

openstack-periodic-master:

Failing tests:

fs001
fs020
fs035
scenario010-ovn - no plan for this to be in criteria now

components

fs1 this is fuzzy atm - periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master
- undercloud install fail https://logserver.rdoproject.org/openstack-component-baremetal/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master/1417807/logs/undercloud/home/zuul/install-undercloud.log.txt.gz
- unable to create route on br-ex; possibly only due to ens3 in network-scipts
- ironic issues
  - cleaning
  - need to hold nodes and debug and discover
- tempest issues
  - failing network-basic-ops

*~~https://bugs.launchpad.net/tripleo/+bug/1867807 periodic- centos-8-ovb-3ctlr_1comp-featureset001-baremetal-master fails to download tempest cirros image~~

https://bugs.launchpad.net/tripleo/+bug/1867945 periodic centos8 standalone-full-tempest-tempest-master timeout
- tempest (?) related issues again 18/03 & no logs :/
  - (compute) https://logserver.rdoproject.org/openstack-component-tempest/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-tempest-master/de4fa04/job-output.txt
  - (tempest) https://logserver.rdoproject.org/openstack-component-compute/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-full-tempest-compute-master/d55ae94/job-output.txt

integration pipeline failures/bugs

https://bugs.launchpad.net/tripleo/+bug/1867664 Master periodic jobs are failing overcloud deploy with ''Container(s) with bad ExitCode: [''container-puppet-neutron''], check logs in /var/log/containers/stdouts/'
OVB is stuck on providing nodes
https://bugs.launchpad.net/tripleo/+bug/1866204
overcloud deploy failing on fs030 and fs016 while pulling mariadb container from undercloud registry
https://bugs.launchpad.net/tripleo/+bug/1867599
overcloud deploy failed due to Systemd start for pcsd failed
https://bugs.launchpad.net/tripleo/+bug/1867602

Reviews still in play to add/move jobs

Scenario 12
- ~~[rdo-jobs] https://review.rdoproject.org/r/25910~~
- ~~[config] https://review.rdoproject.org/r/25911~~
- ~~[ci-config] https://review.rdoproject.org/r/25912~~
f30 centos8
- ~~[rdo-jobs] https://review.rdoproject.org/r/25921~~
- ~~[config] https://review.rdoproject.org/r/25922~~
- ~~[ci-config] https://review.rdoproject.org/r/25923~~
podman c8
- ~~[rdo-jobs] https://review.rdoproject.org/r/25916~~
- ~~[config] https://review.rdoproject.org/r/25920~~
scenario010-ovn-provider-standalone check @ https://review.opendev.org/711507 , ~~periodic @ https://review.rdoproject.org/r/25745, layout @ https://review.rdoproject.org/r/25746 , criteria @https://review.rdoproject.org/r/25747~~, test @ https://review.rdoproject.org/r/25712
fs039

fs39 tracked there now https://tree.taiga.io/project/tripleo-ci-board/task/1604
- ~~definitions@ https://review.rdoproject.org/r/25793~~
- ~~layout@ https://review.rdoproject.org/r/25794~~
- ~~criteria@ https://review.rdoproject.org/r/25795~~
- ~~fix tripleo-inventory https://review.opendev.org/712962~~
- ~~fix https://review.rdoproject.org/r/25932 config master: Set right centos image for ovb-manage rdo baremetal_image~~
- fix https://review.opendev.org/713659 tripleo-quickstart-extras master: Update freeipa-setup role for centos-8 - ansible_pkg_mgr and packages
- fix https://review.opendev.org/714065 tripleo-quickstart-extras master: Use correct novajoin package for centos8 featureset 39 freeipa
- fix https://review.opendev.org/715397 openstack/tripleo-quickstart master: Move featureset039 to os-tempest conditionally adds required vars
  * fix https://review.rdoproject.org/r/26036 rdo-jobs master: WIP Adds use_os_tempest for centos8 fs39 check and periodic No instead do /715397
- fix … not needed dont build images do r/#/c/26127 instead ~~https://review.opendev.org/714627 tripleo-quickstart-extras master: Fix build-images for centos8 jobs set tripleoclient and image-yaml~~
- fix ~~https://review.rdoproject.org/r/#/c/26127/ Set to_build false for periodic centos8 ovb featureset 039~~
- test@ https://review.rdoproject.org/r/25796
Dont run c7 upgrades on master
- https://review.opendev.org/713277 Only run upgrades centos7 jobs on stable branches
Reproducer
- ~~ Update get-dlrn-hash-newest to work with aggregate hashes https://review.opendev.org/713333~~
Added jobs for distgit testing - https://review.rdoproject.org/r/25659
Baremetal (downstream) reviews
- ~~https://code.engineering.redhat.com/gerrit/194899 Add upstream-centos-8 node~~
- ~~https://code.engineering.redhat.com/gerrit/#/c/194604/ Add baremetal base job for centos-8~~
- ~~https://code.engineering.redhat.com/gerrit/194901 Add centos-8 baremetal nodeset~~
- ~~https://softwarefactory-project.io/r/#/c/17819/ bump version to 2.34.0~~
- https://review.rdoproject.org/r/25995 Add get-hash role to dlrn-vars-setup.yml
- https://code.engineering.redhat.com/gerrit/195040 DNM - Testing bm centos 8 master jobs
  - ~~"msg": "Failed to find required executable virtualenv in paths: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin/:/usr/local/sbin"~~
  - ~~issue with upstream-centos-8 node … need to chat with nhicher~~
- https://code.engineering.redhat.com/gerrit/195145 Adjust envA settings for centos 8

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck and rover notes #24

tags: ruck_rover

PCCI Ruck Rover Sprint #23 - 5 Mar to 26 Mar 2020

Sprint 23

New / Transient / No bug yet:

Earlier unclosed things tracked here (first is new):

OSP ISSUES

TripleO ISSUES

train

other

Closed things tracked here:

Tripleo Closed

OSP Closed

UNPINNING NOTES / pipeline:

Divide and conquer

openstack-periodic-master:

components

integration pipeline failures/bugs

Reviews still in play to add/move jobs

fs39 tracked there now https://tree.taiga.io/project/tripleo-ci-board/task/1604

tags: `ruck_rover`