---
# System prepended metadata

title: 'Ruck and Rover notes #29'
tags: [ruck_rover]

---

# Ruck and Rover notes #29
###### tags: `ruck_rover`

:::info
Important links for ruck rover's [ruck/rover links to help](https://hackmd.io/07z0xroHTFi2IbX93P5ZfQ)
**Ruck Rover - Unified Sprint #29
Dates: June 18 - July 8th

Tripleo CI team ruck|rover: Soniya Vyas, Sagi Shnaidman, Ronelle Landy
OSP CI team ruck|rover: psedlak, tkorol?

Previous notes: https://hackmd.io/YAqFJrKMThGghTW4P2tabA
**Next #30 notes: https://hackmd.io/6Bx0FXwlRNCc75l39NSKvg**
:::

[TOC]

---

## on-going issues
:::danger
## TripleO



### zuul patch
https://review.opendev.org/#/c/738668/


- https://trello.com/c/ZrPzcm1x/1558-cixlp1883977tripleociproa-spike-in-node-failures-in-rdo-3rd-party-zuul - again 7 failures today
Bug: [ spike in node failures in rdo 3rd party zuul ](https://bugs.launchpad.net/tripleo/+bug/1883977) 
https://review.rdoproject.org/zuul/builds?result=NODE_FAILURE

- [1884518 - OVB metalsmith deployment fails: Failed to attach VIF ... to bare metal node, Node ... is locked by host undercloud](https://bugs.launchpad.net/tripleo/+bug/1884518)

### gate

Gate failures are at 28 - um make that 32 now:
* lools like we have a number of failures on RAX - asked open opensrack-infra - pls follow up

### periodic / 3rd party



## OSP


### osp15 0624.n.2
* first failed in phase1 due missing container images
* after rebuild it passed phase1
* FAILED in phase2 - ceph-jobs fail OC deploy
    * once rhcs4beta is out of the way (ceph-external or IR revert see below)
        * jobs are also affected by ansible-2.9.10
            * cloned as https://bugzilla.redhat.com/show_bug.cgi?id=1850978
                * mistake solved by rhos-release fix, it should have been using ansible 2.8
    * **EnableRhcs4Beta: true**   is missing due to bug in infrared
        * while need for this should be removed in z3, not yet in this CVE compose
            * https://bugzilla.redhat.com/show_bug.cgi?id=1794530
            * https://bugzilla.redhat.com/show_bug.cgi?id=1790906
            * there is delivery ticket for considering shipping the drop-Beta change https://trello.com/c/MAT790lU
            * this seems to be rejected as such change (non-beta ceph4) has too big risk/impact
        * see newer/latest comments in jira ticket for which the breaking IR change was done
            * https://projects.engineering.redhat.com/browse/RHOSINFRA-3277
        * **revert/fix of infrared issue in https://review.gerrithub.io/c/redhat-openstack/infrared/+/496293**
            * OSP15 + rhos-release/ansible fixed
                * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase2-15_director-rhel-8.2-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph-ssl/5/
            * tested in 16.1 phase1 - all ok, no change in ceph config
                * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase1-16.1_director-rhel-8.2-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/83/

:::


### Reviews / Fixes
::: spoiler PATCHES
**Remove docker_host configuration from env files** 
https://review.opendev.org/737235 for bug [1884371](https://bugs.launchpad.net/tripleo/+bug/1884371)
**Configure docker host for local container build** 
https://review.opendev.org/737234 for bug [1884371](https://bugs.launchpad.net/tripleo/+bug/1884371)
### RETRY_LIMIT
https://bugs.launchpad.net/tripleo/+bug/1885701
https://bugs.launchpad.net/tripleo/+bug/1885697
https://bugs.launchpad.net/tripleo/+bug/1885286

### SegFaults
https://bugs.launchpad.net/tripleo/+bug/1885728

### node failures
https://bugs.launchpad.net/tripleo/+bug/1885715

### image builds
need to land ( image builds )
https://review.opendev.org/#/c/738434/
https://review.opendev.org/#/c/738469/

:::

### Bugs reported
::: spoiler BUGS
[1884371 - periodic master - queens jobs using docker.io in container prep](https://bugs.launchpad.net/tripleo/+bug/1884371)
[1884518 - OVB metalsmith deployment fails: Failed to attach VIF ... to bare metal node, Node ... is locked by host undercloud ](https://bugs.launchpad.net/tripleo/+bug/1884518)
[1885279 - 
TestVolumeBootPattern.test_volume_boot_pattern tests on master are failing on updating to cirros-0.5.1 image](https://bugs.launchpad.net/tripleo/+bug/1885279)
[1885286 - Increase in RETRY_LIMIT errors in zuul.openstack.org is preventing jobs from passing check/gate](https://bugs.launchpad.net/tripleo/+bug/1885286)
[1884287 - 
ipa-server install error: 2020-06-19T13:10:53Z DEBUG The ipa-server-install command failed, exception: DNSZoneAlreadyExists: DNS zone](https://bugs.launchpad.net/tripleo/+bug/1884287)
[1885314 - 
OVB master job running on vexxhost show some nodes failing introspection step](https://bugs.launchpad.net/tripleo/+bug/1885314)
[1885315 - tripleo-buildimage-overcloud-full-centos-8 is failing on the update of libnghttp2 package](https://bugs.launchpad.net/tripleo/+bug/1885315)
[1886068 - multinode-ipa tests are failing standalone deployment - 'regsubst' parameter 'target' expects a value of type Array or String, got Undef- ](https://bugs.launchpad.net/tripleo/+bug/1886068)

:::


---


[TOC]

---

### Reviews / Fixes
::: spoiler PATCHES

:::

### Bugs reported
::: spoiler PATCHES

:::

---

:::info
add dates in decending order so the latest date is at the top.  Break out TripleO and OSP sections.
:::

## July 2nd
### TripleO

scenario010-ovn-provider-standalone fails

## July 1st
### TripleO
reported:

- ~~1885865~~ Periodic C8 Ceph Integration/ Ceph ansible integration jobs are failing Error: msg": "The conditional check 'release is search(\"queens|rocky|stein|train\")' failed - sandeep proposed patch
- ~~1885911~~: C8 Ceph Ansible integration train/ussuri jobs pulling master bits - sandeep proposed patch

### OSP
* auto-promote of passed phase compose symlinks is broken, already for last few days
    * FIXED today July 1st
    * `bash: /home/boston/lhh/puddle-promote: No such file or directory`
    * this means once phase2 passes, symlinks are not updated and phase3 when started will consume wrong - previous - compose instead
    * 16.0 compose promoted manually (p3 aborted as it was container grade test compose)
    * 16.1 and 13 in progress - if auto-promote not fixed, it will trigger p3 with wrong compose


## June 29th
### TripleO

reported: 
1885642 tripleo-ci-centos-7-scenario000-multinode-oooq-container-upgrades queens fails on pip install contextlib2 between 0.5 < 0.6 when queens

1885637 tripleo-upgrade needs to update yum repos prior to yum updates for upgrade jobs.


## June 28th
### TripleO

<sshnaidm>

- rocky push containers job is timing out consistently,
suspect of trunk registry problem, need to wait for infra folks to look tomorrow

</sshnaidm>


## June 26th
### TripleO

<rlandy>

* rdocloud promoter - had a restart ... need panda's help to fix it
up and running now.
* https://review.opendev.org/#/c/738025/ - need to get through - revert cirros change on master
https://bugs.launchpad.net/tripleo/+bug/1885279
* https://bugs.launchpad.net/tripleo/+bug/1885286 - 
Increase in RETRY_LIMIT errors in zuul.openstack.org is preventing jobs from passing check/gate


</rlandy>


### OSP
* psedlak on pto friday 26. 6. 2020 (and tkorol/tlv not on friday)
    * afazekas/wznoinsk to provide cover fire
#### 16.1
* **new compose for RC 0625.n.0**
    * passed phase1
    * phase2 in progress - check it on friday morning
        * seems most passed, two jobs left ...
* previous compose **0623.n.0** failed in phase1
    * failed in ceph-ansible - failed to create temporary directory, cix https://trello.com/c/zDFpdGiy
    * caused by update of ansible-2.9.10 arrived in rhel-8.2
    * fix should be on the way but it is possibly affecting also other parts, not just the first one identified

## June 25th
### TripleO
<soniya>We are not facing stack delete failure currently</soniya>

## June 24th
### TripleO
<soniya>No more stacks failed to delete issues
~~too many Post failures and retry_limits issues in Upstream gate jobs~~
Most of above issues are resolved and patches have been merged</soniya>
Multiple failures on each release - tracking here

Also stuck stacks are still an issue:
https://bugs.launchpad.net/tripleo/+bug/1884845
Soniya, please see if this is still an issue tomorrow - ie: are we getting more stacks failing to delete?
</rlandy>

<rlandy>
TODO: Component job failures - especially tripleo
TODO for tomorrow - promote train, queens
</rlandy>

### OSP
* 16.1
    * 0622.n.2 fails in phase1 in UC install
        * `puppet-user: Error: Evaluation Error: Error while evaluating a Function Call, Could not find class ::panko::client for undercloud-0.redhat.local (file: /var/lib/tripleo-config/puppet_step_config.pp, line: 51, column: 1) on node undercloud-0.redhat.local"], ...`
    * new compose **0623.n.0** in phase1
        * passed the UC stage so previous puppet issue is resolved
        * failed in ceph-ansible - failed to create temporary directory, cix https://trello.com/c/zDFpdGiy

* rhos-slave-00..03 (nodes in rdu2 rhev-ci-vms) are having issues with dhcp not providing dns info, without dns they are broken (no git cloning etc)
    * ~~discovered yesterday, they are off in jenkins now~~
    * manually injected rdu2 nameservers in their resolv.conf
    * ~~but that will not survive with NetworkManager updating it according to dhcp info~~
        * they do not use dhcp at all, but static config, so simply dns entry is missing there
    * manual fix still works 3 of 4 are back online for now
    * 4th (the 00) one to be used for debug/testing the issue and PnT ticket to be filed then
    * resolved, details in https://projects.engineering.redhat.com/browse/RHOSINFRA-3513


## June 22nd
### TripleO

- ~~<soniya> noticed multinode fs10-stein failed: https://bugs.launchpad.net/tripleo/+bug/1884487
log: https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-stein/9138146/logs/undercloud/home/zuul/undercloud_install.log.txt.gz</soniya>~~
dup of 1884371
- 
<soniya>failed centos8-multinode job once in the Upstream gate jobs - https://review.opendev.org/#/c/736089/3
recheck is given</soniya>

<soniya> failed centos-8-standalone-on-multinode-ipa job once in the Upstream gate jobs - https://review.opendev.org/#/c/736521/</soniya>

:warning: <sshnaidm>[1884518 - OVB metalsmith deployment fails: Failed to attach VIF ... to bare metal node, Node ... is locked by host undercloud ](https://bugs.launchpad.net/tripleo/+bug/1884518)</sshnaidm>
<sshnaidm>running https://review.rdoproject.org/r/#/c/28004 for reproducing</sshnaidm>


## June 21st
### TripleO

master pipeline failing on container registry settings
https://bugs.launchpad.net/tripleo/+bug/1884371

### OSP
* mostly everything ok (no new compose with issues or major outage)
* still bit of fallout after last weeks shutdown of tlv and jenkins
    * based on https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/running/ there are few jobs running(=hanging) for days
        * likely last case of the job which lost its node while running https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/running/job/DFG-all-unified-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-ceph-native-default/201/
        * qe-ciosp-03 was moved to tlv2 jenkins but this job depends on it https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/running/job/gate-infrared-openstack-ci-osp7/528/
            * via gate-infrared-openstack node label, unkown to me at this moment why the label exists and why it would have just single slave
* also at end of week it was discovered that on rhos-qe-jenkins it was allowed for regular users to manually create&modify jobs (should be only via jjb in git), it was corrected (noted here in case anyone hits issues with that)

## June 19th
### TripleO
[14:38:37] <weshay|ruck> unit test / DLRN FAIL fix https://review.opendev.org/#/c/736816/2

~~[14:38:52] <weshay|ruck> preventative action https://review.opendev.org/#/c/736823/3~~ - merging

0/<soniya>looking at periodic jobs... </soniya>

<wes>noticed ovn-fs010 failed.. debugging and waiting another failure to raise a lp. https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/2b62fd0/</wes>

<rlandy>pinged kforde re: stacks in rdocloud</rlandy>

<rlandy>rdocloud bmc-template image updated per sagi's email</rlandy>

<rlandy> do we need to w+ ~~https://review.opendev.org/#/c/736816/ [DONE]~~ and https://review.opendev.org/#/c/736823?/</rlandy>

<rlandy>https://logserver.rdoproject.org/openstack-component-security/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-component-security-promote-consistent-to-component-ci-testing/3082b35/job-output.txt
 2020-06-19 15:05:00.530944 | primary | /home/zuul/workspace/dlrnapi_venv/bin/activate: line 31: $1: unbound variable
 Testing: https://review.rdoproject.org/r/28170 Add VIRTUAL_ENV_DISABLE_PROMPT to avoid unbound error
 
^^ didn't work - revert did ... https://review.rdoproject.org/r/#/c/28171/
 </rlandy>
 
 <rlandy> multinode-ip failures:
*  https://bugs.launchpad.net/tripleo/+bug/1884287
*  Failed to download packages: Cannot download Packages/python3-qrcode-core-5.1-12.module_el8.2.0+370+b142e101.noarch.rpm: All mirrors were tried
https://bugs.launchpad.net/tripleo/+bug/1884570
</rlandy>
<rlandy>
https://review.rdoproject.org/r/28173 Reduce where OVB jobs are run - overloaded clouds
</rlandy>

### OSP

## June 18th
### TripleO
* https://bugs.launchpad.net/tripleo/+bug/1884115 FTR
<openstack> Launchpad bug 1884115 in tripleo "AH00534: httpd: Configuration error: More than one MPM loaded." [High,Triaged] - Assigned to Emilien Macchi (emilienm)
* https://bugs.launchpad.net/tripleo/+bug/1884138
tripleoclient fails to build on delorean: ModuleNotFoundError: No module named 'tripleo_common.actions.config' 
https://review.opendev.org/#/c/736816/
https://review.opendev.org/#/c/736823/
see Ravi's comment "I think it would need https://review.opendev.org/#/c/736944/"
* Stacks in RDO cloud are VERY stuck - will talk to infra tomorrow (rlandy)
### OSP
