Ruck and Rover notes #NovDec2020

tags: ruck_rover

Important links for ruck rover's ruck/rover links to help Ruck Rover - Unified Sprint #36 (NovDec 2020 timeframe) Dates: Nov 11 - Jan 11

Tripleo CI team ruck|rover: marios, rlandy, wes OSP CI team ruck|rover:

Previous notes: https://hackmd.io/a_PhAlijTr-KTWHzGuf3qQ

RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

Please try this out instead of cutting pasting bugs. Give feedback to wes please

on-going issues

TripleO

gate

periodic / 3rd party

http://dashboard-ci.tripleo.org/d/HkOLImOMk/upstream-and-rdo-promotions?orgId=1

OSP

Reviews / Fixes

PATCHES

Bugs reported

PATCHES

add dates in decending order so the latest date is at the top. Break out TripleO and OSP sections.

Jan 25 - Jan 29

Jan 18 - Jan 22

Jan 7 - Jan 15

Jan 6th

Jan 5th

  • periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master (recent two failures)

https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-integration-main&job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master

test project passed - https://review.rdoproject.org/r/#/c/31424/

Dec 31st

Dec 30th

  • ussuri

https://bugs.launchpad.net/tripleo/+bug/1909658

periodic-tripleo-ci-centos-8-singlenode-featureset050-upgrades-ussuri is failing during undercloud install with message - error checking if pulling from registry is blocked: unable to parse the registries configuration (/etc/containers/registries.conf): mixing sysregistry v1/v2 is not supported

  • stein

Fixed https://bugs.launchpad.net/tripleo/+bug/1909671

Centos 7 based jobs are failing on TASK [Enable container-tools and disable rhel-modules] with error: /bin/sh: line 1: dnf: command not found

Dec 29th

  • Component pipeline https://bugs.launchpad.net/tripleo/+bug/1909574 "periodic-tripleo-ci-centos-8-scenario010-kvm-standalone-octavia-master/victoria/ussuri failing on tempest tests with SSHTimeout issue with error message "socket.timeout: timed out""

  • train promotion blocker

Fixed https://bugs.launchpad.net/tripleo/+bug/1909582 "Introspection failing in train branch with message: mistral.exceptions.InvalidActionException: Failed to find action [action_name=baremetal_introspection.get_status]\n', 'introspection_attempt': 0}" [Critical,Triaged] [High,Triaged]

Dec 28th

Dec 24th

Dec 23rd

Dec 22nd

Dec 21st

Dec 18th

AMOL's notes

Dec 17th

Dec 16th

Dec 15th

Dec 14th

summary of status.. shitty week last week..

pypy depsolv, ubi-container in check, centos 8.2 - 8.3, content-provider patches all contributing to failures upstream atm. We're through most issues atm. OVB in check is full red, but as I said looks to be coming back to green via periodic promotions.

​​​​* pypy - lower-constraints https://review.opendev.org/q/topic:%2522tripleo-lower%2522+(status:open+OR+status:merged)
​​​​* ubi-8 - container-builds e.g. https://review.opendev.org/c/openstack/tripleo-ci/+/766839
​​​​* c8-train -> ussuri upgrade jobs failing on pacemaker mismatch - requires promotion to fix or method to inject change to content-providers. 
​​​​
​​​​* content-provider ip failures https://bugs.launchpad.net/tripleo/+bug/1907657
​​​​* overcloud-full-hardened moved to nv until we promote all c8 branches https://bugs.launchpad.net/tripleo/+bug/1907457

Dec 10th

Need help landing https://review.opendev.org/c/openstack/tripleo-common/+/766516

Please review: https://review.opendev.org/c/openstack/tripleo-ci/+/766621

Dec 8th

Please focus on unblocking: https://review.opendev.org/#/q/owner:"amolkahat+%253Camolkahat%2540gmail.com%253E"+branch:stable/ussuri

gate - release issue https://bugs.launchpad.net/tripleo/+bug/1907122 https://review.opendev.org/q/topic:"release-aware-gate"+(status:open OR status:merged)

Not sure if https://bugs.launchpad.net/tripleo/+bug/1907006 is related to the above issue or not

Dec 7th

new bugs in https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

Dec 4th & earlier ongoing things

https://bugs.launchpad.net/tripleo/+bug/1906769 periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master failing intermitently with paramiko.ssh_exception.AuthenticationException: Authentication failed

https://bugs.launchpad.net/tripleo/+bug/1883843 tripleo-ci-centos-8-scenario007-multinode-oooq-container is having intermittently timeouts

https://bugs.launchpad.net/tripleo/+bug/1906486 FS01 ovb job fails with failed to stat() /home/zuul/overcloud-full.raw

https://bugs.launchpad.net/tripleo/+bug/1905386 Centos 7 train jobs failing with pymod2pkg requiring python 3.6

https://bugs.launchpad.net/tripleo/+bug/1903508 periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

https://bugs.launchpad.net/tripleo/+bug/1903882 tripleo-ci-centos-8-content-provider failed on TASK [container-build : Pull failed containers from RDO registry] with error was: 'dict object' has no attribute 'split'

stop here eveything still open is duplicated above this line

Dec 3rd

Dec 2nd

Dec 1st

Nov 30th

Nov 27th

Nov 26th

  • Nothing much, a lot of stuff is broken right now
  • 90+ gate failures

Nov 25th

Nov 24th

Nov 23th

  • ussuri gate jobs still failing
  • ussuri repo-setup
  • introspection bug 1904936 is still a thing
  • certificate tempest failure on victoria - 1903508 (waiting to merge)
  • cold migration bug 1903033 - waiting to merge
  • centos-8-scenario002-standalone ssh time out bug 1904223 (unresolved)
    • affects victoria promotion?
  • ensure private network exists bug 1902048 >> FIXED
  • fs037-stein >> FIXED
  • bug 1905036 (scen10 ovn) >> FIXED
  • pulling containers from docker.io (stein) >> FIXED

(details below…)

Nov 20th

new/transient/not filed yet

periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently: >> fixed fs037-stein is green: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein
ussuri gate is blocked on our content provider stuff merging STILL!
scen10-ovn

https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect 15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/

pulling containers from docker.io

https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io

ussuri repo-setup

https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0 waiting to merge: https://review.opendev.org/c/openstack/tripleo-quickstart/+/763633

https://bugs.launchpad.net/tripleo/+bug/1905036 Scenario10 octavia jobs failing deploy stack creation with "Error Referenced Attribute (OctaviaBase update_tasks) is incorrect

info

15:40 < beagles> rlandy|rover: marios|ruck https://review.opendev.org/#/c/763561/

https://bugs.launchpad.net/tripleo/+bug/1903581 tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io

info

https://bugs.launchpad.net/tripleo/+bug/1905034 repo-setup fail: undercloud upgrade for ussuri fails on package dep python3-openstackclient >= 5.2.0

Nov 19th

https://bugs.launchpad.net/tripleo/+bug/1904936 openstack.exceptions.ResourceNotFound: No Introspection found for node, Could not find node

https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

info

https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'

info

https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]

info

https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

new/transient/not filed yet

chkumar working on rdo->vexx remaining jobs

09:32 < chandankumar> marios: sshnaidm I am tracking rdocloud nodeset removal here https://etherpad.opendev.org/p/remove-rdocloud-jobs

still chasing a queens promotion
Merged Update rdo-openvswitch in ussuri to support ovs/ovn2.13 NFV SIG https://review.rdoproject.org/r/30774

12:07 < ykarel> marios|ruck, fyi ^ in case u see something wron gin ussuri jobs, let us know 12:07 < ykarel> we tested multiple jobs and it went good, but just in case if there is some issue

https://review.rdoproject.org/r/#/c/30776/ - network component still very red
periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein is still failing consistently:

https://logserver.rdoproject.org/25/25325/67/check/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-stein/d358af8/logs/undercloud/home/zuul/overcloud_update_prepare.log.txt.gz

Should have been fixed by: https://review.opendev.org/#/c/762871/

Nov 18th

RUCK / ROVER BUGS: https://docs.google.com/spreadsheets/d/1M1U-ekjEsec-bRjRq7q5rzjbJWKE2uT-ESX4SkeC0Uc/edit#gid=0

Please try this out instead of cutting pasting bugs. Give feedback to wes please

https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

info

https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'

info

https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]

info

https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

info
  • Moved fs19 and fs18 to a single playbook. May be fixed

https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source

info

blocks victoria gate fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112

new/transient/not filed yet

ussuri just missed promotion
Vexx mirrors outdated

Nov 17th

https://bugs.launchpad.net/tripleo/+bug/1904565 standalone-upgrade-victoria and undercloud-upgrade-victoria failing during upgrade pulling from wrong source

info

blocks victoria gate fix at https://review.opendev.org/#/c/763005/ testing with https://review.rdoproject.org/r/31112

https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

info

https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'

info

https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]

info

https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

info
  • Moved fs19 and fs18 to a single playbook. May be fixed

https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]

info

reopened bug fix at https://review.rdoproject.org/r/#/c/31114/ tested at https://review.rdoproject.org/r/#/c/25325/ @marios pls confirm that the upgrades/updates jobs are ok with the one playbook for multinode.

new/transient/not filed yet

undercloud-upgrade & standalone-upgrade victoria problem filed https://bugs.launchpad.net/tripleo/+bug/1904565

https://7085ed736f9157d6d701-223d8b88d73ea59070ac36b627fdc3bc.ssl.cf1.rackcdn.com/762658/1/check/tripleo-ci-centos-8-undercloud-upgrade-victoria/ad84735/logs/undercloud/home/zuul/undercloud_upgrade.log tripleo_common.image.exception.ImageNotFoundException: Not found image: http://23.253.57.32:5001/v2/tripleovictoria/openstack-keystone/manifests/7f974e10d7184d5fc45445a3073333bd"], "stdout": "", "stdout_lines": []}

vexx infra/introspection issues

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/835c3e6/logs/undercloud/var/log/extra/baremetal_list.txt.gz https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/b99e7f1/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master/2726291/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/970a3ea/job-output.txt https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_1comp-featureset002-master/72f523b/job-output.txt

fs050 victoria jobs are failing:

Related fixes: https://review.opendev.org/#/c/763097/ Make victoria upgrades job non-voting until fixed ~~https://review.rdoproject.org/r/31120 Add undercloud fs050 upgrades to master/v/u periodic ~~ @marios : your thoughts here? where though? are you referring to upstream undercloud upgrade job fixed by https://review.opendev.org/#/c/763005/ ? or periodic job? can't be periodic since you're adding it with https://review.rdoproject.org/r/31120

Nov 16th

https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

info

https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'

info

https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]

info

https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

https://bugs.launchpad.net/tripleo/+bug/1904418 ( IN PROGRESS) buildimage centos7 jobs are failing to fetch mirror info 404

info

https://bugs.launchpad.net/tripleo/+bug/1904483 (Require attention)centos-7 overcloud image build jobs are failing on pre-install.d/03-centos-rt - rt repo no longer exists

info

shoudl be fixed by https://review.opendev.org/#/c/762935 (will need cherry-pick to train, stein, rocky) Related fix on removing rt-repo: https://review.opendev.org/#/c/762926/

https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'

info

https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498)

https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests.

info

https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762871/

Nov 13th

https://bugs.launchpad.net/tripleo/+bug/1903980 ( IN PROGRESS) tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-train job is failing on TASK [undercloud-deploy : Write containers-prepare-parameter.yaml] with error 'dict object' has no attribute 'registry_ip_address_branch'

info

https://review.opendev.org/#/c/761892/ (it is the fix for another bug https://bugs.launchpad.net/tripleo/+bug/1903498)

https://bugs.launchpad.net/tripleo/+bug/1903508 (IN PROGRESS ) periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

info

https://bugs.launchpad.net/tripleo/+bug/1903033 ( ROUTED to migi ) periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master is failing while executing cold migration & resize tempest tests with Stderr: 'mkdir: cannot create directory ‘/var/lib/nova/instances/<uuid>’: Permission denied\n'

info

https://bugs.launchpad.net/tripleo/+bug/1904223 (TEST SKIPPED ) centos-8-scenario002-standalone failing tempest - SSH timed out

https://bugs.launchpad.net/tripleo/+bug/1902048 ( In Progress ) Network component jobs is failing on tempest TASK [os_tempest : Ensure private network exists]

info

https://bugs.launchpad.net/tripleo/+bug/1903996 (low priority) Queens periodic jobs are failing on tempest test: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps with testtools.matchers._impl.MismatchError: 'ACTIVE' != u'DOWN': - FloatingIP: is at status: DOWN. failed to reach status: ACTIVE

https://bugs.launchpad.net/tripleo/+bug/1903993 (IN PROGRESS ) Intemittenly tripleo-ci-centos-8-standalone-upgrade-ussuri timeouts while running tempest tests.

info

https://bugs.launchpad.net/tripleo/+bug/1901996 (PASSING TO SWAPNIL ) tripleo-ci-centos-8-scenario010-standalone is failing on tempest test with error 'Load Balancer is immutable and cannot be updated.'

https://bugs.launchpad.net/tripleo/+bug/1903581 ( In Progress )tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates is pulling containers from docker.io - fixed by https://review.opendev.org/#/c/762547/

Additional fix: https://review.opendev.org/#/c/762871/

https://bugs.launchpad.net/tripleo/+bug/1903498 tripleo-ci-centos-8-undercloud-upgrade-ussuri is failing RuntimeError: Update extra packages failed: b'sudo: yum: command not found\n

info

https://bugs.launchpad.net/tripleo/+bug/1904214 centos-7 train jobs are failing on dependency resolution: glibc-common = 2.17-307.el7.1 and python3-libs(x86-64) = 3.6.8-13.el7 * centos7 promotion blocker

https://bugs.launchpad.net/tripleo/+bug/1903961 Intermittently some jobs are timing out while gathering facts on different tasks : [tripleo-inventory : Ensure gather_facts has been run against localhost] or [validate-undercloud : gather facts used by role]

.

promotion status

victoria promotion:

Last promotion 10th Nov Last buildset - https://review.rdoproject.org/zuul/buildset/f8e8812c49c14d2c8915af307d85731c

ussuri promotion:

Last promotion 11/13

train promotion:

<amoralej> yes, 7.9.20009 was pushed yesterday to mirrors

<rlandy|rover> that's probably it

<amoralej> let me dig into it

<jpena> amoralej: that server mounts opendev's AFS, just like the rest of the mirrors we have in opendev <jpena> looking at https://grafana.opendev.org/d/HxrNXt2Gk/afs?orgId=1, it looks like the mirror was synced 1 hour ago <jpena> we might check where does it sync from, maybe that one seed does not have 7.9 yet <jpena> so, what I see in https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-mirror-update#L51 is that the opendev AFS mirrors sync from mirror.lstn.net/centos <jpena> looking at http://mirror.lstn.net/centos/7/os/x86_64/Packages/?C=M;O=D, it looks like they don't have the 7.9 content yet

Blocking Sagi's patch will solve the issue https://review.opendev.org/#/c/761892/ for https://bugs.launchpad.net/tripleo/+bug/1903498

2020-11-13 12:22:22.839674 | primary | Error: Package: glibc-2.17-307.el7.1.i686 (base)
2020-11-13 12:22:22.839814 | primary |            Requires: glibc-common = 2.17-307.el7.1
2020-11-13 12:22:22.839848 | primary |            Installed: glibc-common-2.17-317.el7.x86_64 (@base)
2020-11-13 12:22:22.839870 | primary |                glibc-common = 2.17-317.el7
2020-11-13 12:22:22.839898 | primary |            Available: glibc-common-2.17-307.el7.1.x86_64 (base)
2020-11-13 12:22:22.839920 | primary |                glibc-common = 2.17-307.el7.1
2020-11-13 12:22:22.839943 | primary | Error: Package: python3-devel-3.6.8-13.el7.x86_64 (base)
2020-11-13 12:22:22.839989 | primary |            Requires: python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 12:22:22.840015 | primary |            Installed: python3-libs-3.6.8-17.el7.x86_64 (@base)
2020-11-13 12:22:22.840037 | primary |                python3-libs(x86-64) = 3.6.8-17.el7
2020-11-13 12:22:22.840062 | primary |            Available: python3-libs-3.6.8-13.el7.x86_64 (base)
2020-11-13 12:22:22.840083 | primary |                python3-libs(x86-64) = 3.6.8-13.el7
2020-11-13 12:22:22.840125 | primary |  You could try using --skip-broken to work around the problem
2020-11-13 12:22:23.195322 | primary |  You could try running: rpm -Va --nofiles --nodigest

rocky promotion:

​​​​* rekicked on vexx https://review.rdoproject.org/r/#/c/29969/16/.zuul.yaml
​​​​* blocked on https://bugs.launchpad.net/tripleo/+bug/1904214

stein promotion:

​​​​* promoted yesterday - leaving for weekend run

queens promotion:

​​​​* testproject there (rdo) https://review.rdoproject.org/r/30906 per https://bugs.launchpad.net/tripleo/+bug/1903996/comments/1 but failing so try vexx
​​​​* testproject there (vexx) https://review.rdoproject.org/r/#/c/25325/
​​​​* blocked on https://bugs.launchpad.net/tripleo/+bug/1904214

Nov 12th

TripleO

Headsup

<amoralej> we are updating ovs/ovn 2.13 for master and victoria today. We've gated it with different jobs so i expect to go smooth but let us know if you find any issue

stein should promote. rocky failed container builds again. need to look at increasing the timeout there

Select a repo