Pipeline

• Pipeline def
  • https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/305840
• Add pipeline
  • https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/306035

Container build (Green run)

Patches:

• testproject: https://code.engineering.redhat.com/gerrit/c/testproject/+/299390

• Add exclude for network-scripts in pre for bug https://bugzilla.redhat.com/show_bug.cgi?id=2043678

• Need to move away from using nightly.

  • https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/307838
  • https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/307840
  • 307913: Define downstream_cert_install_command release file | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/307913

• Job def: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/299521

* Release file: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/300678

* 822503: Introduce iptables_package var |https://review.opendev.org/c/zuul/zuul-jobs/+/822503

* Only add temp repos on el8 - https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/302181

Image Build(Green run):

Patches:

• Main patch with job def: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/302630

• 307913: Define downstream_cert_install_command release file | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/307913 * Add exclude for network-scripts in pre for bug https://bugzilla.redhat.com/show_bug.cgi?id=2043678

• 823365: Preliminary work to support CentOS 9 Stream | https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/823365

* Release file: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/300678

* 822503: Introduce iptables_package var | https://review.opendev.org/c/zuul/zuul-jobs/+/822503

• 824598: Add configs for RHEL 9 image | https://review.opendev.org/c/openstack/tripleo-common/+/824598 (Merged)

  • Above cherry-pick: https://review.opendev.org/c/openstack/tripleo-common/+/825227

* 824660: Fixes for centos-9-stream efi behaviour | https://review.opendev.org/c/openstack/diskimage-builder/+/824660

• openstack/tripleo-puppet-elements master: Add config for RHEL9 in overcloud-base element https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/825259 (Merged)

  • cherry-pick https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/825850

• openstack/tripleo-puppet-elements master: Add config for EL9 in overcloud-agent element https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/825263 (Merged)

  • https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/825748 (cherry-pick)

• 825696: delorean-repo remove yum_plugin_priorities_package | https://review.opendev.org/c/openstack/tripleo-image-elements/+/825696 Merged

  • https://review.opendev.org/c/openstack/tripleo-image-elements/+/825749 cherry-pick

Issues -

Bug#1: https://bugzilla.redhat.com/show_bug.cgi?id=2043678 (openstack-network-scripts and network-script conflict)

• fix: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/305867
• Also need to add same workaround in container and image build pre.

Bug#2 https://bugzilla.redhat.com/show_bug.cgi?id=2043922 - Standalone deployment on RHEL9 failing Error: Evaluation Error: Error while evaluating a Virtual Query, Could not autoload puppet/type/pcmk_resource: Could not autoload puppet/provider/pcmk_resource/default: cannot load such file – rexml/document

• Solved with new puppet-headless

Bug#3 https://bugzilla.redhat.com/show_bug.cgi?id=2044776 - RHOSP17 Standalone deployment failing with Error: Evaluation Error: Error while evaluating a Function Call, undefined method `escape' for URI:Module (file: /etc/puppet/modules/nova/manifests/migration/libvirt.pp, line: 201, column: 49) on node standalone

• Need new puppet-stdlib
• https://code.engineering.redhat.com/gerrit/c/puppet-stdlib/+/306409 - Add .gitreview
• https://code.engineering.redhat.com/gerrit/c/puppet-stdlib/+/306411 -[DOWNSTREAM ONLY] Replacing URI.escape with URI-DEFAULT_PARSER

Bug#4 https://bugzilla.redhat.com/show_bug.cgi?id=2048488 - RHOSP17 Standalone deployment failing to start rabbitmq_wait_bundle container, RabbitMQ version cannot run on Erlang 23.0.4: minimum required version is 23.2 (erts 11.1

• fixed with new erlang

Issues -

1 : Fetch-zuul-cloner-fork causing issue

https://sf.hosted.upshift.rdu2.redhat.com/logs/69/305869/2/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17/5b28867/job-output.txt

2022-01-22 03:53:15.085655 | TASK [fetch-zuul-cloner-fork : Install zuul-cloner shim dependencies]
2022-01-22 03:53:15.889468 | primary | ERROR
2022-01-22 03:53:15.889907 | primary | {
2022-01-22 03:53:15.889957 | primary |   "msg": "Failed to find required executable virtualenv in paths: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin"
2022-01-22 03:53:15.889988 | primary | }

Fix: https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/305898

• No package epel-release available." fix: https://review.rdoproject.org/r/c/config/+/39141

• Node provisioning failure - Fixed

• Overcloud deploy failing with "Error: /Stage[main]/Pacemaker::Corosync/User[hacluster]/password: change from [redacted] to [redacted] failed: chpasswd said chpasswd: cannot execute /usr/sbin/sss_cache: Permission denied" - We have workaround - selinux-policy in override repo

• Issue - build_image.sh taking wrong release:

  • 312889: Add dib_release in release file. | https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/312889

Ongoing issues

• Podman container network issue: http://pastebin.test.redhat.com/1034041

jboyer[m]> we're doing monthly updates of the RHEL 9 Beta channels
<jboyer[m]> another update will come in April

• https://bugzilla.redhat.com/show_bug.cgi?id=2060047 - RHOSP17 overcloud-full image build failing with Error: "Problem: The operation would result in removing the following protected packages: grub2-pc"

• https://bugzilla.redhat.com/show_bug.cgi?id=2060027 - RHOSP17 on RHEL-9 jobs failure with Error: nothing provides chkconfig and bc needed by openstack-network-scripts-10.11-1.el9osttrunk.x86_64

  • Looks like auto-fixed - this morning.

Bug #1:- Standalone deployment on sc07 failing to start rabbitmq_wait_bundle container, rabbitmq crash with error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}} - https://bugzilla.redhat.com/2050624 Fix: https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/309499 https://code.engineering.redhat.com/gerrit/c/openstack/sf-config/+/309607

Bug #2:- Hitting same bug which we hit for wallaby: https://bugs.launchpad.net/tripleo/+bug/1959582

Awaiting new tripleo component

https://sf.hosted.upshift.rdu2.redhat.com/logs/95/311995/1/check/periodic-tripleo-ci-rhel-9-containers-multinode-rhos-17/d15a216/logs/undercloud/home/zuul/undercloud_install.log

2022-02-17 06:09:20.623820 | fa163e41-308d-b41c-929e-00000000075a |      FATAL | Ensure system is NTP time synced | undercloud | error={"changed": true, "cmd": ["chronyc", "waitsync", "30"], "delta": "0:04:50.219164", "end": "2022-02-17 06:09:20.594276", "msg": "non-zero return code", "rc": 1, "start": "2022-02-17 06:04:30.375112", "stderr": "", "stderr_lines": [], "stdout": "try: 1, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 2, refid: 00000000, correction: 0.000000000, skew: 0.000\ntry: 3, refid: 00000000, correction:

Testing with fix: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/312419