Ruck Rover 2022-10-14 - 2022-10-20

tags: `ruck_rover`

Next RR notes: https://hackmd.io/wtT4lbOSSeuLcRS2aPTQAQ

Previous RR notes: https://hackmd.io/J4_ZyTvITtS51Wvmd5feRw

ruck & rover: marios & dasm

RDO Cockpit / RHOS Cockpit

RDO Promoter / RHOS Promoter

OpenStack Program Meeting 2022

Zuul Status:

Active bugs

https://bugs.launchpad.net/tripleo/+bug/1990480 - Tempest test test_create_update_port_with_dns_domain failure KeyError: 'dns_domain'
https://bugs.launchpad.net/tripleo/+bug/1991093 - compute and network tempest tests failing on fs35 train
https://bugs.launchpad.net/tripleo/+bug/1987092 - Pacemaker performance causes intermittent galera issues in loaded CI env
https://bugs.launchpad.net/tripleo/+bug/1993262 - periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-component-master-validation log pollution leads to intermittent failures
https://bugs.launchpad.net/tripleo/+bug/1993730 - Wallaby c8 and c9 OVB jobs are failing the modify image step - mount point does not exist (steve left notes)

Oct 21

New/Transient/No bug yet:

d/stream

rhel8/16.2 - still hitting registry issues https://bugzilla.redhat.com/show_bug.cgi?id=2135432#c6 & rekicked manually the openstack-periodic-integration-rhos-16.2 1 currently running

centos8 components (ibm cloud) are stuck and holding component lines e.g. https://review.rdoproject.org/zuul/buildset/e41b7c8fbba142b0b0be4d5929ca6739 15 hours in progress

https://bugs.launchpad.net/tripleo/+bug/1984237 -> hitting check and also periodic integration https://review.rdoproject.org/zuul/build/e2c88a92218c4f1f98b4e03010d13b3f

Oct 20

Upstream Integration
- master: 2022-10-20
  - tp: https://review.rdoproject.org/r/c/testproject/+/45352
- wallabyc9: 2022-10-19
  - tp with image mount revert: https://review.rdoproject.org/r/c/testproject/+/45405
- wallaby c8: 2022-10-20
  - revert temp skip: ~~https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/44769~~
  - tp: https://review.rdoproject.org/r/c/testproject/+/45405
- train: 2022-10-17: 2022-10-20
Upstream Integration

tp: https://review.rdoproject.org/r/c/testproject/+/45709 Run build-images.sh failing
- master components:
- wallabyc9 components:
- wallabyc8 components:
- train components:
Downstream:

Blocker:
- https://bugzilla.redhat.com/show_bug.cgi?id=2135432 - containers build push time out
  - https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=incident&sys_id=e14a38db872e999807c9ed3c8bbb35c8
- https://bugzilla.redhat.com/show_bug.cgi?id=2136053
- https://bugzilla.redhat.com/show_bug.cgi?id=2135616 - ovb issue
- https://redhat.service-now.com/help?id=rh_ticket&table=sc_req_item&sys_id=36aeadc78726d9d0d5cc642c8bbb3518&view=ess
- Integration lines:
  - rhos17 on rhel9: promoted 14-oct
  - rhos17.1 on rhel9: promoted 12-oct
    - rerunning 3 failing OVB jobs
  - rhos17.1 on rhel8: promoted 12-oct
    - all passed except mixed rhel
  - rhos16.2: promoted 13-oct
    - all jobs passed expect baremetal - line kicked again now (may need to manually fix baremetal will check that tomorrow)

Oct 19

Upstream Integration
- master: 2022-10-14
  - All jobs blocked: ~~https://bugs.launchpad.net/tripleo/+bug/1993343~~
  - tp: https://review.rdoproject.org/r/c/testproject/+/45352 to watch
- wallabyc9: 2022-10-19
  - All jobs blocked: ~~https://bugs.launchpad.net/tripleo/+bug/1993343~~
  - tp: ~~https://code.engineering.redhat.com/gerrit/c/testproject/+/430001~~
- wallaby c8: 2022-10-16
  - tp: https://review.rdoproject.org/r/c/testproject/+/45352
    - fs001 failed tempest tests. If rerun doesnt have the same failed test we can skip and promote
    - tp with different sets of tempest failure:
      - https://review.rdoproject.org/zuul/build/6615dc954f3d40d39d3d6413728f2f2a
      - https://review.rdoproject.org/zuul/build/2dbb016c110f46899ba0203597704ed6
    - Skip and promote patch: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45695
- train: 2022-10-17:
  - tp: ~~https://review.rdoproject.org/r/c/testproject/+/45690~~
  - tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/430001 to watch
  - tp: https://review.rdoproject.org/r/c/testproject/+/45701 to watch
  - tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/431902 to watch
Upstream Integration
- master components:
  - https://bugs.launchpad.net/tripleo/+bug/1993343
- wallabyc9 components:
- wallabyc8 components:
- train components:
Downstream:

Blocker: https://bugzilla.redhat.com/show_bug.cgi?id=2135432#c3
- Integration lines:
  - rhos17 on rhel9: promoted 14-oct
  - rhos17.1 on rhel9: promoted 12-oct
  - rhos17.1 on rhel8: promoted 12-oct
  - rhos16.2: promoted 13-oct

pingd on rhos-ops:

     ~~~
     <bhagyashris> Hi Team, we are still hitting retry limit issue and that is causing promtion blocker at downstream'
    <bhagyashris> 2022-10-19 05:42:29.219305 | primary |   "msg": "Failure downloading http://download.devel.redhat.com/rcm-guest/puddles/OpenStack/rhos-release/rhos-release-latest.noarch.rpm, Request failed: <urlopen error [Errno -2] Name or service not known>",
    
    <bhagyashris> fbo, wznoinsk|ruck ^
    <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-cloudops/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-9-scenario002-standalone-cloudops-rhos-17.1/ab8ba26/job-output.txt

    <dpawlik> cc kforde ^^
    <dpawlik> do we have some network issues?

    <dpawlik> only one outage topic is related to network: https://groups.google.com/u/0/a/redhat.com/g/outage-list/c/h8-ZkLuspxk 
    <dpawlik> and its not related
    <dpawlik> bhagyashris: did you hold the node and check if its reachable?

    <bhagyashris> dpawlik, currently in the running integration line job we are hitting this issue 
    <bhagyashris> 2022-10-18 18:03:17.050564 | TASK [get_hash : get md5 file]
    <bhagyashris> 2022-10-18 18:03:37.609246 | primary | ERROR
    <bhagyashris> 2022-10-18 18:03:37.609628 | primary | {
    <bhagyashris> 2022-10-18 18:03:37.609678 | primary |   "dest": "/home/zuul/workspace/delorean.repo.md5",
    <bhagyashris> 2022-10-18 18:03:37.609705 | primary |   "elapsed": 20,
    <bhagyashris> 2022-10-18 18:03:37.609732 | primary |   "msg": "Request failed: <urlopen error [Errno -2] Name or service not known>",
    <bhagyashris> 2022-10-18 18:03:37.609781 | primary |   "url": "https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17-1/promoted-components/delorean.repo.md5"
    <bhagyashris> 2022-10-18 18:03:37.609805 | primary | }
    <bhagyashris> locally it's accessible "https://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp17-1/promoted-components/delorean.repo.md5
    <bhagyashris> not sure why it's causing issue on job node 
    * evallesp (~evallesp@10.39.194.108) has joined
    <bhagyashris> dpawlik, added this job https://code.engineering.redhat.com/gerrit/c/testproject/+/431169/6/.zuul.yaml on node hold
    <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/status/change/431169,6
    <bhjf> Title: Zuul (at sf.hosted.upshift.rdu2.redhat.com)
    <dpawlik> bhagyashris: on vexxhost we have partially same issue: on some host it can not reach trunk.rdoproject.org server
    <dpawlik> they fix that, it was something wrong with the host
     ~~~

List of hashes that we can promote:
- rhos16-2 on rhel8:
  - ac7a781ab85cfc2c9b1a1b6aad4a50ab:
    - Missing Jobs:
      - periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2
      - periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset035-internal-rhos-16.2
      - periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-rhos-16.2
      - periodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2
- rhos17-1 on rhel9:
  - 85b7a0a2481df9e73096a6bc88dc71f7
    - Missing Jobs:
      - periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1
      - periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17.1
      - periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-rbac-internal-rhos-17.1
      - periodic-tripleo-ci-rhel-9-ovb-1ctlr_2comp-featureset020-internal-rhos-17.1
- Component line:

Oct 18

Upstream Integration
- master: 2022-10-13
  - tp: https://review.rdoproject.org/r/c/testproject/+/45352
  - Again a new bug: https://bugs.launchpad.net/tripleo/+bug/1993343
- wallabyc9: 2022-10-13
  - tp: ~~https://review.rdoproject.org/r/c/testproject/+/45405~~
- wallaby c8:
  - tp: https://review.rdoproject.org/r/c/testproject/+/45352
- train: 2022-10-13
  - tp: https://review.rdoproject.org/r/c/testproject/+/45690
Upstream Integration
- master components:
- wallabyc9 components:
- wallabyc8 components:
- train components:
Downstream:
- Integration lines:
  - rhos17 on rhel9: promoted 14-oct
  - rhos17.1 on rhel9: promoted 12-oct
    - containers build push job is failing: https://bugzilla.redhat.com/show_bug.cgi?id=2135432
  - rhos17.1 on rhel8: promoted 12-oct
  - rhos16.2: promoted 12-oct
    - containers build push job is failing: https://bugzilla.redhat.com/show_bug.cgi?id=2135432
    - ovb jobs are failing with RETY_LIMIT: https://bugzilla.redhat.com/show_bug.cgi?id=2135616

pinged on rhos-ops: ~~~

        <bhagyashris> evallesp, wznoinsk|ruck hey currently we are facing this issue for ovb jobs https://bugzilla.redhat.com/show_bug.cgi?id=2135616

        <bhagyashris> and this one https://bugzilla.redhat.com/show_bug.cgi?id=2135432  we hit on friday and yesterday on container build push job looks like it's intermittent but some how feeling like infra is not stable

        <bhagyashris> and one more is "Could not resolve host: download.devel.redhat.com" is also coming intermittently 
        <bhagyashris> could you please check

        <dpawlik> bhagyashris: did you check outage list
        <dpawlik> if there are some DNS maintenance? 
        <apevec> for upshift registry, I pinged internal pnt infra gchat there where rlandy reported registry issues last week, no new replies yet

        <evallesp> Yesterday I found some DNS errors as well... I though it was similar the internal SSO.
        <apevec> bhagyashris (IRC): which nameservers do we have now in resolve.conf ?
        <apevec> there's other thread in pnt-infra gchat about some nameservers not working
        <apevec> > 10.11.142.1  seems to not work
        <apevec> > These are the resolvers within RDU2 near RHOS-D:
        <apevec> nameserver 10.11.5.160
        <apevec> nameserver 10.11.5.19

        <apevec> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-16.2/a2166e8/logs/hostvars-variables.yaml
        <apevec>     ansible_dns:
        <apevec>         nameservers:
        <apevec>         - 10.11.5.19
        <apevec>         - 10.5.30.45
        <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario012-standalone-rhos-16.2/fe9fbf1/logs/undercloud/etc/resolv.conf
        <apevec> nameserver 10.11.5.19
        <apevec> nameserver 10.5.30.45
        <apevec> ok so first one is what pnt-infra said, but what is the other one

        <dpawlik> if someone is wondering why upstream zuul does not take any new request: "2022-10-18 07:29:32,336 DEBUG zuul.GithubRateLimitHandler: GitHub API rate limit (ansible-collections/community.digitalocean, 20166502) resource: core, remaining: 12500, reset: 1666081772"

        <apevec> ah opendev doesn't get some free unlimited account?
        <dpawlik> dunno
        <dpawlik> I don't think they are using GH a lot
        <dpawlik> just a mirror, most things are on opendev side
        <apevec> bhagyashris (IRC): so in which tasks Failed to discover available identity versions happens, can you point to the code and how we can reproduce outside CI job?

        <bhagyashris> apevec, here is the log https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-16.2/a2166e8/job-output.txt
        <bhagyashris> let me pass the taskwhere it failed

        <bhagyashris> some where in ovb-manage: Create stack it failed
        <apevec> is ovb-manage not producing more debug info?
        <bhagyashris> https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/ovb-create-stack.yml#L43
        <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-16.2/a2166e8/logs/bmc-console.log
        <bhagyashris> https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-clients/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-ovb-3ctlr_1comp-featureset001-internal-clients-rhos-16.2/a2166e8/logs/failed_ovb_stack.log

        <marios> apevec: https://bugzilla.redhat.com/show_bug.cgi?id=2135616#c3 keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens
        <bhjf> Bug 2135616: urgent, unspecified, ---, ---, rhos-maint, distribution, NEW , Failed to discover available identity versions when contacting https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3. Attempting to parse version from URL.

        <apevec> bhagyashris (IRC): marios (IRC) https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens is reachable from my laptop on VPN, was it temp failure then, is it working now?
        <apevec> if still failing, can we hold the node ?
        <apevec> but not sure how we do that with an OVB node?
        <apevec> this is failing on OC nodes?

        <apevec> <bhagyashris> "https://sf.hosted.upshift.rdu2...." <- hmm in this case cloud-init failed b/c > [  224.941268] cloud-init[1292]: Failed to start openstack-bmc-baremetal-81610_3.service: Unit not found.
        <apevec> marios (IRC): which machine's console is what we see bmc-console.log ? It's CentOS 7 ??
        <apevec> CentOS Linux 7 (Core)
        <apevec> Kernel 3.10.0-1127.10.1.el7.x86_64 on an x86_64
        <apevec> and using public centos mirrors: bmc-81610 login: [   54.231122] cloud-init[1292]: * base: centos.mirrors.hoobly.com
        <apevec> [   54.232969] cloud-init[1292]: * centos-ceph-nautilus: mirror.steadfastnet.com
        <apevec> [   54.233245] cloud-init[1292]: * centos-nfs-ganesha28: mirror.siena.edu
        <apevec> [   54.234583] cloud-init[1292]: * centos-openstack-stein: centos.hivelocity.net
        <apevec> [   54.235488] cloud-init[1292]: * centos-qemu-ev: mirror.umd.edu
        <apevec> [   54.236472] cloud-init[1292]: * epel: forksystems.mm.fcix.net
        <apevec> [   54.238592] cloud-init[1292]: * extras: mirror.umd.edu
        <apevec> [   54.239339] cloud-init[1292]: * updates: mirror.datto.com
        <apevec> then using https://trunk.rdoproject.org/centos7/current/

        <apevec> after this keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to https://rhos-d.infra.prod.upshift.rdu2.redhat.com:13000/v3/auth/tokens: ('Connection aborted.', error(104, 'Connection reset by peer'))
        <apevec> it continues like error didn't happen, should probably stop, are those systemd unit files generated on the fly?
        <apevec> [  224.790606] cloud-init[1292]: + systemctl daemon-reload
        <apevec> [  224.887689] cloud-init[1292]: + systemctl enable config-bmc-ips
        <apevec> [  224.901780] cloud-init[1292]: Failed to execute operation: No such file or directory
        <apevec> [  224.902855] cloud-init[1292]: + systemctl start config-bmc-ips
        <apevec> [  224.907713] cloud-init[1292]: Failed to start config-bmc-ips.service: Unit not found.
        <marios|call> apevec: yeah the bmc is still in c7 
        <apevec> sigh
        <apevec> that's unsupported ;)
        <apevec> I mean really, OSC must be old, also it should retry few times
        <apevec> https://trunk.rdoproject.org/centos7/current/ is 2020-04-13
        <apevec> in any case, bhagyashris (IRC) do we still see that failure or is intermittent ?
        <apevec>  * in any case, bhagyashris (IRC) do we still see that failure or is it random ?
        <apevec> I still don't have a clear case to report to PSI ops

        <apevec> before I start looking deeper into OVB code, is stable/2.0 the branch currently in use, based on C7 ?
        <apevec> and new dev is in master, based on CS9 ?
        <apevec> (while at it, what are the current blockers to move OVB to CS9  ?)
        ~~~

Component line:

Oct 17

master: 2022-10-13 * tp: https://review.rdoproject.org/r/c/testproject/+/45352 * to check: fs020 * fs35: * /usr/share/openstack-tripleo-heat-templates/ci/environments/ovb-ha.yaml–disable-protected-resource-types file not found * probably due to a typo? * Bug: https://bugs.launchpad.net/tripleo/+bug/1993139 * Fix WIP: https://review.opendev.org/c/openstack/tripleo-quickstart/+/861590
wallabyc9: 2022-10-13 * tp: https://code.engineering.redhat.com/gerrit/c/testproject/+/430001 * tp: https://review.rdoproject.org/r/c/testproject/+/45405
train: 2022-10-13 * tp: https://review.rdoproject.org/r/c/testproject/+/45407 * fs35 tempest failures

Downstream:

Retry_limit : https://osp-trunk.hosted.upshift.rdu2.redhat.com/ or https://bootstrap.pypa.io/pip/3.6/get-pip.py or https://docker-registry.upshift.redhat.com/v2/tripleorhos-17-1-rhel-8/openstack-heat-base/blobs/sha256:3574ac17976440865039b44e5bfd58ca7e504d41eb5f357e4623b234e54b9148 is not reachable

pinged fbo on rhos-ops

      <bhagyashris> Hi Team we are currently facing retry_limit issue on most of the jobs at downstream due to ("msg": "Status code was -1 and not [200]: Request failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)>",)
<bhagyashris> fbo ^ https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-8-rhos-16.2-promote-promoted-components-to-tripleo-ci-testing-internal/15ae3df/job-output.txt
<fbo> bhagyashris (IRC) That's probably fixed already
<bhagyashris> fbo, we recently hit with same issue https://code.engineering.redhat.com/gerrit/c/testproject/+/431140/4#message-3fcb221e2c8e36f2c63be255c0b0c7160c43edeb
<bhagyashris> sorry let me share new https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17.1-rhel9/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-rhel-9-rhos-17.1-promote-promoted-components-to-tripleo-ci-testing-internal/a203734/job-output.txt
<bhagyashris> ignor first link
<bhagyashris> https://code.engineering.redhat.com/gerrit/c/testproject/+/431140/4#message-3732b5c2fbf3f9b548471cd036d6c5e69aa69166
<fbo> it was yesterday
<evallesp> I've updated the certs for osp-trunk prod environment. Now we are getting autorenew certs by: https://gitlab.cee.redhat.com/ansible-playbooks/idm-client-playbooks/-/blob/main/inventory/group_vars/osp-dlrn.yml So I am going to remove its certs deployment by sf-infra ansible-playbook (as we are not going to renew them any more).
<bhagyashris> fbo, https://sf.hosted.upshift.rdu2.redhat.com/logs/40/431140/4/check/periodic-tripleo-rhel-8-rhos-16.2-promote-promoted-components-to-tripleo-ci-testing-internal/a0873e2/job-output.txt
<fbo> curl https://osp-trunk.hosted.upshift.rdu2.redhat.com/ works now
<bhagyashris> fbo, ok let me recheck again

Integration lines:
- rhos17 on rhel9: promoted 14-oct
- rhos17.1 on rhel9: promoted 12-oct
  - containers build push job is failing: https://bugzilla.redhat.com/show_bug.cgi?id=2135432
- rhos17.1 on rhel8: promoted 12-oct
- rhos16.2: promoted 12-oct
  - containers build push job is failing: https://bugzilla.redhat.com/show_bug.cgi?id=2135432
Component line:

Oct 14

Upstream Integration
- master: 2022-10-13
  - tp: ~~https://review.rdoproject.org/r/c/testproject/+/45352~~
- wallabyc9: 2022-10-13
  - tp: https://review.rdoproject.org/r/c/testproject/+/45405
- train: 2022-10-13
  - tp: https://review.rdoproject.org/r/c/testproject/+/45407
Upstream Integration
- master components:
  - ~~https://review.rdoproject.org/r/c/testproject/+/45657~~
- wallabyc9 components:
  - ~~tp: https://review.rdoproject.org/r/c/testproject/+/45420~~
- wallabyc8 components:
  - tp: https://review.rdoproject.org/r/c/testproject/+/45659
- train components:
  - ~~https://review.rdoproject.org/r/c/testproject/+/45660~~
Downstream:
- Retry_limit : https://osp-trunk.hosted.upshift.rdu2.redhat.com/ or https://bootstrap.pypa.io/pip/3.6/get-pip.py or https://docker-registry.upshift.redhat.com/v2/tripleorhos-17-1-rhel-8/openstack-heat-base/blobs/sha256:3574ac17976440865039b44e5bfd58ca7e504d41eb5f357e4623b234e54b9148 is not reachable
- Integration lines:
  - rhos17 on rhel9: promoted 09-oct
    - fs001, containers-multinode and bm-fs001 job is failing re-running those jobs here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803
      - fs001 and containers multinode job passed.
      - now bm-fs001-envB job is still failing re-running that job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431169 (looks like issue is due to outage)
  - rhos17.1 on rhel9: promoted 12-oct
  - rhos17.1 on rhel8: promoted 12-oct
    - Retry : https://bootstrap.pypa.io/pip/3.6/get-pip.py not reachable
    - containers build failing re-running here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431140
      - we are still hitting time out issue
        
        2022-10-14 07:16:56.005258 | primary | Head "https://docker-registry.upshift.redhat.com/v2/tripleorhos-17-1-rhel-8/openstack-heat-base/blobs/sha256:3574ac17976440865039b44e5bfd58ca7e504d41eb5f357e4623b234e54b9148": dial tcp: lookup docker-registry.upshift.redhat.com on 10.5.30.45:53: read udp 192.168.200.26:44983->10.5.30.45:53: i/o timeout
      - in the second re-run it failed with 503 so again it's like outage issue so will need to keep eye on https://code.engineering.redhat.com/gerrit/c/testproject/+/429803
  - rhos16.2: promoted 12-oct
- Component line:
  - check all the component line jobs for all the above releases and hit the testproject patch for failing jobs : https://code.engineering.redhat.com/gerrit/c/testproject/+/431159
    - All the failing jobs passed here: https://code.engineering.redhat.com/gerrit/c/testproject/+/431159 except the container-multinode-client-rhso16.2 job so re-running that job here: https://code.engineering.redhat.com/gerrit/c/testproject/+/429803

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`	在筆記中貼入程式碼
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.

Ruck Rover 2022-10-14 - 2022-10-20

tags: ruck_rover

Next RR notes: https://hackmd.io/wtT4lbOSSeuLcRS2aPTQAQ

Previous RR notes: https://hackmd.io/J4_ZyTvITtS51Wvmd5feRw

ruck & rover: marios & dasm

Active bugs

Oct 21

New/Transient/No bug yet:

d/stream

rhel8/16.2 - still hitting registry issues https://bugzilla.redhat.com/show_bug.cgi?id=2135432#c6 & rekicked manually the openstack-periodic-integration-rhos-16.2 1 currently running

centos8 components (ibm cloud) are stuck and holding component lines e.g. https://review.rdoproject.org/zuul/buildset/e41b7c8fbba142b0b0be4d5929ca6739 15 hours in progress

https://bugs.launchpad.net/tripleo/+bug/1984237 -> hitting check and also periodic integration https://review.rdoproject.org/zuul/build/e2c88a92218c4f1f98b4e03010d13b3f

Oct 20

Oct 19

Oct 18

Oct 17

Oct 14

tags: `ruck_rover`