Ruck and rover notes #26

# Ruck and rover notes #26 ###### tags: `ruck_rover` ## INFO AND LINKS :::spoiler ruck/rover primer: https://docs.openstack.org/tripleo-docs/latest/ci/ruck_rover_primer.html Infrared gerrit: https://review.gerrithub.io/q/project:redhat-openstack/infrared Infrared doc: https://infrared.readthedocs.io/en/latest/ Cockpit: http://tripleo-cockpit.usersys.redhat.com/d/9DmvErfZz/cockpit?orgId=1 Internal Cockpit (WIP) http://tripleo-cockpit.usersys.redhat.com/?orgId=1 http://cistatus.tripleo.org/ https://trello.com/b/j4IcIomh/production-chain-escalation http://rhos-release.virt.bos.redhat.com:3030/rhosp Debugging Tools https://docs.google.com/document/d/1VZhje7ZN9sk4E31fYVrPxpqMJGz5ZhHRfhte_RYMXxg/edit# Review.rdoproject.org dashboard: https://review.rdoproject.org/grafana/?orgId=1&var-datasource=default&var-server=registry.rdoproject.org.rdocloud&var-inter=$__auto_interval_inter CentOS pre-release rpm updates for minor releases http://mirror.centos.org/centos/7/cr/x86_64/Packages/ hackmd.io rh-openstack-dev https://hackmd.io/team/rh-openstack-ci?nav=overview Internal software factory: https://sf.hosted.upshift.rdu2.redhat.com upstream rsync mirror logs: files.openstack.org/mirror/logs/rsync-mirrors/centos.log TRELLO RETROSPECTIVE https://trello.com/b/0VFswmht/rdo-infra-retrospective?menu=filter&filter=label:UniSprint21 Internal Dashboard - https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/QE/view/OSP16/ OSP-10 - OSP-16 RHOS INFRA INFRARED ISSUES https://projects.engineering.redhat.com/issues/?filter=34183 CIX escalation https://mojo.redhat.com/docs/DOC-1098748#jive_content_id_CIX_Escalation_Automation_and_email_format CIX board https://trello.com/b/j4IcIomh/production-chain-escalation Nodepool image logs: https://softwarefactory-project.io/nodepool-log/ We may want to move this etherpad to something internal at this point please add your (colored) name here: time to move to hackmd WDYT? +1 (either now - start of the sprint/rr - or in 3 weeks) marios (baby blue) fhubik("green lantern") wznoinsk (orange)Amnon(Marrooned) ::: ## POST BELOW THIS :::warning Dates: April 16 - May 7 Tripleo CI team ruck|rover: Gabriele (panda) && Amol (akahat) OSP CI team ruck|rover (April 24 - May 15): Filip (fhubik), Vadim (vgriner) Previous notes: [link](https://hackmd.io/7MBqFHurTA2e5H8kYRwgag?view) ::: ### Issues to track on-going put these issues in the spoiler. :::danger #### tripleo @akahat FYI.. @arxcruz is investigating the tempest failures in stein. @TheG Please work the networking team to bring https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-8-scenario010-ovn-provider-standalone online. CentOS-7 OVB jobs are RED fs001 https://bugs.launchpad.net/tripleo/+bug/1875731 https://bugs.launchpad.net/tripleo/+bug/1876972 TRAIN: GREEN STEIN: Tempest fail ( arx is looking at it ) ROCKY: Tempest fail ( @arxcruz FYI) QUEENS: Tempest fail ( @arxcruz FYI) Thank you! ::: :::danger #### OSP * OSP17 still without attention (!) because of fires in OSP<16 (details below) * foreign jobs still invading p1/p2 views * not yet tested https://code.engineering.redhat.com/gerrit/#/c/198375 * fhubik's headaches - because all of below these 2 weeks ::: ### Bugzillas Reported :::spoiler | Bugzilla | Name | status | Review | | -------- | ---- |------- | ------ | | [1873770](https://bugs.launchpad.net/tripleo/+bug/1873770) | OVB fs001 in centos8 master fails to push certificates contents to controllers | Incomplete | | | [1873892](https://bugs.launchpad.net/tripleo/+bug/1873892) | Non root login prevented on overcloud machines | Fixed Release | | |[1874019](https://bugs.launchpad.net/tripleo/+bug/1874019)| scenario009-multinode.yaml and openshift.yaml is missing | In Progress | | [1873770](https://bugs.launchpad.net/tripleo/+bug/1873770) | OVB fs001 in centos8 master fails to push certificates contents to controllers | Incomplete | | | [1875352](https://bugs.launchpad.net/tripleo/+bug/1875352)| keystone container failed to start in scenario000 | Triged | | | [1875871](https://bugs.launchpad.net/tripleo/+bug/1875871) | periodic rocky jobs failing with missing --name argument for pcs | Triged | | | [1875846](https://bugs.launchpad.net/tripleo/+bug/1875846) | Overcloud stack creation failed because of failed dependencies. | Closed | | | [1875833](https://bugs.launchpad.net/tripleo/+bug/1875833) | The WebSocket timed out before the Workflow completed in rocky/stain jobs | New | | | [1876087](https://bugs.launchpad.net/tripleo/+bug/1876087) | Queens, tempest.scenario.test_network_basic_ops.TestNetworkBasicOps failing. Timeout | Triged | | | [1876096](https://bugs.launchpad.net/tripleo/+bug/1876096) | Queens: tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern tests failed | Triged | | | [1876672](https://bugs.launchpad.net/tripleo/+bug/1876672) | Python 2 - AttributeError: 'module' object has no attribute 'get_makefile_name' | Fixed Release | | | [1876893](https://bugs.launchpad.net/tripleo/+bug/1876893)| Error: error removing container - device or resource busy | In Progress | | | [1877031](https://bugs.launchpad.net/tripleo/+bug/1877031) | queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version | | ::: ### 8 May ### TripleO #### OSP holiday in CZ (fhubik), off day in TLV (vgriner) ### 7 May ### TripleO #### OSP * Serious breakage of majority of jobs(!) * http://post-office.corp.redhat.com/archives/rhos-qe-dept/2020-May/msg00419.html * /brewroot ceased to exist - poorly advertised in advance * hacking image links to be available on all sites * also https://code.engineering.redhat.com/gerrit/#/c/199624/ * Blocked p2 osp16.1 because of stucked mjob * running for >1 day - https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP16/job/multijob-phase2-osp-16.0-RHEL-8.1/1/ ### 6 May ### TripleO * Tripleo * [1877031](https://bugs.launchpad.net/tripleo/+bug/1877031): queens tripleo-ci-centos-7-undercloud-upgrades broken for ansible version #### OSP * New OSP16+16.1 contents going thru CI * multijobs were deleted this morning by mistake (http://post-office.corp.redhat.com/archives/rhos-infrared-dev/2020-May/msg00033.html) * discovered by RelDel team * still digging out reports from older puddles, escpecially p2 results * UMB issues, manualy triggering, manual promoting * conditions for p2->p3 triggering not clear still * current passed_phase2 links: 16: 20200427.n.0 / 16.1: 20200428.n.0 ### 5 May #### TripleO * tripleo-ci-centos-7-containerized-undercloud-upgrades * https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containerized-undercloud-upgrades * Podman not able to remove the containers memcached. * Bugzilla: ~~[1876901](https://bugs.launchpad.net/tripleo/+bug/1876901)~~ (Closed duplicate) * Bugzillla: [1876893](https://bugs.launchpad.net/tripleo/+bug/1876893) * Need to watch: * periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master * Error: msg: '[''rabbitmq_restart_bundle''] failed to start, check logs in /var/log/containers/stdouts/' * Log [overcloud_update_run_controller.log](https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/c9efb5c/logs/undercloud/home/zuul/overcloud_update_run_Controller.log.txt.gz) [Container-puppet-rabbitmq.log](https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario000-multinode-oooq-container-updates-master/c9efb5c/logs/undercloud/var/log/containers/stdouts/container-puppet-rabbitmq.log.txt.gz) #### OSP * new OSP16.1 content fails p1 right away (05-05) * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase1-16.1_director-rhel-8.2-virthost-1cont_1comp_1ceph-ipv4-geneve-ceph/51/console * https://bugzilla.redhat.com/show_bug.cgi?id=1831551 created * no escalation since nothing is blocked, details in BZ * update: retrigger passed(!) - content changed? * yes - as explained^ in BZ ansible 2.9->2.8 downsgrade on OC was made * phase-customized doesn't support 16.1 * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/phase-customized/1128/console * TODO - needs fix asap also to have tool for reproducing 16.1 stuff also * https://review.gerrithub.io/c/redhat-openstack/infrared/+/491322 finally well tested and merged ### 4 May #### TripleO * [1876672](https://bugs.launchpad.net/tripleo/+bug/1876672): Python 2 - AttributeError: 'module' object has no attribute 'get_makefile_name' * Affected on CentOS7 jobs. * #### OSP * Testing https://review.gerrithub.io/c/redhat-openstack/infrared/+/491322 patchset 2 on multiple OSPs to prevent broken master again * Investigation into problems with OSP16 multijob triggering * OSP16 misbehaving multijob issue hopefully fixed by Waldek's https://code.engineering.redhat.com/gerrit/#/c/199036/2/jobs/release_delivery/scripts/report.groovy - not seen +-since ### 1 May #### TripleO Pacemaker patch for queens, rocky ~~https://review.rdoproject.org/r/#/c/27035/~~ Latest patch for ssh deployment failures: https://review.opendev.org/#/c/723824/ containers-multinode Featureset010 is failing in queens, stein and rock jobs.. There is only one tempest test configured in that job. Since there is only one tempest tempest test, instead of skipping it I have to turn tempest off completely. Arx, Soniya..I need your help to diagnose and bug queens, rocky stein for the test failure if it appears to be unique per release. We have queens: https://bugs.launchpad.net/tripleo/+bug/1876087 We need bugs on: Stein: https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-stein/05b9be9/logs/tempest.html.gz Rocky: https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-rocky/e5f9368/logs/tempest.html.gz These all could be related, but we need to confirm that of course. Thank you for interrupting your schedule to help check these out. ### 1st May #### Tripleo Investigating periodic queens failures * Validate tempest failures: * periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset030-queens(1/1 ssh test failing) * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-queens (22/48 test failing) * periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens (1/1 ssh test failing) * periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens (1/1 ssh test failing) * Validate tempest timeout * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-queens * Deployment failed * periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-queens (failure running haproxy_init_bundle container) reviewing to unblock queens https://review.opendev.org/724703 (still failing, network hiccups) #### OSP holiday in CZ (fhubik), off day in TLV (vgriner) ### 30 April #### Tripleo * Promotion blocker * [1876087](https://bugs.launchpad.net/tripleo/+bug/1876087): Network tests are failing on queens jobs * [1876073](https://bugs.launchpad.net/tripleo/+bug/1876073): Zuul CI is giving false positive on role-addition and molecule consistently * [1876096](https://bugs.launchpad.net/tripleo/+bug/1876096): Queens: tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern tests failed #### OSP * octavia issues - another patch being tested * https://review.gerrithub.io/c/redhat-openstack/infrared/+/491322 * still issue in 16 post step, catched early * octavia meetings about explaining stuff on going * OSP16.1 passed_phase2 link create finally, report sent * release status clarified * OSP16.0 multijobs passed also finally * TODO mjob still broken before * still no work on 17 ### 29th April #### Tripleo * Openstack periodic latest release [pipeline](https://review.rdoproject.org/zuul/builds?pipeline=openstack-periodic-latest-released) * Failed jobs: * periodic-tripleo-ci-centos-7-ovb-1ctlr_1cellctrl_1comp-featureset063-train * periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train * periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train * periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-train-upload * Failure reason: * Before review [723343](https://review.opendev.org/#/c/723343/2/tripleoclient/workflows/deployment.py) all the jobs were printing the error messages and continue deployment. * In this patch `raise` keyword added and it causing job failures. Work on [1873892](https://bugs.launchpad.net/tripleo/+bug/1873892) bug is in progress. * Promotion blocker [1875833](https://bugs.launchpad.net/tripleo/+bug/1875833), [1875846](https://bugs.launchpad.net/tripleo/+bug/1875846) * List of issues found: * https://bugs.launchpad.net/tripleo/+bug/1875833 (Websocket timeout) * Branches: rocky and stein * jobs: * all periodic * https://bugs.launchpad.net/tripleo/+bug/1875846 (dependency problems) * This depends on some other erros we didnt yet find * branches: queens * https://bugs.launchpad.net/tripleo/+bug/1873892 (Non root login prevented on overcloud) * branches: train * jobs: periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train * https://bugs.launchpad.net/tripleo/+bug/1875871 (missing argument for pcs) * branches: rocky * jobs: all periodic * BUG N/A yet * logs: https://logserver.rdoproject.org/openstack-periodic-latest-released/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train/0c8653a/logs/undercloud/home/zuul/overcloud_image_build.log.txt.gz * branch: train * jobs: overclud images build (fs002) * BUG N/A yet * logs: https://logserver.rdoproject.org/openstack-periodic-wednesday-weekend/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-queens/0fc356c/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz * branch: queens * jobs: #### OSP * resolving situation around 16 and 16.1 related to Octavia jobs * Issue with 16.1 octavia, revert https://review.gerrithub.io/c/redhat-openstack/infrared/+/491116 because of breaking 16 * product platform meeting doc updated with details * gathering data from Jenkins not trivial * Still no time for 17 * more details below day before ### 28th April #### Tripleo #### OSP * fires in OSP16.0 & 1 phase2s since "new" octavia jobs are "unstable" * breaking chain and multijob reporting: * 16.0 https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/ReleaseDelivery/view/OSP16/job/multijob-phase2-osp-16.0-RHEL-8.1/89/ mjob broken for uknown reason (%TODO) * manual promotions of passed_phase2, triggering of p3 likely broken * 16.1 One Octavia job possibly blocking p2 promotion, waiting for a root cause update in order to make any decisions: * https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/QE/view/OSP16.1/job/DFG-network-octavia-16.1_director-rhel-virthost-3cont_2comp-ipv4-vxlan-ml2ovs/ * possibly related, TBU: https://trello.com/c/p6QXaZip/1446-cixbz1824397ospphase2neutronosp161networkingha-l3-connectivity-issue?menu=filter&filter=octavia * OSP17 * just no time to deal with this at all fhubik dealing with foreing jobs invading p2 for no reason * https://code.engineering.redhat.com/gerrit/#/c/198375 to filter them * also meetings to explain how things work downstream for new people doing "weird things" having "weird requests" (update p2 jobs), clearly there are communication issues how CI works ### 27th April #### Tripleo * periodic-tripleo-ceph-integration-centos-8-scenario004-standalone * job was failing due to missing release variable. * Fixed: https://review.rdoproject.org/r/c/26835/ * Job name * [tripleo-podman-integration-centos-8-scenario000-standalone](https://review.rdoproject.org/zuul/builds?job_name=tripleo-podman-integration-centos-8-scenario000-standalone) * Bugzilla: https://bugs.launchpad.net/fuel/+bug/1875352 * WATCH tripleo-ci-centos-7-containerized-undercloud-upgrades * failed in gate (once ) * 2020-04-27 16:20:03 | TASK [tripleo-container-manage : Gather podman infos] ************************** 2020-04-27 16:20:03 | Monday 27 April 2020 16:20:03 +0000 (0:00:00.323) 1:04:59.202 ********** 2020-04-27 16:20:06 | fatal: [undercloud]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_168/722648/2/gate/tripleo-ci-centos-7-containerized-undercloud-upgrades/1681532/logs/README.html #### OSP * Phase 3 of OSP16.1 was triggered with puddle, RHOS-16.1-RHEL-8-20200424.n.0 that passed phase2. * most jobs failed on faulty slave titan66.lab.eng.tlv2.redhat.com. * need to re-run the jobs. * many OSP13 jobs failed on a patch for OSP16.1 changing the int to float. * had to revert the change. * still need to resolve the problem on float vs int product version in Infrared. * 15 fine, 16 fine, 16.1 octavia issues - retriggering * asked from RelDel to look at 17 p1 issues * maybe help needed in osp16.1 l3 ha esca debug? * R&R handover info afazekas -> fhubik ### 24th April #### Tripleo * master centos8 failing to build containers because of a network issue while downloading repo info for openvswitch * tempest component still failing on the IP allocation error. Tempest guys are aware of the CIX bug there but are joining the investigation * Promoter raises an error during promotion, but the promotion is not affected. Investigating promotion code * Root cause is a mismatch between api request and response expectation. Promotion actually takes place but we identify the response as an error. * Stein is 2 days behind. looking at logs. * ~~periodic-tripleo-ci-centos-7-standalone-stein fails a single test on tempest~~ * ~~https://logserver.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-standalone-stein/8305c7e/logs/undercloud/home/zuul/tempest.log.txt.gz~~ * ~~2020-04-23 07:19:28 | {1} tempest.api.compute.servers.test_attach_interfaces.AttachInterfacesUnderV243Test.test_add_remove_fixed_ip [22.967979s] ... FAILED~~ * STEIN PROMOTED * Need to watch * **periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master** * If this job failed again on the [image upload](https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-master/71b5386/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz). ### 23rd April #### Tripleo * Waived master featureset20 job to get a promotion. It fails in three known tempest tests ### 22nd April #### Tripleo * Jobs: * periodic-tripleo-ceph-integration-rhel-8-scenario{001,004}-standalone Replaced with * periodic-tripleo-ceph-integration-centos-8-scenario{001,004}-standalone * https://review.rdoproject.org/r/c/26679/ Jobs need to watch: - periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master (Tempest failure. Failed with exception: "Request Timeout") Bugs: * did not reproduce in latest integration jobs https://bugs.launchpad.net/tripleo/+bug/1873770 #### promotion issues :::spoiler 020-04-22 17:14:47,863 25243 ERROR promoter Candidate hash 'aggregate: b3720367a6a0349abcfb06939bed3101, commit: 50837618bdbc4ee18ba25da00a4d98cae9744d68, distro: 99ace58fa85ff53a3de0c282131df46336f81d66, component: ui, timestamp: None': client dlrn_client FAILED promotion attempt to current-tripleo 2020-04-22 17:14:47,863 25243 ERROR promoter API returned different promoted hash Traceback (most recent call last): File "/home/centos/ci-config-refactored/ci-scripts/dlrnapi_promoter/logic.py", line 140, in promote candidate_label=candidate_label) File "/home/centos/ci-config-refactored/ci-scripts/dlrnapi_promoter/dlrn_client.py", line 359, in promote candidate_label=candidate_label) File "/home/centos/ci-config-refactored/ci-scripts/dlrnapi_promoter/dlrn_client.py", line 550, in promote_hash raise PromotionError("API returned different promoted hash") PromotionError: API returned different promoted hash 2020-04-22 17:14:47,866 25243 ERROR promoter Error while trying to promote tripleo-ci-testing to current-tripleo 2020-04-22 17:14:47,866 25243 WARNING promoter Candidate label 'tripleo-ci-testing': NO candidate hash promoted to current-tripleo 2020-04-22 17:14:47,866 25243 INFO promoter Candidate label 'current-tripleo': Attempting promotion to 'current-tripleo-rdo' 2020-04-22 17:14:48,810 25243 INFO promoter Candidate label 'current-tripleo': Fetched 10 hashes 2020-04-22 17:14:49,527 25243 WARNING promoter Target label 'current-tripleo-rdo': No hashes fetched. This could mean that the target label is new or it's the wrong label ::: ### 21st April #### Tripleo Gate jobs failing: * **tripleo-ci-centos-7-undercloud-containers** * we found a bug in fs27: https://bugs.launchpad.net/tripleo/+bug/1873535 @ysandeep worked with Tengu and https://review.opendev.org/#/c/721538/3 is up. But the main issue is a transient race condition, bug reopened: https://bugs.launchpad.net/tripleo/+bug/1820576/ * Branch: stable/train * **tripleo-ci-centos-7-scenario009-multinode-oooq-container** * Bug: https://bugs.launchpad.net/tripleo/+bug/1874019 * Root Cause: In overcloud deploy `openshift.yaml` and `scenario009-multinode.yaml` files are missing. * Branch: stable/train * tripleo-ci-centos-8-containers-multinode * overcloud deploy took 1:15 to complete, the rest TIMED_OUT * it took 20 minutes to download the containers Periodic jobs: * periodic-tripleo-centos-8-master-containers-build-push * Failed to push some containers to rdo registry * https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/83517a9/job-output.txt * Pinged jpena to check if registry has problems, nothing relevant found, so just a network hiccup * Need to watch: * **periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset035-train** ### 20th April #### Tripleo * **periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-master** fails, opened https://bugs.launchpad.net/tripleo/+bug/1873770 * Pinged Tengu as he's in the blame list for that file * https://review.opendev.org/721247 * https://review.opendev.org/721255 * Need to watch jobs: * **periodic-tripleo-ci-centos-8-standalone-full-tempest-scenario-master** * Updated bug https://bugs.launchpad.net/tripleo/+bug/1852770 facing same issue on C8 builds. * **periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train** job failing due to ssh issue/misconfiguration. Not able to find rootcause. Failed bug: https://bugs.launchpad.net/tripleo/+bug/1873892 * (Date: 22/04/2020) Had discussion with Tengu. Found that root cause is `ansible_user: tripleo-admin`. Primary guess this job is not ported properly in Train. ### 17th April #### Tripleo * noticing the latest patches for glance https://review.opendev.org/#/c/712533/ are not consistently resolving previous scenario01/02 issues.. watching * @TheG tripleo-ci-centos-7-containerized-undercloud-upgrades should be voting for everything that is not master ... let's take a look * https://review.opendev.org/721217 * @5rFAC3bRTASHvK6LfOxGWA Amol please watch periodic-openstack-master and openstack-periodic-latest-released in review.rdoproject.org * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-train (Known failure) * periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset021-train (Known failure) * periodic-tripleo-rhel-8-train-containers-build-pushfailure (non-voting)

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.