TripleO CI Community Meeting Notes 2022

# TripleO CI Community Meeting Notes 2022 ###### tags: `Meeting` ## Team meeting pinglist arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, dasm **History:** https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw **Gmeet:** meet.google.com/igc-nxwj-gws :::spoiler Gerrit ids - amolkahat@gmail.com - anbanerj@redhat.com - arxcruz@{redhat|gmail}.com - bshewale@redhat.com - chkumar@redhat.com - marios@redhat.com - pojadhav@redhat.com - rlandy@redhat.com - sandyada@redhat.com - svyas@redhat.com - dviroel@redhat.com - dsmigiel@redhat.com ::: --- # 2022-12-20 Community Call ## Attendees: pojadhav, chandan, marios o/, ysandeep ## Agenda: * # 2022-12-13 Community Call ## Attendees: Tengu, pojadhav, dasm, ananya, chandan, marios, jm1, ralfieri, ysandeep, bhagyashris, soniya29, akahat, apevec ## Agenda: * Check/gate/opendev testing requirements going forward * OSP Next Gen CI Notes: https://hackmd.io/OJoLHEVHT52SQA8NOcqYnQ # 2022-12-06 Community Call ## Attendees: chandan, dasm, ananya, bhagyashris, jm1, ,rlandy,akahat, pojadhav, dviroel, marios ## Agenda: * Github Zuul Workflow Presentation (Chandan) * Prsentation : https://docs.google.com/presentation/d/1lmPPJr-jt5zdI6wLuK0gPi8Rpe0uH5lhY6VeQIBTWws/edit#slide=id.p * Patches : https://review.rdoproject.org/r/q/topic:pcp_integration_edpm # 2022-11-29 Community Call ## Attendees: Tengu, akahat, dasm, bhagyashris, rlandy, chandankumar, arxcruz, jm1, marios, ## Agenda: * [Tengu] State of DNS caching * no dnsmasq (multiple reasons) * correct the way nodepool image is configured is in! * https://review.opendev.org/c/openstack/project-config/+/865433 * Follow build and propagation: * http://nl01.opendev.org/dib-image-list * http://nl01.opendev.org/image-list * [Tengu] ansible-galaxy content caching * Use existing per-region proxies provided by Infra * Config already [exists](https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror/templates/mirror.vhost.j2#L127-L133) (via [Ib5664e5588f7237a19a2cdb6eec3109452e8a107 ](https://review.opendev.org/c/opendev/system-config/+/818787)) * We would just need to use `https://${NODEPOOL_MIROR_HOST}/galaxy`, and inject it either under `galaxy.server` in ansible.cfg or as `GALAXY_SERVER` env var * Question: where would that take place (listing some proposal)? * [tripleo-ci](https://opendev.org/openstack/tripleo-ci/src/branch/master/roles/prepare-node/tasks/main.yaml)? * [oooq](https://opendev.org/openstack/tripleo-quickstart/src/branch/master/roles/repo-setup/tasks)? * [oooq-extras](https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/install-built-repo/tasks)? * Note that [this change](https://review.opendev.org/c/opendev/base-jobs/+/865970) may make it easier * **RDO doesn't provide this endpoint** - we should consider adding it imho * According to Infra, zuul is apparently caching git repos, we may want to clean this (would also lessen the /opt content from what I understand) * ACTIONS: * Tengu to propose a patch against tripleo-ansible to switch the project to the proxy * https://review.opendev.org/c/openstack/tripleo-ansible/+/866031 * Tengu to propose a patch against rdo-jobs to expose the ansibe-galaxy proxy var * https://review.rdoproject.org/r/c/rdo-jobs/+/46289 * Alternatives * ensure zuul has git content for *all* the collections we install * then tweak ansible-galaxy to use those repository to install collections instead of relying on the galaxy.ansible.com remote # 2022-11-22 Community Call ## Attendees: Tengu, ysandeep,pojadhav, marios, dasm, rlandy, jm1, ## Agenda: * [Tengu] Caching in CI * Why caching? * avoid the kind of errors we saw lately with broken DNS and ansible-galaxy downloads (among things) * less bandwidth * better perfs * DNS * Why unbound instead of dnsmasq? * unbound implementation in NetworkManager is deprecated (from `man NetworkManager.conf` on fc-35 - and no mention of unbound in [latest man](https://networkmanager.dev/docs/api/latest/NetworkManager.conf.html)) * What's the current state (unbound)? * [Mentions in tripleo-ci](https://https://opendev.org/openstack/tripleo-ci/search?q=unbound) * [Mentions in oooq-extras](https://opendev.org/openstack/tripleo-quickstart-extras/search?q=unbound) * How to "correctly" set things up (with dnsmasq) * install dnsmasq package * configure /etc/NetworkManager/NetworkManager.conf * `[main] dns=dnsmasq rc-manager=file` * configure dnsmasq options in /etc/NetworkManager/dnsmasq.d/ci-config * for better caching (`max-cache-ttl`, `min-cache-ttl`, `cache-size`, `no-negcache`, `dnssec`/`proxy-dnssec` for instance) * for better security (`interface lo`) * for custom hosts (we may even do this to replace the /etc/hosts at some point (I'm dreaming, I know...)) * add some other DNS forwarders in case the "official" ones are dead (downtime mitigation) * `nmcli g reload` * Good part is: there's (probably?) no need to do anything in freeIPA scenarios * Alternatives * unbound (with dns=unbound in nm conf) - support in nm is ending * unbound (without networkmanager support, meaning `rc-manager unmanaged`) - but we may face issues with the actual DNS forwarder, or need hacks to inject the nameservers * systemd-resolved (with dns=systemd-resolved in nm conf): not available in cs8, not installed by default in cs9 (but available) * don't use any DNS caching, leading to potential issues in case of Infra issues # 2022-11-15 Community Call ## Attendees: chandan, rlandy, jm1, akahat, Tengu, rcastillo, marios, pojadhav bhagyashris, ### Agenda: * Next Gen meetup info * [Tengu] Avoid future ansible sh*tstorm due deprecation * basically catch deprecation warning in a log, and get ppl to analyse it * Action: Tengu to provide somehting in the collect-log * [jm1]: How to reset our cockpit(s)?! (optional) --- # 2022-11-08 Community Call ## Attendees: marios, akahat, jm1, arxcruz, soniya29, ananya, ### Agenda: * Change the community call time as well as scrum time --- # 2022-11-01 Community Call ## Attendees: arxcruz, ysandeep, dviroel, akahat, dasm ### Agenda: --- # 2022-10-25 Community Call ## Attendees: dasm, marios, arxcruz, dviroel, ### Agenda: * [dasm]: INFRA CS9 changes: * Problem statement: * Current infra deployment code doesn't work for other clouds than `infra-tripleo` * Code cannot be tested before being merged, without abrupting existing infra * Proposed resolution: * Allow for deployment on different environment for testing purposes, by making deployment code modular * Plan of tackling * Decoupling `base` role into separate sub-roles (manual movement to new location) * started work: https://review.rdoproject.org/r/q/topic:infra_updates+is:open * Decouple "deployment" for a cloud from "service" configuration. * Create modular playbooks, indicating order of execution to achieve fully operational environment: * Configuration steps could be skipped if not needed * Objections: * The code is not tested. Can be used "staging" tenant: ``` https://rhos-d.infra.prod.upshift.rdu2.redhat.com/dashboard/project/ - ci-scripts/infra-setup/tenant_vars/infra-tripleo/→ change according to incockpit psi stuff - ci-scripts/infra-setup/tenant_vars/infra-tripleo/servers.yml → keep only incockpit - source the psi rc file - ansible-playbook ci-scripts/infra-setup/provision-all.yml ``` --- # 2022-10-11 Community Call ## Attendees: dasm, marios, pojadhav, arxcruz, ysandeep, soniya29, chandan, bhagyashris, jm1, ### Agenda: * [ysandeep, marios, jm1] dlrn monitoring to sent alerts when its stuck?! * [dasm] How often does it happen? * [dasm] According to ysandeep it happend 3-4 times in last year. * [dasm] What does it mean "stuck"? * DLRN stops and needs to be restarted by RDO team. * need followup with rdo team - they have some monitoring (sandeep to ping dpawlik amoralej or apevec to find out ) * we are just concerned with this particular case when building stops completely * [dasm] How can we verify|understand|notice when "builds stop completely"? * now we go https://trunk.rdoproject.org/centos8-train/report.html and check last dates * but still don't know if there is new commits or not * would be nice to have a status page * reminder about https://etherpad.opendev.org/p/antelope_ptg_tripleoci we need to complete this week --- # 2022-10-04 Community Call ## Attendees: artom, jparker, ananya, jm1, arxcruz, marios, rlandy, pojadhav, dviroel, ### Agenda: * artom/jparker: RHEx! We finally have Beaker machines available to run baremetal jobs on, is there an existing multinode baremetal job we can inherit from and run on machines? * PTG slots set * https://etherpad.opendev.org/p/antelope_ptg_tripleoci * https://etherpad.opendev.org/p/tripleo-antelope-topics * What do we want to do with scrum? (part 2) # 2022-09-27 Community Call ## Attendees: Tengu, pojadhav, ananya, chandan, jm1,marios, dasm, arxcruz, soniya29, ysandeep ### Agenda: * Tengu: we're ready to switch to nftables! * https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852808 * https://review.opendev.org/c/openstack/tripleo-ansible/+/853282 * extended testing were conducted with all of the CI jobs, in order to ensure we don't miss anything * huge thank you to Sandeep for all the help on the testproject * [dasm]: tripleo-jobs-gerrit script * https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/tripleo-jobs-gerrit.py * (emit release teardown for victoria) pojadhav noticed time out with test for the script: https://review.opendev.org/c/openstack/tripleo-ci/+/856051 * dasm noticed the script, running unchanged files, is timing out as well: https://review.opendev.org/c/openstack/tripleo-ci/+/859345 * The test *just* runs the script, and does not mock or does not verify if its content is valid. It just fetches return code and checks if it's 0 * Script is not tested even when the script is changed * It is tested only when other *.py files are modified * Who uses this script? * Zero results: https://review.rdoproject.org/codesearch/?q=tripleo-jobs-gerrit&i=nope&files=&repos= * One result - just test: https://codesearch.opendev.org/?q=tripleo-jobs-gerrit&i=nope&literal=nope&files=&excludeFiles=&repos= * What's the purpose of the script? * Can we delete that? * +1 +1 :) * @arxcruz : Let's get rid of the pingtest * testrepository.subunit * tempest * [opensearch](https://opensearch.logs.openstack.org/_dashboards/app/dashboards#/view/7c87d570-3d9d-11ed-b73f-c5b7928a4140?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30m,to:now))&_a=(description:'',filters:!(),fullScreenMode:!f,options:(hidePanelTitles:!f,useMargins:!t),query:(language:kuery,query:''),timeRestore:!f,title:'Subunit%20-%20Test%20Results',viewMode:view)) --- # 2022-09-20 Community Call ## Attendees: ananya, dasm, marios, rlandy, akahat, arxcruz, jm1, ### Agenda: * [jm1] Announcement of Ruck Rover Robot (RRR) aka Promotion Enforcer * What do we want to do with scrum? * too many stories * better enforcement of time limits * by whoever is running the call * tech dicussions need to be taken off line * focus on kanban, less on scrum (dariusz, want to write down your thoughts here?) * [dasm] Scrum focuses on planning sessions: * Entire team agrees on assigned number/points (how long is going to take to finish particular task -- not difficulty) * Scope of sprint cannot be altered - it means there is no place for incoming tasks/interrupts * [dasm] Kanban focuses on one person per task, which means no more hoarding tasks for one engineer. If there is someone else who can work on the task -- it can be grabbed and worked on * [dasm] Extra column on the Board, which represents "Blocked". It gives visual queues on what's really not feasible to be done. * +1 like that * [dasm] Engineers are responsible for moving cards, there is no need for separate facilitator of meetings. * three line status? * what im doing, * whats next * blockers/risk for delivery * breaking down tasks * adding definition of done for tasks * [dasm] yes, we need to have more granular tasks, which represents what needs to be achieved. * Book * The art of scrum - Jeff Sutherland - https://www.amazon.com/Scrum-audiobook/dp/B00NHZ6PPE * https://www.amazon.com/Scrum-Doing-Twice-Work-Half/dp/038534645X --- # 2022-09-13 Community Call ## Attendees: marios, ananya, pojadhav, dasm, chandan, doug, arxcruz, rlandy, jm1, ### Agenda: * scrum meeting guidance from Robert * https://docs.google.com/presentation/d/1sUg-y6Xre82PD27-HfnlxT1tD5ft9YOFNSA2L_Fqp2E/edit?usp=sharing --- # 2022-09-06 Community Call ## Attendees: ananya, pojadhav, rlandy, arxcruz, akahat, marios, jm1, dasm, chandan ### Agenda: * [dasm]: Infrastructure changes for TripleO CI: Toolbox, Cockpit, Promoter * Update C7 VM to CS9 VM * Ansible code updates * Some updates were done manually, and not reflected in the code * Move downstream baremetal workload to VM * Monitoring for tools * To be decided * Future plans: migrate tools towards openshift environment * [marios, jm1]: Jira card for updating links, notes and warnings for our infra, e.g. * current hostnames of cockpits, promoters etc. * link to our infra-setup repo * warning about ansible-pull running in the background aka **DONT DO ANY MANUAL CHANGES TO INFRA UNLESS YOU KNOW WHAT YOU ARE DOING** --- # 2022-08-30 Community Call ### Attendees: dasm, pojadhav, chandan, jm1, akahat, rlandy, bhagyashris, arxcruz, ### Agenda: * (chandankumar) IBM cloud updates * nodesets of IBM cloud - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L606 * you need to mention nodeset to run a job on IBM cloud - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby.yaml#L17 * you need to mention cloud name and nodeset for ovb jobs - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby.yaml#L26-L27 * secrets for IMB cloud - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml#L569 * OVB stack vars - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml#L95 * cert file - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/files/ibm-bm2-nodepool.yaml * create cert file and pass cert file - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/ovb-clouds-config.yml#L1 * new cloud may this helful - https://opendev.org/openstack/tripleo-ci/commit/4e348fd9eb74f79bfddf8f1dde32436737d0e020 * new clouds.yaml file - https://baremetal02.rdoproject.org/dashboard/project/ * https://review.rdoproject.org/r/c/config/+/9 * make sure your keys are present here to access node - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-rdo-base/pre.yaml#L7 --- # 2022-08-23 Community Call ### Attendees: arxcruz, pojadhav, jm1, akahat, bhagyashris, rlandy, ananya, dasm, dviroel ### Agenda: * New learnings. [Amol] GO lang * akahat and arxcruz are interested in using golang in more non-critical places for our team. Especially around "promoted" area, as well as "skiplist". * jm1 and dviroel mentioned that the team works mainly with OpenStack, where all tooling is written in Python, hence we should not introduce new dependencies. * dasm mentioned that the skiplist code required some tender loving care, before it could be compiled and used again. The code was not widely used across the team, hence there was not interest in keeping it up-to-date. * rlandy is opened for new code, as long as it covers team's interest. golang, due to team's involvement into openshift might be a right thing. R, C or Rust -- not really. * rlandy suggested we might look into right place to use new golang code. It might be Operator involvement, with new tools. * arxcruz and akahat were asked if they're gonna take care of supporting old code, doing reviews and keeping it up-to-date. Both confirmed interest. * ysandeep and dviroel mentioned that currently main code in Operator CI code is written in bash (bash scripts) and there is no a lot of area for new tools, especially written in golang. * doug: The tool is written in Go, but steps/jobs/config are bash and yaml files (what we are using do far) * Meeting ended up without consensus. This topic needs to be discussed again. --- # 2022-08-16 Community Call ### Attendees: ananya, marios, arxcruz, akahat, chandan, bhagyashris, ### Agenda: * periodic mixed rhel job current-tripleo vs tripleo-ci-testing * Network update in downstream: * Because of metadata issue. --- # 2022-08-09 Community Call ### Attendees: dasm, jm1, Tengu, pojadhav, bhagyashris, arxcruz, chandan, rlandy ### Agenda: * (Tengu) nftables status: we're almost there! * Thank you Sandeep for the hard work * Leading to some changes in order to get a better logging, better rules and ensure things will be working in the future * Logging was already useful in order to spot firewall issues * nftables will make TripleO more secure thanks to the way rules are now implemented! (INPUT chain policy, dedicated chains and so on) * (Tengu) enable auditd service in a (periodic?) job to ensure default proposed ruleset is valid (hint: for now, it's NOT) * will need 2 patches in before: * https://review.opendev.org/c/openstack/tripleo-ansible/+/848758 * https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851958 * [Example patch to enable auditd](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851969) (probably needs some cleaning though) * Impact: * Probably a bigger audit.log file due to the addition * No resource should be impacted * (jm1) How we do reviews and how we can improve them (continued) # 2022-07-26 Community Call ### Attendees: dasm, chandan, ananya, arxcruz, bhagyashris, akahat, jm1, rlandy, marios, ### Agenda: * (dasm) OVB reparenting: * https://review.rdoproject.org/r/c/rdo-jobs/+/44153 * reparenting to keep branches separate * enable ovb master branch for master release * keep ovb stable/2.0 branch for stable releases * (dasm) skiplist * (dasm) tooling to simplify our lives - ideas --- # 2022-07-19 Community Call ### Attendees: marios, Cédric(Tengu), ysandeep, soniya29, bhagyashris, jm1, ### Agenda: * (Tengu) One more new file for better firewall/nftables debugging * https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/850331 * (Tengu) Working on proper nftables logging * https://review.opendev.org/q/topic:nftables%252Flogging --- # 2022-07-12 Community Call ### Attendees: dasm, jm1, ananya, marios, abregman ### Agenda: * [dasm]: Skiplist `\( ﾟヮﾟ)/` * Last fix: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/849425 * Dashboard: http://dashboard-ci.tripleo.org/d/3pUqDadGk/tempest-skipped-tests?orgId=1 * Job definition: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/tempest-skiplist-jobs.yaml * [abregman]: Dependency Pipeline * Documentation? https://docs.openstack.org/tripleo-docs/latest/ci/tripleo_dependencies_pipelines.html * Instance? Tenant? * Products/Projects tested? * OVS * Any information you can give us, would be useful * Running in RDO --- # 2022-07-05 Community Call ### Attendees: ananya, pojadhav, jm1, dasm, arxcruz, rlandy, marios, dviroel, akahat ### Agenda: * [Tengu] openstack-selinux install validation * [last patch](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845569) is about to be in * want some help in order to implement an early check as we discussed last time? * maybe on the content-provider so that it fails as early as possible? * probably on another location to catch 100% of the situation? * master and wallaby too https://review.rdoproject.org/r/c/openstack/openstack-selinux-distgit/+/43635 * [dasm] ovb bmc jobs: https://review.rdoproject.org/r/c/config/+/43166 * test project: https://review.rdoproject.org/r/c/testproject/+/43185 * comparison master vs stable/2.0 https://github.com/openstack/openstack-virtual-baremetal/compare/master...stable/2.0 * https://docs.google.com/document/d/14GHgvDWoDpcNQKlQ5pS9My5qDcmhMFyXYUAfXZ2brX0/edit#heading=h.u9f78ig9zqqu * follow up with Harald wrt master vs stable/2.0 changes. Do we want to keep it? * [dasm] tempest skiplist: * skiplist jobs - periodic: https://review.rdoproject.org/r/c/rdo-jobs/+/43778 * testproject: https://review.rdoproject.org/r/c/testproject/+/41367 * skiplist tool updates: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43815 * Update jobs definitions with changed names, including `-skipped` to avoid reporting back to delorean and messing up with test run history. # 2022-06-28 Community Call ### Attendees: arxcruz, chandan, rlandy, dasm, ananya, marios, abregman , bhagyashris, akahat, fultonj, dviroel ### Agenda: * [fultonj] Ceph Wallaby Patches: https://etherpad.opendev.org/p/tripleo-integration-squad-status * < dpawlik> arxcruz, rlandy|out, marios: Hello folks. I would like to ask if we can move forward with decommissioning old registry and move to quay.rdoproject.org * we need few weeks, after that it can be decommisioned * [akahat]: Ruck/Rover Tool: https://github.com/amolkahat/ci_dashboard * additional info: https://hackmd.io/fXxuOOrkQw6HAYguEUakTg # 2022-06-21 Community Call, ### Atendees: ibernal, ananya, pojadhav, akahat, dviroel, marios, fultonj, jm1, arxcruz, jgilaber ### Agenda: * [ibernal] Adding a new job on osp-component-pipeline 1. https://review.opendev.org/c/openstack/tripleo-ci/+/845763 2. https://docs.openstack.org/tripleo-docs/latest/ci/tripleo_ci_job_parenting.html 3. https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs-master-centos9.yaml#L753 4. http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d * [fultonj] Ceph related changes * Changes: * https://review.opendev.org/q/topic:deployed_ceph_ci * centos mirror update * Still cleaning up * https://hackmd.io/OkRqfQ0SRUWwllxJthdO3Q?both * Why is https://review.rdoproject.org/r/c/testproject/+/36256 not using redis-bundle container with RPMs from https://review.opendev.org/c/openstack/tripleo-common/+/846287 ? * add in the testptoject under vars add "build_container_images: true" --- # 2022-06-14 Community Call, ### Atendees: dviroel, Chandan, ananya, akahat, rlandy, arxcruz, marios, jgilaber, fultonj, dasm ### Agenda: * [Tengu] Improvements in error management for openstack-selinux package 1. History: until now, it was silently failing its install, leading to major issues later, with uneasy errors that weren't pointing to the actual cause. Last issue: libselinux-3.4 update 2. Actions: * modify openstack-selinux to support the "--verify" option from RPM * https://github.com/redhat-openstack/openstack-selinux/pull/93 * https://review.rdoproject.org/r/c/openstack/openstack-selinux-distgit/+/43578 * [create a new validation](https://review.opendev.org/c/openstack/validations-common/+/845559) * [call it as early as possible from within deployment itself](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/845569) * Targets: master, wallaby, train 3. Those actions should help a lot understanding SELinux issues in upstream CI, especially since upstream is usually in Permissive. * [fultonj] Ceph on multinode * keep 001 standalone as is * try moving 004 from standalone to multinode * Undercloud, Controller, ComputeHCI --- # 2022-06-07 Community Call ### Atendees: ibernal, marios, ysandep, abregman, jsilvahe, eliadcohen, fultonj, ### Agenda: * [<fultonj>] ceph: https://review.opendev.org/q/topic:deployed_ceph_ci * standalone 001 004 010 will change when TQE from above merges (today?) * adding network isolation (storage network) and ceph before openstack * `openstack overcloud ceph deploy` called before `openstack overcloud deploy` * will then backport to wallaby standalone * will also update multinode OVB jobs and backport to wallaby * https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/844968 * [<eliadcohen>] Horizon integration tests (Selenium) We would like to add the horizon integration tests to one or more of the existing jobs. We ask for guidance - where do we start, what to read etc. Who may assist on this? * https://github.com/openstack/horizon/tree/master/openstack_dashboard/test/integration_tests * [<ibernal>] Tripleo-Quickstart. We are attempting to replicate a jenkins job from Ironic into Quickstart, but we are not sure where we should go about it. * [<afuscoar>] Ruck & Rover process in CRE team. TripleO CI Grafana: Can we have permissions to modify the dashboard / folder that store our dashboards? * https://github.com/rdo-infra/ci-config * ci-config is here https://review.rdoproject.org/r/q/project:rdo-infra%252Fci-config+status:open (gerrit review process rdo gerrit) * https://code.engineering.redhat.com/gerrit/q/project:openstack/rrcockpit * git repo: http://git.app.eng.bos.redhat.com/git/openstack/rrcockpit.git/ * http://10.0.111.235:8080/ the staging dashboard * some docs on getting setup in the role there https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit * https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit --- # 2022-05-31 Community Call, ### Atendees: Tengu, rlandy, rcastillo, pojadhav, marios, arxcruz, dasm, dviroel ### Agenda: * (Tengu): nftables - welcome to the future! * TO Do: add card on sprint board for this task * nftables ?! You're already using it - you just don't know it! * compatibility wrapper: iptables-nft * [gerrit topic](https://review.opendev.org/q/topic:nftables) * CI interest: * [Use nft instead of iptables in collect-logs](https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/843135) * [Example new log](https://logserver.rdoproject.org/35/843135/3/openstack-check/tripleo-ci-centos-9-ovb-1ctlr_1comp-featureset001/9c0e8e8/logs/undercloud/var/log/extra/nftables.txt.gz) * «Can I have a (CI) job?» * [t-h-t toggle switch](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/841445) * [undercloud.conf toggle switch](https://review.opendev.org/c/openstack/python-tripleoclient/+/842546) * Issue preventing multi-node job for now: [Masquerade breaks on nftables](https://review.opendev.org/c/openstack/tripleo-heat-templates/+/843275) --- # 2022-05-24 Community Call, ### Atendees: chandan, marios, pojadhav, arxcruz, dasm, bhagyashris, akahat ### Agenda: * (marios): Mixed RHEL mode: https://review.opendev.org/q/topic:oooci_mixed_rhel * Currently Chandan and Brandon are gonna help with initial implementation * There is no need for additional resources, because there are not clearly defined tasks. * Marios is gonna be on a RR Shift for next week. Current work on Mixed Mode will be paused * (arxcruz): PTG is starting 2022-06-06 * Arx is going to attend it. * Schedule: https://openinfra.dev/summit-schedule/ * In case of any interest in attending sessions, please inform Arx to cover them on behalf of the team * (chandankumar): Tempest Debug - explanation session * https://hackmd.io/Yuh4O6-iREipBiR4DR0IOQ?view * https://github.com/redhat-openstack/rdoinfo/blob/master/tags/zed-uc.yml#L1038 * https://github.com/rdo-packages --- # 2022-05-17 Community Call, ### Atendees: pojadhav, chandan, rlandy, ysandeep, arxcruz, ananya, marios, bhagyashris, jm1, dviroel, ### Agenda: * Tempest part 1 * https://hackmd.io/Yuh4O6-iREipBiR4DR0IOQ --- # 2022-05-10 Community Call, ### Atendees: ### Agenda: * [jm1] Ansible OpenStack collection: Zuul CI jobs and Grafana Dashboard https://hackmd.io/@jakobmeng/ansible-openstack-collection-ci --- # 2022-05-03 Community Call, ### Atendees: ### Agenda: * ~~[jm1] Ansible OpenStack collection: Zuul CI jobs and Grafana Dashboard~~ (postponed to next week because of public holiday in india) --- # 2022-04-26 Community Call ### Atendees: chandan, rlandy, ananya, ibernal, abregman, ysandeep, bhagyashris, akahat, dasm, jm1, dviroel, ### Agenda: * https://docs.engineering.redhat.com/display/PRODCHAIN/OSP+Component+Pipeline --- # 2022-04-19 Community Call ### Atendees: chandan, ibernal, marios, arxcruz, dasm, pojadhav, ### Agenda: * [abregman] on behalf of Ironic Team, can we have a separate view on component pipeline dashboard (Grafana) for the DFG purpose? so they can customize it the way they want it to look : https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit * main question is what is missing/what they want to see * They don't want to see all the data, related to all components. Just the status of one specific component * CI meetings on Wednesday / Thursday? (or do you want to ask Ronelle etc. first, Bhagyashri?) * Retro on Thursday --- # 2022-04-12 Community Call ### Atendees: dasm, rlandy, marios, ananya, arxcruz, bhagyashris, pojadhav, ibernal, jm1, abregman, ibernal, jpodivin ### Agenda: * [dasm] on behalf of Ironic Team \ Ironic Team is having challenges with understanding what is tested as part of each job. They're working on filling gaps in coverage. * Is there an easier way than traversing jobs definition to gather full information about: configuration, deployed services, tempest tests run? * What tests does the job runs? * What is the ironic configuration? What does it mean if ironic is marked in a scenario * Do we have mapping between job and tests executed: featureset/scenario. If not, would it be possible to add some context into existing scenario table, explaining what each scenario is supposed to test. * Can we create a dedicated view for DFG in Grafana * [dasm]: is there an easy way of "squashing" all job levels into one view? * https://review.rdoproject.org/zuul/jobs * https://ee27ebc613fa1782cd48-1d34e488101c55f157653eae27a60bf1.ssl.cf1.rackcdn.com/837444/4/check/tripleo-ci-centos-9-containers-multinode/d4a3890/zuul-info/inventory.yaml * [dasm]: is `config/roles/ovb-manage` able to be tested with a testproject? [change](https://review.rdoproject.org/r/c/config/+/41127) * [jpodivin] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836988 * https://review.opendev.org/c/openstack/tripleo-ci/+/834631 --- # 2022-03-29 Community Call ### Atendees: dasm, pojadhav, rlandy, bhagyashris, rcastillo, jm1 ### Agenda: * ~~[jm1] Zuul in-depth session: Job variants and branches~~ (will be held in future) * [jm1] [issues like we had yesterday with tripleo ansible]( https://github.com/containers/ansible-podman-collections/pull/402#issuecomment-1081563552) currently cannot be discovered because tripleo jobs on ansible podman collection do not redeploy containers. Shall we add a job for [`github.com/containers/ansible-podman-collections`]( https://github.com/rdo-infra/review.rdoproject.org-config/blob/165262419feac66c19c00783945f910ed31f942b/zuul.d/projects.yaml#L163) which redeploys overcloud, e.g. runs overcloud deploy twice or runs standalone twice? * Jakob * Suggestion: We create one more job that test redeploy * We will brainstorm about doing redeploy in vanilla standalone. * Sandeep: * I have discussed c8 wallaby coverage in checkpoint call today, the suggestion was to keep one job which really test overcloud (and not only undercloud) as minimum set for c8 wallaby coverage. People on call have suggestion that atleast vanilla standalone job should run for both c8 and c9 for wallaby in check/gate. * naming stable1 line * wallaby cs9: stable1-cs9 & stable1: https://review.rdoproject.org/r/c/config/+/41041 * +1: +1(marios), +1 (bhagyashris), +1(dviroel) * -1: * wallaby cs8: stable1-cs8 & stable1 : * +1: dasm, rlandy, ysandeep, rcastillo * -1: * 3rd option? stable1-cs8 & stable1-cs9? * +1: * -1: * [dasm] retrospective/planning meetings time --- # 2022-03-22 Community Call ### Atendees: ananya, jpodivin ### Agenda: * ~~[jm1] Zuul in-depth session: Job variants and branches~~ (rescheduled to 2022-03-29) * [fpantano] Ceph upstream CI promotion * https://docs.google.com/presentation/d/1gIG7RWMJKfPNFnRvo2NV0XnldzxkpVmD3tRum4Ulqac/edit#slide=id.g10607c309e4_0_53 * Periodic jobs and promotion * https://review.opendev.org/c/openstack/tripleo-ci/+/834678 * [Sandeep] OVB based Ceph job(Third party rdo) - something like 1 controller, 1 compute and 1 ceph node * [jpodivin] Job execution on nonmatching stable branches * https://review.opendev.org/c/openstack/validations-libs/+/833228 --- # 2022-03-15 Community Call, ### Atendees: dasm, marios, pojadhav, arxcruz, jm1, ananya, dviroel, ### Agenda: * [jm1] Ansible in-depth session (10 minutes at the end of the meeting) * https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_includes.html --- # 2022-03-08 Community Call ### Atendees: jm1, chandan, rlandy, pojadhav, ananya, ysandeep, Tengu, rcastillo, bhagyashris, jistr ### Agenda: * [jm1] DevStack intro (10 minutes at the end of the meeting) * https://docs.openstack.org/devstack/latest/ * https://opendev.org/openstack/devstack.git * https://hackmd.io/PI10x-iCTBuO09duvpeWgQ?view * [jistr] ovb on infra-red setup * https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/roles/ovb-manage * https://opendev.org/openstack/openstack-virtual-baremetal/src/branch/stable/2.0 * https://github.com/redhat-openstack/infrared/tree/master/plugins/openstack/roles/bmc * https://github.com/sshnaidm/avb --- # 2022-03-01 Community Call ### Atendees: pojadhav, ananya, jm1, ### Agenda: * [TripleO] centos9 jobs only for master, centos 8 & 9 for wallaby * http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027403.html * Upgrades requirements - are we supporting upgrading to Wallaby on 8? For example, the coming undercloud-upgrade-ffu job will be train 8 to wallaby 8. In which case we definitely need to keep at least some subset of 8 jobs (and can't entertain the removal of 8 from Wallaby completely). * Are we importing from Wallaby 8 or Wallaby 9? Currently it is 8 but this will soon switch. * For the wallaby c8 'subset of jobs' e.g. multinode, vanilla standalone (no scenarios? some subset of them?), undercloud-ffu, minor update. --- # 2022-02-22 Community Call, ### Atendees: Chandan, pojadhav, ysandeep, marios, ananya, arxcruz, jm1, rcastillo, akahat ### Agenda: * alfrgarc, fressi - Tobiko Gating * https://review.opendev.org/c/x/tobiko/+/827765 * Sandeep - Downstream ruck/rovering refreshed if we have no agenda/no guests. --- # 2022-02-15 Community Call ### Atendees: ysandeep, dasm, dviroel, chandan, ananya, marios, arxcruz, jm1, akahat, pojadhav ### Agenda: * (jakob, sshnaidm) Updated guidelines on https://hackmd.io/szgyWa5qSUOWw3JJBXLmOQ about collections work --- # 2022-02-08 Community Call ### Atendees: bhagyashris, rlandy, pojadhav, ### Agenda: * Upgrades team: * FFU in the component line * undercloud ffu https://review.opendev.org/c/openstack/tripleo-heat-templates/+/823966 * standalone TBD next * lets get check/gate & integration lines ready to merge together * e.g. undercloud-upgrade periodic oooci-jobs.sh: *** DEFINITION *** https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L412 * then go after components (or same time as above) --- # 2022-02-01 Community Call ### Atendees: ysandeep, bhagyashris, ananya, dviroel, alfrgarc, jm1, arxcruz, akahat, rcastillo ### Agenda: * the next steps to implement a single node Tobiko gate job upstream - https://docs.google.com/document/d/1xxhlHPCvAqMiZ3vPZRDuqqHX5U_DG1-YKwDhW-klr3g/edit# * https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/aae7ca4b6b52873aa5fa0ce7c2f59c642db6a9a8/playbooks/multinode-standalone.yml#L58 * https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml * https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L963 * https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/ * https://paste.opendev.org/show/812463/ * CI Coverage tool * https://docs.google.com/document/d/13uFHHxD473D6CtxBhNC1lGWTgkCP7k7UeKmNctgXTyc/edit#heading=h.3868ge8mw9pz --- # 2022-01-25 Community Call ### Atendees: Tengu, ananya, rlandy, arxcruz, ysandeep, marios, alfrgarc, dviroel, jm1, rcastillo, akahat, bhagyashris, ### Agenda: * [Tengu] CI status regarding uefi+dib+OC provisioning? (curiosity, wanting to know where I'm sited compare to CI) * Tooling: https://github.com/cjeanner/tripleo-tools/tree/main/build-oc-images (build OC things from within a container) * https://paste.opendev.org/show/812333/ (patch applied on the fly via my tooling before I switched to Steve [latest patch](https://review.opendev.org/c/openstack/tripleo-common/+/826205)) * [alfrgarc] Discussion for upstream gating for Tobiko * https://docs.google.com/document/d/1xxhlHPCvAqMiZ3vPZRDuqqHX5U_DG1-YKwDhW-klr3g/edit * https://opendev.org/x/tobiko/src/branch/master/tobiko/openstack/stacks/_redhat.py * https://review.opendev.org/c/x/tobiko/+/683126 * https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/aae7ca4b6b52873aa5fa0ce7c2f59c642db6a9a8/roles/validate-tobiko * https://review.opendev.org/c/x/tobiko/+/651828 --- # 2022-01-18 Community Call ### Atendees: dviroel, rlandy, ananya,marios, akahat, rcastillo, jm1, dasm, bhagyashris, ### Agenda: * osp director operator intro * compose pinning work - next steps * Add CentOS 8 * Test backwards * Add more jobs --- # 2022-01-11 Community Call ### Atendees: bhagyashris, chandan, Tengu, dasm, rlandy, marios, akahat, ananya, dviroel, jm1 ### Agenda: * ansible-lint usage and its value * osp director operator intro * (Tengu) container building containers, container building OC images * One VM able to build things for multiple releases right away? --- # 2022-01-04 Community Call ### Atendees: rlandy, marios, arxcruz, dviroel, dasm, ananya ### Agenda: * ruck rover handoff and responsibilities (Thursday) * https://docs.openstack.org/tripleo-docs/latest/ci/ruck_rover_primer.html - add some point from below list also add question and answer * if ruck/rover switches - make sure public on ruck/rover sheet * at least a week * rr schedule : https://docs.google.com/spreadsheets/d/1-p0MU38u4NAOqNC_fPIgmRfcrCrSftdbf8CLA9K_JuI/edit#gid=1931857401 * check/gate status * promotions * troublesome * component line status * actively investigating * DOWNSTREAM status * 16.2 and 17 status * check intergation lines - leave a note on list of failures * current-tipleo-rdo * find creds and restart lines * CIX board * cards must be triaged * list of active bugs and status * triage of bugs * correct release (milestone) * status - triaged * prio - critical * promotion blocker * anything from the program call * who is EoD on #rhos-ops * I would I would have know by starting of ruck/rover. * For example: How to kick off a promotion pipeline * going forward with c9 on master * going voting check * Current Parity: https://review.opendev.org/c/openstack/tripleo-common/+/823370/1#message-b66443dffc0e9eb3d4c67cd3c2faa5711e9be447 * in gates currently: https://review.opendev.org/c/openstack/tripleo-ci/+/822497 * Last promotion on 21st Dec, waiting for one more promotion * last of OVB * https://review.rdoproject.org/r/c/testproject/+/35174 https://review.rdoproject.org/r/c/testproject/+/36267 * left component? * scenario 10 downstream missing and ovb missing * patch [1] is ready for ovb component jobs, waiting for ovb integration line job [1] https://review.rdoproject.org/r/c/rdo-jobs/+/36987/ * start remove c8 check * discussion started on rhos mailing list --- # 2021-12-21 Community Call ### Atendees: ### Agenda: * Plan to scale down c8 testing * master moving c9 (DF/prod chain) * ok to nuke c8 altogether on master? * downside - backports will be tested for the first time on c8 when we hit wallaby * possibility: run provider and (standalone?) containers-multinode (intermediate) - only reason we would do this is for backports. If backports are of concern, let's not do this. There is no product value for master on c8. * wallaby c8/c9 split (upgrades/prod chain) * how much c8 do we need? - consider c7 right now, container build, standalone, scenarios and containers-multinode * train on 8 -> wallaby on 9] * so do we do os upgrade on T or on W... i.e. do we need support 8->8 upgrades from T->W...? * upgrade w->? what is the upgrade path? * target the line to support upgrade path * backports (minimal c8?) * rhos-17? * 16.2 -> 17 upgrade * Thursday - propose patch to run c9 on all tripleo repos * patch 1 - moves c9 from branchful to all templates ( as non-voting) * patch 2 - makes voting * patch 3 - reduces c8 * Templates * keep one template for c8 and c9 * lets try that first * once posted, also try posting something that will limit c8 on master (to see what it looks like) * create specific c9 templates * would need to create the new templates in tripleo-ci * then go wire them up in all the repos --- # 2021-12-14 Community Call ### Atendees: chandan, rlandy, ananya, marios, dviroel, ysandeep, bhagyashris, ### Agenda: * c9 topics * check jobs: * https://review.opendev.org/c/openstack/tripleo-ci/+/821241 (template) * https://codesearch.openstack.org/?q=tripleo-standalone-scenarios-pipeline&i=nope&literal=nope&files=&excludeFiles=&repos= * https://review.opendev.org/c/openstack/tripleo-ci/+/821698 * add them to trigger just on ci repos (voting?) * all tripleo (non-voting until proved stable) * Notes: Run standalone jobs on all the tripleo-ci repos * backports to wallaby * https://review.opendev.org/c/openstack/tripleo-common/+/821511 * https://review.opendev.org/c/openstack/tripleo-ansible/+/821663 * Blocking issues: * https://bugs.launchpad.net/tripleo/+bug/1954456 (OVB blocker) * https://bugs.launchpad.net/tripleo/+bug/1953738 (cedric and brent) --- # 2021-12-07 Community Call ### Atendees: dviroel, marios, ananya, rlandy, bhagyashris, ### Agenda: * (dviroel) - Turn off dependencies pipelines (remove triggers) * https://hackmd.io/G6b1q7wDSImTMTlKgPP75w?view * cs9 status * https://review.opendev.org/c/openstack/tripleo-ci/+/820172 & https://review.opendev.org/c/openstack/tripleo-ci/+/820850 (test https://review.rdoproject.org/r/c/testproject/+/36817/13#message-92e3f165909ebe0f5faeded43059d32b34796c69 ) * https://review.rdoproject.org/r/c/rdo-jobs/+/37067 * we need a promotion https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37068 * VF: * Some reviews need attention: * https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36861 * https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36862 * https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36873 * Remove victoria and ussuri from CI repos * https://review.opendev.org/c/openstack/tripleo-ci/+/819615 --- # 2021-11-30 Community Call ### Atendees: ysandeep, matbu, pojadhav, marios, arxcruz, dviroel, bhagyashris, ### Agenda: * --- # 2021-11-23 Community Call ### Atendees: ysandeep, rlandy, marios, fultonj, pojadhav, fmount, gfidente, matbu ### Agenda: * Ceph upstream CI promotion * https://docs.google.com/document/d/1bzXrIHJMAzDvoknBqnQx6yHwx051EJVCcHFjBc45AwQ/edit# * Builds and containers * Coverage * Testing / Continuous Testing * Promotion * This is a typical promotion pipe and it could be automated * Provided ceph squad will monitor the automation, CI team will help set up * At the end we need the job (if everything passes) add code to do the following: * Tag the cephadm RPM to -release * We need an example script to do this * Update tripleo-common to specify the newer ceph container * Need to identify what we want in more detail next * we probably want an actual flow diagram, on paper :D * Relevant Ceph Logs * Validations enabled on OVB jobs for Wallaby: * https://review.rdoproject.org/r/c/rdo-jobs/+/36850 * For clients & security a WIP review is up, waiting for some validations-common to land: * https://review.rdoproject.org/r/c/rdo-jobs/+/36855 (WIP) * Collections installation and usage in CI * [Add additional collections to Zuul projects](https://review.opendev.org/c/openstack/project-config/+/818813) (merged) * [Add required-projects to TripleO jobs for collections](https://review.opendev.org/c/openstack/tripleo-ci/+/818820) patch * [Install collection deps for quickstart](https://review.opendev.org/c/openstack/tripleo-quickstart/+/818794) patch * [Caching proxy for Ansible Galaxy in upstream infra!](https://review.opendev.org/c/opendev/system-config/+/818787) # 2021-11-16 Community Call ### Atendees: ysandeep, dviroel, rlandy, pojadhav ### Agenda: * Validations execution logs (VF around 6 minutes, total >3hrs) * http://pastebin.test.redhat.com/1007546 * https://logserver.rdoproject.org/08/32408/14/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-component-master-validation/2d38d3a/ * Is it possible to make it any faster? --- # 2021-11-09 Community Call ### Atendees: chandan, marios, ysandeep, ananya, dviroel, matbu (VF), pojadhav ### Agenda: * Discussion with validation squad * What kind of multinode new jobs they want? * They are trying to create a new featureset/reuse fs010 + also adding ovb periodic jobs * https://review.rdoproject.org/r/c/rdo-jobs/+/36416/21/zuul.d/component-jobs-master.yaml * https://review.rdoproject.org/r/c/rdo-jobs/+/36384 <<< OVB 001 --- # 2021-11-02 Community Call ### Atendees: akahat, chandan, ananya, marios, , rlandy, pojadhav, jfrancoa (Upgrades), holser (Upgrades) ### Agenda: * Updates/upgrades (FFU), * undercloud-upgrade job in check/gate master (no xena so wallaby->master ) * to add jobs back into upstream cos feedback loop is too big now. * aim: adding standalone ffu job from t->w (wallaby jobs) * not -readding the 'old' standalone-upgrade jobs to master/wallaby * https://bugs.launchpad.net/tripleo/+bug/1947959 * Standalone upgrade role: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/standalone-upgrade * aim to add 8 to 8 job for train to w cannot do 9 to 9 (cos implies 9 wallaby or being able to do leapp during the job from 8 to 9 ;) * Scenario upgrade jobs in component line (for future) ---

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.