TripleO CI Community Meeting Notes 2022

tags: Meeting

Team meeting pinglist

arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel, dasm

History: https://hackmd.io/IhMCTNMBSF6xtqiEd9Z0Kw

Gmeet: meet.google.com/igc-nxwj-gws

2022-12-20 Community Call

Attendees: pojadhav, chandan, marios o/, ysandeep

Agenda:

2022-12-13 Community Call

Attendees: Tengu, pojadhav, dasm, ananya, chandan, marios, jm1, ralfieri, ysandeep, bhagyashris, soniya29, akahat, apevec

Agenda:

• Check/gate/opendev testing requirements going forward
  • OSP Next Gen CI Notes: https://hackmd.io/OJoLHEVHT52SQA8NOcqYnQ

2022-12-06 Community Call

Attendees: chandan, dasm, ananya, bhagyashris, jm1, ,rlandy,akahat, pojadhav, dviroel, marios

Agenda:

• Github Zuul Workflow Presentation (Chandan)
  • Prsentation : https://docs.google.com/presentation/d/1lmPPJr-jt5zdI6wLuK0gPi8Rpe0uH5lhY6VeQIBTWws/edit#slide=id.p
  • Patches : https://review.rdoproject.org/r/q/topic:pcp_integration_edpm

2022-11-29 Community Call

Attendees: Tengu, akahat, dasm, bhagyashris, rlandy, chandankumar, arxcruz, jm1, marios,

Agenda:

• [Tengu] State of DNS caching
  • no dnsmasq (multiple reasons)
  • correct the way nodepool image is configured is in!
    • https://review.opendev.org/c/openstack/project-config/+/865433
    • Follow build and propagation:
      • http://nl01.opendev.org/dib-image-list
      • http://nl01.opendev.org/image-list
• [Tengu] ansible-galaxy content caching
  • Use existing per-region proxies provided by Infra
    • Config already exists (via Ib5664e5588f7237a19a2cdb6eec3109452e8a107 )
    • We would just need to use https://${NODEPOOL_MIROR_HOST}/galaxy, and inject it either under galaxy.server in ansible.cfg or as GALAXY_SERVER env var
    • Question: where would that take place (listing some proposal)?
      • tripleo-ci?
      • oooq?
      • oooq-extras?
    • Note that this change may make it easier
    • RDO doesn't provide this endpoint - we should consider adding it imho
    • According to Infra, zuul is apparently caching git repos, we may want to clean this (would also lessen the /opt content from what I understand)
  • ACTIONS:
    • Tengu to propose a patch against tripleo-ansible to switch the project to the proxy
      • https://review.opendev.org/c/openstack/tripleo-ansible/+/866031
    • Tengu to propose a patch against rdo-jobs to expose the ansibe-galaxy proxy var
      • https://review.rdoproject.org/r/c/rdo-jobs/+/46289
  • Alternatives
    • ensure zuul has git content for all the collections we install
    • then tweak ansible-galaxy to use those repository to install collections instead of relying on the galaxy.ansible.com remote

2022-11-22 Community Call

Attendees: Tengu, ysandeep,pojadhav, marios, dasm, rlandy, jm1,

Agenda:

• [Tengu] Caching in CI
  • Why caching?
    • avoid the kind of errors we saw lately with broken DNS and ansible-galaxy downloads (among things)
    • less bandwidth
    • better perfs
  • DNS
    • Why unbound instead of dnsmasq?
      • unbound implementation in NetworkManager is deprecated (from man NetworkManager.conf on fc-35 - and no mention of unbound in latest man)
    • What's the current state (unbound)?
      • Mentions in tripleo-ci
      • Mentions in oooq-extras
    • How to "correctly" set things up (with dnsmasq)
      • install dnsmasq package
      • configure /etc/NetworkManager/NetworkManager.conf
      • [main] dns=dnsmasq rc-manager=file
      • configure dnsmasq options in /etc/NetworkManager/dnsmasq.d/ci-config
        • for better caching (max-cache-ttl, min-cache-ttl, cache-size, no-negcache, dnssec/proxy-dnssec for instance)
        • for better security (interface lo)
        • for custom hosts (we may even do this to replace the /etc/hosts at some point (I'm dreaming, I know…))
        • add some other DNS forwarders in case the "official" ones are dead (downtime mitigation)
      • nmcli g reload
    • Good part is: there's (probably?) no need to do anything in freeIPA scenarios
    • Alternatives
      • unbound (with dns=unbound in nm conf) - support in nm is ending
      • unbound (without networkmanager support, meaning rc-manager unmanaged) - but we may face issues with the actual DNS forwarder, or need hacks to inject the nameservers
      • systemd-resolved (with dns=systemd-resolved in nm conf): not available in cs8, not installed by default in cs9 (but available)
      • don't use any DNS caching, leading to potential issues in case of Infra issues

2022-11-15 Community Call

Attendees: chandan, rlandy, jm1, akahat, Tengu, rcastillo, marios, pojadhav bhagyashris,

Agenda:

• Next Gen meetup info

• [Tengu] Avoid future ansible sh*tstorm due deprecation

  • basically catch deprecation warning in a log, and get ppl to analyse it
  • Action: Tengu to provide somehting in the collect-log

• [jm1]: How to reset our cockpit(s)?! (optional)

2022-11-08 Community Call

Attendees: marios, akahat, jm1, arxcruz, soniya29, ananya,

Agenda:

• Change the community call time as well as scrum time

2022-11-01 Community Call

Attendees: arxcruz, ysandeep, dviroel, akahat, dasm

Agenda:

2022-10-25 Community Call

Attendees: dasm, marios, arxcruz, dviroel,

Agenda:

• [dasm]: INFRA CS9 changes:
  • Problem statement:
    • Current infra deployment code doesn't work for other clouds than infra-tripleo
    • Code cannot be tested before being merged, without abrupting existing infra
  • Proposed resolution:
    • Allow for deployment on different environment for testing purposes, by making deployment code modular
  • Plan of tackling
    • Decoupling base role into separate sub-roles (manual movement to new location)
      • started work: https://review.rdoproject.org/r/q/topic:infra_updates+is:open
    • Decouple "deployment" for a cloud from "service" configuration.
    • Create modular playbooks, indicating order of execution to achieve fully operational environment:
      • Configuration steps could be skipped if not needed
  • Objections:
    • The code is not tested. Can be used "staging" tenant:

      ​​​​​​​​​​​​https://rhos-d.infra.prod.upshift.rdu2.redhat.com/dashboard/project/
      ​​​​​​​​​​​​- ci-scripts/infra-setup/tenant_vars/infra-tripleo/→ change according to incockpit psi stuff
      ​​​​​​​​​​​​- ci-scripts/infra-setup/tenant_vars/infra-tripleo/servers.yml → keep only incockpit
      ​​​​​​​​​​​​- source the psi rc file
      ​​​​​​​​​​​​- ansible-playbook ci-scripts/infra-setup/provision-all.yml
      

2022-10-11 Community Call

Attendees: dasm, marios, pojadhav, arxcruz, ysandeep, soniya29, chandan, bhagyashris, jm1,

Agenda:

• [ysandeep, marios, jm1] dlrn monitoring to sent alerts when its stuck?!

  • [dasm] How often does it happen?
    • [dasm] According to ysandeep it happend 3-4 times in last year.
  • [dasm] What does it mean "stuck"?
    • DLRN stops and needs to be restarted by RDO team.
    • need followup with rdo team - they have some monitoring (sandeep to ping dpawlik amoralej or apevec to find out )
      • we are just concerned with this particular case when building stops completely
        • [dasm] How can we verify|understand|notice when "builds stop completely"?
        • now we go https://trunk.rdoproject.org/centos8-train/report.html and check last dates
          • but still don't know if there is new commits or not
          • would be nice to have a status page

• reminder about https://etherpad.opendev.org/p/antelope_ptg_tripleoci we need to complete this week

2022-10-04 Community Call

Attendees: artom, jparker, ananya, jm1, arxcruz, marios, rlandy, pojadhav, dviroel,

Agenda:

• artom/jparker: RHEx! We finally have Beaker machines available to run baremetal jobs on, is there an existing multinode baremetal job we can inherit from and run on machines?
• PTG slots set
  • https://etherpad.opendev.org/p/antelope_ptg_tripleoci
  • https://etherpad.opendev.org/p/tripleo-antelope-topics
• What do we want to do with scrum? (part 2)

2022-09-27 Community Call

Attendees: Tengu, pojadhav, ananya, chandan, jm1,marios, dasm, arxcruz, soniya29, ysandeep

Agenda:

• Tengu: we're ready to switch to nftables!

  • https://review.opendev.org/c/openstack/tripleo-heat-templates/+/852808
  • https://review.opendev.org/c/openstack/tripleo-ansible/+/853282
  • extended testing were conducted with all of the CI jobs, in order to ensure we don't miss anything
  • huge thank you to Sandeep for all the help on the testproject

• [dasm]: tripleo-jobs-gerrit script

  • https://opendev.org/openstack/tripleo-ci/src/branch/master/scripts/tripleo-jobs-gerrit.py
  • (emit release teardown for victoria) pojadhav noticed time out with test for the script: https://review.opendev.org/c/openstack/tripleo-ci/+/856051
  • dasm noticed the script, running unchanged files, is timing out as well: https://review.opendev.org/c/openstack/tripleo-ci/+/859345
  • The test just runs the script, and does not mock or does not verify if its content is valid. It just fetches return code and checks if it's 0
    • Script is not tested even when the script is changed
    • It is tested only when other *.py files are modified
  • Who uses this script?
    • Zero results: https://review.rdoproject.org/codesearch/?q=tripleo-jobs-gerrit&i=nope&files=&repos=
    • One result - just test: https://codesearch.opendev.org/?q=tripleo-jobs-gerrit&i=nope&literal=nope&files=&excludeFiles=&repos=
  • What's the purpose of the script?
  • Can we delete that?
    • +1 +1 :)

• @arxcruz : Let's get rid of the pingtest

  • testrepository.subunit
  • tempest
  • opensearch

2022-09-20 Community Call

Attendees: ananya, dasm, marios, rlandy, akahat, arxcruz, jm1,

Agenda:

• [jm1] Announcement of Ruck Rover Robot (RRR) aka Promotion Enforcer
• What do we want to do with scrum?
  • too many stories
  • better enforcement of time limits
    • by whoever is running the call
  • tech dicussions need to be taken off line
  • focus on kanban, less on scrum (dariusz, want to write down your thoughts here?)
    • [dasm] Scrum focuses on planning sessions:
      • Entire team agrees on assigned number/points (how long is going to take to finish particular task – not difficulty)
      • Scope of sprint cannot be altered - it means there is no place for incoming tasks/interrupts
    • [dasm] Kanban focuses on one person per task, which means no more hoarding tasks for one engineer. If there is someone else who can work on the task – it can be grabbed and worked on
    • [dasm] Extra column on the Board, which represents "Blocked". It gives visual queues on what's really not feasible to be done.
      • +1 like that
    • [dasm] Engineers are responsible for moving cards, there is no need for separate facilitator of meetings.
  • three line status?
    • what im doing,
    • whats next
    • blockers/risk for delivery
  • breaking down tasks
    • adding definition of done for tasks
    • [dasm] yes, we need to have more granular tasks, which represents what needs to be achieved.
  • Book
    • The art of scrum - Jeff Sutherland - https://www.amazon.com/Scrum-audiobook/dp/B00NHZ6PPE
    • https://www.amazon.com/Scrum-Doing-Twice-Work-Half/dp/038534645X

2022-09-13 Community Call

Attendees: marios, ananya, pojadhav, dasm, chandan, doug, arxcruz, rlandy, jm1,

Agenda:

• scrum meeting guidance from Robert
• https://docs.google.com/presentation/d/1sUg-y6Xre82PD27-HfnlxT1tD5ft9YOFNSA2L_Fqp2E/edit?usp=sharing

2022-09-06 Community Call

Attendees: ananya, pojadhav, rlandy, arxcruz, akahat, marios, jm1, dasm, chandan

Agenda:

• [dasm]: Infrastructure changes for TripleO CI: Toolbox, Cockpit, Promoter
  • Update C7 VM to CS9 VM
    • Ansible code updates
      • Some updates were done manually, and not reflected in the code
  • Move downstream baremetal workload to VM
  • Monitoring for tools
    • To be decided
  • Future plans: migrate tools towards openshift environment
• [marios, jm1]: Jira card for updating links, notes and warnings for our infra, e.g.
  • current hostnames of cockpits, promoters etc.
  • link to our infra-setup repo
  • warning about ansible-pull running in the background aka DONT DO ANY MANUAL CHANGES TO INFRA UNLESS YOU KNOW WHAT YOU ARE DOING

2022-08-30 Community Call

Attendees: dasm, pojadhav, chandan, jm1, akahat, rlandy, bhagyashris, arxcruz,

Agenda:

• (chandankumar) IBM cloud updates
  • nodesets of IBM cloud - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/nodesets.yaml#L606
  • you need to mention nodeset to run a job on IBM cloud - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby.yaml#L17
  • you need to mention cloud name and nodeset for ovb jobs - https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-pipeline-wallaby.yaml#L26-L27
  • secrets for IMB cloud - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/zuul.d/secrets.yaml#L569
  • OVB stack vars - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/defaults/main.yml#L95
  • cert file - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/files/ibm-bm2-nodepool.yaml
  • create cert file and pass cert file - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/roles/ovb-manage/tasks/ovb-clouds-config.yml#L1
  • new cloud may this helful - https://opendev.org/openstack/tripleo-ci/commit/4e348fd9eb74f79bfddf8f1dde32436737d0e020
  • new clouds.yaml file - https://baremetal02.rdoproject.org/dashboard/project/
  • https://review.rdoproject.org/r/c/config/+/9
  • make sure your keys are present here to access node - https://github.com/rdo-infra/review.rdoproject.org-config/blob/master/playbooks/tripleo-rdo-base/pre.yaml#L7

2022-08-23 Community Call

Attendees: arxcruz, pojadhav, jm1, akahat, bhagyashris, rlandy, ananya, dasm, dviroel

Agenda:

• New learnings. [Amol] GO lang
  • akahat and arxcruz are interested in using golang in more non-critical places for our team. Especially around "promoted" area, as well as "skiplist".
  • jm1 and dviroel mentioned that the team works mainly with OpenStack, where all tooling is written in Python, hence we should not introduce new dependencies.
  • dasm mentioned that the skiplist code required some tender loving care, before it could be compiled and used again. The code was not widely used across the team, hence there was not interest in keeping it up-to-date.
  • rlandy is opened for new code, as long as it covers team's interest. golang, due to team's involvement into openshift might be a right thing. R, C or Rust – not really.
  • rlandy suggested we might look into right place to use new golang code. It might be Operator involvement, with new tools.
  • arxcruz and akahat were asked if they're gonna take care of supporting old code, doing reviews and keeping it up-to-date. Both confirmed interest.
  • ysandeep and dviroel mentioned that currently main code in Operator CI code is written in bash (bash scripts) and there is no a lot of area for new tools, especially written in golang.
    • doug: The tool is written in Go, but steps/jobs/config are bash and yaml files (what we are using do far)
  • Meeting ended up without consensus. This topic needs to be discussed again.

2022-08-16 Community Call

Attendees: ananya, marios, arxcruz, akahat, chandan, bhagyashris,

Agenda:

• periodic mixed rhel job current-tripleo vs tripleo-ci-testing
• Network update in downstream:
  • Because of metadata issue.

2022-08-09 Community Call

Attendees: dasm, jm1, Tengu, pojadhav, bhagyashris, arxcruz, chandan, rlandy

Agenda:

• (Tengu) nftables status: we're almost there!

  • Thank you Sandeep for the hard work
  • Leading to some changes in order to get a better logging, better rules and ensure things will be working in the future
  • Logging was already useful in order to spot firewall issues
  • nftables will make TripleO more secure thanks to the way rules are now implemented! (INPUT chain policy, dedicated chains and so on)

• (Tengu) enable auditd service in a (periodic?) job to ensure default proposed ruleset is valid (hint: for now, it's NOT)

  • will need 2 patches in before:
    • https://review.opendev.org/c/openstack/tripleo-ansible/+/848758
    • https://review.opendev.org/c/openstack/tripleo-heat-templates/+/851958
  • Example patch to enable auditd (probably needs some cleaning though)
  • Impact:
    • Probably a bigger audit.log file due to the addition
    • No resource should be impacted

• (jm1) How we do reviews and how we can improve them (continued)

2022-07-26 Community Call

Attendees: dasm, chandan, ananya, arxcruz, bhagyashris, akahat, jm1, rlandy, marios,

Agenda:

• (dasm) OVB reparenting:
  • https://review.rdoproject.org/r/c/rdo-jobs/+/44153
  • reparenting to keep branches separate
  • enable ovb master branch for master release
  • keep ovb stable/2.0 branch for stable releases
• (dasm) skiplist
• (dasm) tooling to simplify our lives - ideas

2022-07-19 Community Call

Attendees: marios, Cédric(Tengu), ysandeep, soniya29, bhagyashris, jm1,

Agenda:

• (Tengu) One more new file for better firewall/nftables debugging
  • https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/850331
• (Tengu) Working on proper nftables logging
  • https://review.opendev.org/q/topic:nftables%252Flogging

2022-07-12 Community Call

Attendees: dasm, jm1, ananya, marios, abregman

Agenda:

• [dasm]: Skiplist \( ﾟヮﾟ)/
  • Last fix: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/849425
  • Dashboard: http://dashboard-ci.tripleo.org/d/3pUqDadGk/tempest-skipped-tests?orgId=1
  • Job definition: https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/tempest-skiplist-jobs.yaml
• [abregman]: Dependency Pipeline
  • Documentation? https://docs.openstack.org/tripleo-docs/latest/ci/tripleo_dependencies_pipelines.html
  • Instance? Tenant?
  • Products/Projects tested?
    • OVS
  • Any information you can give us, would be useful
    • Running in RDO

2022-07-05 Community Call

Attendees: ananya, pojadhav, jm1, dasm, arxcruz, rlandy, marios, dviroel, akahat

Agenda:

• [Tengu] openstack-selinux install validation
  • last patch is about to be in
  • want some help in order to implement an early check as we discussed last time?
    • maybe on the content-provider so that it fails as early as possible?
    • probably on another location to catch 100% of the situation?
    • master and wallaby too https://review.rdoproject.org/r/c/openstack/openstack-selinux-distgit/+/43635
• [dasm] ovb bmc jobs: https://review.rdoproject.org/r/c/config/+/43166
  • test project: https://review.rdoproject.org/r/c/testproject/+/43185
  • comparison master vs stable/2.0 https://github.com/openstack/openstack-virtual-baremetal/compare/master…stable/2.0
  • https://docs.google.com/document/d/14GHgvDWoDpcNQKlQ5pS9My5qDcmhMFyXYUAfXZ2brX0/edit#heading=h.u9f78ig9zqqu
  • follow up with Harald wrt master vs stable/2.0 changes. Do we want to keep it?
• [dasm] tempest skiplist:
  • skiplist jobs - periodic: https://review.rdoproject.org/r/c/rdo-jobs/+/43778
    • testproject: https://review.rdoproject.org/r/c/testproject/+/41367
  • skiplist tool updates: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43815
  • Update jobs definitions with changed names, including -skipped to avoid reporting back to delorean and messing up with test run history.

2022-06-28 Community Call

Attendees: arxcruz, chandan, rlandy, dasm, ananya, marios, abregman , bhagyashris, akahat, fultonj, dviroel

Agenda:

• [fultonj] Ceph Wallaby Patches: https://etherpad.opendev.org/p/tripleo-integration-squad-status
• < dpawlik> arxcruz, rlandy|out, marios: Hello folks. I would like to ask if we can move forward with decommissioning old registry and move to quay.rdoproject.org
  • we need few weeks, after that it can be decommisioned
• [akahat]: Ruck/Rover Tool: https://github.com/amolkahat/ci_dashboard
  • additional info: https://hackmd.io/fXxuOOrkQw6HAYguEUakTg

2022-06-21 Community Call,

Atendees: ibernal, ananya, pojadhav, akahat, dviroel, marios, fultonj, jm1, arxcruz, jgilaber

Agenda:

• [ibernal] Adding a new job on osp-component-pipeline
  1. https://review.opendev.org/c/openstack/tripleo-ci/+/845763
  2. https://docs.openstack.org/tripleo-docs/latest/ci/tripleo_ci_job_parenting.html
  3. https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/component-jobs-master-centos9.yaml#L753
  4. http://git.app.eng.bos.redhat.com/git/openstack/tripleo-ci-internal-jobs.git/tree/zuul.d
• [fultonj] Ceph related changes
  • Changes:
    • https://review.opendev.org/q/topic:deployed_ceph_ci
    • centos mirror update
  • Still cleaning up
    • https://hackmd.io/OkRqfQ0SRUWwllxJthdO3Q?both
    • Why is https://review.rdoproject.org/r/c/testproject/+/36256 not using redis-bundle container with RPMs from https://review.opendev.org/c/openstack/tripleo-common/+/846287 ?
      • add in the testptoject under vars add "build_container_images: true"

2022-06-14 Community Call,

Atendees: dviroel, Chandan, ananya, akahat, rlandy, arxcruz, marios, jgilaber, fultonj, dasm

Agenda:

• [Tengu] Improvements in error management for openstack-selinux package
  1. History: until now, it was silently failing its install, leading to major issues later, with uneasy errors that weren't pointing to the actual cause. Last issue: libselinux-3.4 update
  2. Actions:
     • modify openstack-selinux to support the "–verify" option from RPM
       • https://github.com/redhat-openstack/openstack-selinux/pull/93
       • https://review.rdoproject.org/r/c/openstack/openstack-selinux-distgit/+/43578
     • create a new validation
     • call it as early as possible from within deployment itself
     • Targets: master, wallaby, train
  3. Those actions should help a lot understanding SELinux issues in upstream CI, especially since upstream is usually in Permissive.
• [fultonj] Ceph on multinode
  • keep 001 standalone as is
  • try moving 004 from standalone to multinode
    • Undercloud, Controller, ComputeHCI

2022-06-07 Community Call

Atendees: ibernal, marios, ysandep, abregman, jsilvahe, eliadcohen, fultonj,

Agenda:

• [<fultonj>] ceph: https://review.opendev.org/q/topic:deployed_ceph_ci

  • standalone 001 004 010 will change when TQE from above merges (today?)
  • adding network isolation (storage network) and ceph before openstack
    • openstack overcloud ceph deploy called before openstack overcloud deploy
  • will then backport to wallaby standalone
  • will also update multinode OVB jobs and backport to wallaby
    • https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/844968

• [<eliadcohen>] Horizon integration tests (Selenium) We would like to add the horizon integration tests to one or more of the existing jobs. We ask for guidance - where do we start, what to read etc. Who may assist on this?

  • https://github.com/openstack/horizon/tree/master/openstack_dashboard/test/integration_tests

• [<ibernal>] Tripleo-Quickstart. We are attempting to replicate a jenkins job from Ironic into Quickstart, but we are not sure where we should go about it.

• [<afuscoar>] Ruck & Rover process in CRE team. TripleO CI Grafana: Can we have permissions to modify the dashboard / folder that store our dashboards?

  • https://github.com/rdo-infra/ci-config
    • ci-config is here https://review.rdoproject.org/r/q/project:rdo-infra%252Fci-config+status:open (gerrit review process rdo gerrit)
  • https://code.engineering.redhat.com/gerrit/q/project:openstack/rrcockpit
    • git repo: http://git.app.eng.bos.redhat.com/git/openstack/rrcockpit.git/
  • http://10.0.111.235:8080/ the staging dashboard
  • some docs on getting setup in the role there https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit
  • https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit

2022-05-31 Community Call,

Atendees: Tengu, rlandy, rcastillo, pojadhav, marios, arxcruz, dasm, dviroel

Agenda:

• (Tengu): nftables - welcome to the future!
  • TO Do: add card on sprint board for this task
    • nftables ?! You're already using it - you just don't know it!
      • compatibility wrapper: iptables-nft
    • gerrit topic
    • CI interest:
      • Use nft instead of iptables in collect-logs
      • Example new log
      • «Can I have a (CI) job?»
        • t-h-t toggle switch
        • undercloud.conf toggle switch
      • Issue preventing multi-node job for now: Masquerade breaks on nftables

2022-05-24 Community Call,

Atendees: chandan, marios, pojadhav, arxcruz, dasm, bhagyashris, akahat

Agenda:

• (marios): Mixed RHEL mode: https://review.opendev.org/q/topic:oooci_mixed_rhel
  • Currently Chandan and Brandon are gonna help with initial implementation
  • There is no need for additional resources, because there are not clearly defined tasks.
  • Marios is gonna be on a RR Shift for next week. Current work on Mixed Mode will be paused
• (arxcruz): PTG is starting 2022-06-06
  • Arx is going to attend it.
  • Schedule: https://openinfra.dev/summit-schedule/
  • In case of any interest in attending sessions, please inform Arx to cover them on behalf of the team
• (chandankumar): Tempest Debug - explanation session
  • https://hackmd.io/Yuh4O6-iREipBiR4DR0IOQ?view
  • https://github.com/redhat-openstack/rdoinfo/blob/master/tags/zed-uc.yml#L1038
  • https://github.com/rdo-packages

2022-05-17 Community Call,

Atendees: pojadhav, chandan, rlandy, ysandeep, arxcruz, ananya, marios, bhagyashris, jm1, dviroel,

Agenda:

• Tempest part 1
  • https://hackmd.io/Yuh4O6-iREipBiR4DR0IOQ

2022-05-10 Community Call,

Atendees:

Agenda:

• [jm1] Ansible OpenStack collection: Zuul CI jobs and Grafana Dashboard https://hackmd.io/@jakobmeng/ansible-openstack-collection-ci

2022-05-03 Community Call,

Atendees:

Agenda:

• [jm1] Ansible OpenStack collection: Zuul CI jobs and Grafana Dashboard (postponed to next week because of public holiday in india)

2022-04-26 Community Call

Atendees: chandan, rlandy, ananya, ibernal, abregman, ysandeep, bhagyashris, akahat, dasm, jm1, dviroel,

Agenda:

• https://docs.engineering.redhat.com/display/PRODCHAIN/OSP+Component+Pipeline

2022-04-19 Community Call

Atendees: chandan, ibernal, marios, arxcruz, dasm, pojadhav,

Agenda:

• [abregman] on behalf of Ironic Team, can we have a separate view on component pipeline dashboard (Grafana) for the DFG purpose? so they can customize it the way they want it to look : https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup/roles/rrcockpit
  • main question is what is missing/what they want to see
    • They don't want to see all the data, related to all components. Just the status of one specific component
• CI meetings on Wednesday / Thursday? (or do you want to ask Ronelle etc. first, Bhagyashri?)
  • Retro on Thursday

2022-04-12 Community Call

Atendees: dasm, rlandy, marios, ananya, arxcruz, bhagyashris, pojadhav, ibernal, jm1, abregman, ibernal, jpodivin

Agenda:

• [dasm] on behalf of Ironic Team
  Ironic Team is having challenges with understanding what is tested as part of each job. They're working on filling gaps in coverage.
  • Is there an easier way than traversing jobs definition to gather full information about: configuration, deployed services, tempest tests run?
    • What tests does the job runs?
    • What is the ironic configuration? What does it mean if ironic is marked in a scenario
  • Do we have mapping between job and tests executed: featureset/scenario. If not, would it be possible to add some context into existing scenario table, explaining what each scenario is supposed to test.
  • Can we create a dedicated view for DFG in Grafana
• [dasm]: is there an easy way of "squashing" all job levels into one view?
  • https://review.rdoproject.org/zuul/jobs
  • https://ee27ebc613fa1782cd48-1d34e488101c55f157653eae27a60bf1.ssl.cf1.rackcdn.com/837444/4/check/tripleo-ci-centos-9-containers-multinode/d4a3890/zuul-info/inventory.yaml
• [dasm]: is config/roles/ovb-manage able to be tested with a testproject? change
• [jpodivin] https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/836988
  • https://review.opendev.org/c/openstack/tripleo-ci/+/834631

2022-03-29 Community Call

Atendees: dasm, pojadhav, rlandy, bhagyashris, rcastillo, jm1

Agenda:

• [jm1] Zuul in-depth session: Job variants and branches (will be held in future)

• [jm1] issues like we had yesterday with tripleo ansible currently cannot be discovered because tripleo jobs on ansible podman collection do not redeploy containers. Shall we add a job for github.com/containers/ansible-podman-collections which redeploys overcloud, e.g. runs overcloud deploy twice or runs standalone twice?

• Jakob

  • Suggestion: We create one more job that test redeploy
    • We will brainstorm about doing redeploy in vanilla standalone.

• Sandeep:

  • I have discussed c8 wallaby coverage in checkpoint call today, the suggestion was to keep one job which really test overcloud (and not only undercloud) as minimum set for c8 wallaby coverage. People on call have suggestion that atleast vanilla standalone job should run for both c8 and c9 for wallaby in check/gate.

• naming stable1 line

  • wallaby cs9: stable1-cs9 & stable1: https://review.rdoproject.org/r/c/config/+/41041

    • +1: +1(marios), +1 (bhagyashris), +1(dviroel)
    • -1:

  • wallaby cs8: stable1-cs8 & stable1 :

    • +1: dasm, rlandy, ysandeep, rcastillo
    • -1:

  • 3rd option? stable1-cs8 & stable1-cs9?

    • +1:
    • -1:

• [dasm] retrospective/planning meetings time

2022-03-22 Community Call

Atendees: ananya, jpodivin

Agenda:

• [jm1] Zuul in-depth session: Job variants and branches (rescheduled to 2022-03-29)
• [fpantano] Ceph upstream CI promotion
  • https://docs.google.com/presentation/d/1gIG7RWMJKfPNFnRvo2NV0XnldzxkpVmD3tRum4Ulqac/edit#slide=id.g10607c309e4_0_53
  • Periodic jobs and promotion
    • https://review.opendev.org/c/openstack/tripleo-ci/+/834678
• [Sandeep] OVB based Ceph job(Third party rdo) - something like 1 controller, 1 compute and 1 ceph node
• [jpodivin] Job execution on nonmatching stable branches
  • https://review.opendev.org/c/openstack/validations-libs/+/833228

2022-03-15 Community Call,

Atendees: dasm, marios, pojadhav, arxcruz, jm1, ananya, dviroel,

Agenda:

• [jm1] Ansible in-depth session (10 minutes at the end of the meeting)
  • https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_includes.html

2022-03-08 Community Call

Atendees: jm1, chandan, rlandy, pojadhav, ananya, ysandeep, Tengu, rcastillo, bhagyashris, jistr

Agenda:

• [jm1] DevStack intro (10 minutes at the end of the meeting)
  • https://docs.openstack.org/devstack/latest/
  • https://opendev.org/openstack/devstack.git
  • https://hackmd.io/PI10x-iCTBuO09duvpeWgQ?view
• [jistr] ovb on infra-red setup
  • https://github.com/rdo-infra/review.rdoproject.org-config/tree/master/roles/ovb-manage
  • https://opendev.org/openstack/openstack-virtual-baremetal/src/branch/stable/2.0
  • https://github.com/redhat-openstack/infrared/tree/master/plugins/openstack/roles/bmc
  • https://github.com/sshnaidm/avb

2022-03-01 Community Call

Atendees: pojadhav, ananya, jm1,

Agenda:

• [TripleO] centos9 jobs only for master, centos 8 & 9 for wallaby

  • http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027403.html

  • Upgrades requirements - are we supporting upgrading to Wallaby on 8? For example, the coming undercloud-upgrade-ffu job will be train 8 to wallaby 8. In which case we definitely need to keep at least some subset of 8 jobs (and can't entertain the removal of 8 from Wallaby completely).

  • Are we importing from Wallaby 8 or Wallaby 9? Currently it is 8 but this will soon switch.

  • For the wallaby c8 'subset of jobs' e.g. multinode, vanilla standalone (no scenarios? some subset of them?), undercloud-ffu, minor update.

2022-02-22 Community Call,

Atendees: Chandan, pojadhav, ysandeep, marios, ananya, arxcruz, jm1, rcastillo, akahat

Agenda:

• alfrgarc, fressi - Tobiko Gating

  • https://review.opendev.org/c/x/tobiko/+/827765

• Sandeep - Downstream ruck/rovering refreshed if we have no agenda/no guests.

2022-02-15 Community Call

Atendees: ysandeep, dasm, dviroel, chandan, ananya, marios, arxcruz, jm1, akahat, pojadhav

Agenda:

• (jakob, sshnaidm) Updated guidelines on https://hackmd.io/szgyWa5qSUOWw3JJBXLmOQ about collections work

2022-02-08 Community Call

Atendees: bhagyashris, rlandy, pojadhav,

Agenda:

• Upgrades team:
  • FFU in the component line

    • undercloud ffu https://review.opendev.org/c/openstack/tripleo-heat-templates/+/823966

    • standalone TBD next

    • lets get check/gate & integration lines ready to merge together

      • e.g. undercloud-upgrade periodic oooci-jobs.sh: *** DEFINITION *** https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/standalone-jobs.yaml#L412

    • then go after components (or same time as above)

2022-02-01 Community Call

Atendees: ysandeep, bhagyashris, ananya, dviroel, alfrgarc, jm1, arxcruz, akahat, rcastillo

Agenda:

• the next steps to implement a single node Tobiko gate job upstream - https://docs.google.com/document/d/1xxhlHPCvAqMiZ3vPZRDuqqHX5U_DG1-YKwDhW-klr3g/edit#

  • https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/aae7ca4b6b52873aa5fa0ce7c2f59c642db6a9a8/playbooks/multinode-standalone.yml#L58
  • https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml
  • https://opendev.org/openstack/tripleo-ci/src/branch/master/zuul.d/standalone-jobs.yaml#L963
  • https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/
  • https://paste.opendev.org/show/812463/

• CI Coverage tool

  • https://docs.google.com/document/d/13uFHHxD473D6CtxBhNC1lGWTgkCP7k7UeKmNctgXTyc/edit#heading=h.3868ge8mw9pz

2022-01-25 Community Call

Atendees: Tengu, ananya, rlandy, arxcruz, ysandeep, marios, alfrgarc, dviroel, jm1, rcastillo, akahat, bhagyashris,

Agenda:

• [Tengu] CI status regarding uefi+dib+OC provisioning? (curiosity, wanting to know where I'm sited compare to CI)
  • Tooling: https://github.com/cjeanner/tripleo-tools/tree/main/build-oc-images (build OC things from within a container)
  • https://paste.opendev.org/show/812333/ (patch applied on the fly via my tooling before I switched to Steve latest patch)
• [alfrgarc] Discussion for upstream gating for Tobiko
  • https://docs.google.com/document/d/1xxhlHPCvAqMiZ3vPZRDuqqHX5U_DG1-YKwDhW-klr3g/edit
  • https://opendev.org/x/tobiko/src/branch/master/tobiko/openstack/stacks/_redhat.py
  • https://review.opendev.org/c/x/tobiko/+/683126
  • https://opendev.org/openstack/tripleo-quickstart-extras/src/commit/aae7ca4b6b52873aa5fa0ce7c2f59c642db6a9a8/roles/validate-tobiko
  • https://review.opendev.org/c/x/tobiko/+/651828

2022-01-18 Community Call

Atendees: dviroel, rlandy, ananya,marios, akahat, rcastillo, jm1, dasm, bhagyashris,

Agenda:

• osp director operator intro
• compose pinning work - next steps
  • Add CentOS 8
  • Test backwards
  • Add more jobs

2022-01-11 Community Call

Atendees: bhagyashris, chandan, Tengu, dasm, rlandy, marios, akahat, ananya, dviroel, jm1

Agenda:

• ansible-lint usage and its value
• osp director operator intro
• (Tengu) container building containers, container building OC images
  • One VM able to build things for multiple releases right away?

2022-01-04 Community Call

Atendees: rlandy, marios, arxcruz, dviroel, dasm, ananya

Agenda:

• ruck rover handoff and responsibilities (Thursday)
  • https://docs.openstack.org/tripleo-docs/latest/ci/ruck_rover_primer.html - add some point from below list also add question and answer
  • if ruck/rover switches - make sure public on ruck/rover sheet
    • at least a week
    • rr schedule : https://docs.google.com/spreadsheets/d/1-p0MU38u4NAOqNC_fPIgmRfcrCrSftdbf8CLA9K_JuI/edit#gid=1931857401
  • check/gate status
  • promotions
    • troublesome
  • component line status
    • actively investigating
  • DOWNSTREAM status
    • 16.2 and 17 status
    • check intergation lines - leave a note on list of failures
  • current-tipleo-rdo
    • find creds and restart lines
  • CIX board
    • cards must be triaged
    • list of active bugs and status
  • triage of bugs
    • correct release (milestone)
    • status - triaged
    • prio - critical
    • promotion blocker
  • anything from the program call
  • who is EoD on #rhos-ops
  • I would I would have know by starting of ruck/rover.
    • For example: How to kick off a promotion pipeline
• going forward with c9 on master
  • going voting check
    • Current Parity: https://review.opendev.org/c/openstack/tripleo-common/+/823370/1#message-b66443dffc0e9eb3d4c67cd3c2faa5711e9be447
    • in gates currently: https://review.opendev.org/c/openstack/tripleo-ci/+/822497
    • Last promotion on 21st Dec, waiting for one more promotion
  • last of OVB
    • https://review.rdoproject.org/r/c/testproject/+/35174 https://review.rdoproject.org/r/c/testproject/+/36267
  • left component?
    • scenario 10 downstream missing and ovb missing
      • patch [1] is ready for ovb component jobs, waiting for ovb integration line job [1] https://review.rdoproject.org/r/c/rdo-jobs/+/36987/
  • start remove c8 check
    • discussion started on rhos mailing list

2021-12-21 Community Call

Atendees:

Agenda:

• Plan to scale down c8 testing
  • master moving c9 (DF/prod chain)
    • ok to nuke c8 altogether on master?
    • downside - backports will be tested for the first time on c8 when we hit wallaby
    • possibility: run provider and (standalone?) containers-multinode (intermediate) - only reason we would do this is for backports. If backports are of concern, let's not do this. There is no product value for master on c8.
  • wallaby c8/c9 split (upgrades/prod chain)
    • how much c8 do we need? - consider c7 right now, container build, standalone, scenarios and containers-multinode
    • train on 8 -> wallaby on 9]
      • so do we do os upgrade on T or on W… i.e. do we need support 8->8 upgrades from T->W…?
  • upgrade w->? what is the upgrade path?
    • target the line to support upgrade path
  • backports (minimal c8?)
  • rhos-17?
    • 16.2 -> 17 upgrade
• Thursday - propose patch to run c9 on all tripleo repos
  • patch 1 - moves c9 from branchful to all templates ( as non-voting)
  • patch 2 - makes voting
  • patch 3 - reduces c8
• Templates
  • keep one template for c8 and c9
    • lets try that first
    • once posted, also try posting something that will limit c8 on master (to see what it looks like)
  • create specific c9 templates
    • would need to create the new templates in tripleo-ci
    • then go wire them up in all the repos

2021-12-14 Community Call

Atendees: chandan, rlandy, ananya, marios, dviroel, ysandeep, bhagyashris,

Agenda:

• c9 topics
  • check jobs:
    • https://review.opendev.org/c/openstack/tripleo-ci/+/821241 (template)
      • https://codesearch.openstack.org/?q=tripleo-standalone-scenarios-pipeline&i=nope&literal=nope&files=&excludeFiles=&repos=
    • https://review.opendev.org/c/openstack/tripleo-ci/+/821698
    • add them to trigger just on ci repos (voting?)
    • all tripleo (non-voting until proved stable)
    • Notes: Run standalone jobs on all the tripleo-ci repos
  • backports to wallaby
    • https://review.opendev.org/c/openstack/tripleo-common/+/821511
    • https://review.opendev.org/c/openstack/tripleo-ansible/+/821663
  • Blocking issues:
    • https://bugs.launchpad.net/tripleo/+bug/1954456 (OVB blocker)
    • https://bugs.launchpad.net/tripleo/+bug/1953738 (cedric and brent)

2021-12-07 Community Call

Atendees: dviroel, marios, ananya, rlandy, bhagyashris,

Agenda:

• (dviroel) - Turn off dependencies pipelines (remove triggers)
  • https://hackmd.io/G6b1q7wDSImTMTlKgPP75w?view
• cs9 status
  • https://review.opendev.org/c/openstack/tripleo-ci/+/820172 & https://review.opendev.org/c/openstack/tripleo-ci/+/820850 (test https://review.rdoproject.org/r/c/testproject/+/36817/13#message-92e3f165909ebe0f5faeded43059d32b34796c69 )
  • https://review.rdoproject.org/r/c/rdo-jobs/+/37067
  • we need a promotion https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/37068
• VF:
  • Some reviews need attention:
    • https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36861
    • https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36862
    • https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/36873
• Remove victoria and ussuri from CI repos
  • https://review.opendev.org/c/openstack/tripleo-ci/+/819615

2021-11-30 Community Call

Atendees: ysandeep, matbu, pojadhav, marios, arxcruz, dviroel, bhagyashris,

Agenda:

2021-11-23 Community Call

Atendees: ysandeep, rlandy, marios, fultonj, pojadhav, fmount, gfidente, matbu

Agenda:

• Ceph upstream CI promotion

  • https://docs.google.com/document/d/1bzXrIHJMAzDvoknBqnQx6yHwx051EJVCcHFjBc45AwQ/edit#
  • Builds and containers
  • Coverage
  • Testing / Continuous Testing
  • Promotion
    • This is a typical promotion pipe and it could be automated
      • Provided ceph squad will monitor the automation, CI team will help set up
    • At the end we need the job (if everything passes) add code to do the following:
      • Tag the cephadm RPM to -release
        • We need an example script to do this
      • Update tripleo-common to specify the newer ceph container
    • Need to identify what we want in more detail next
      • we probably want an actual flow diagram, on paper :D
  • Relevant Ceph Logs

• Validations enabled on OVB jobs for Wallaby:

  • https://review.rdoproject.org/r/c/rdo-jobs/+/36850
  • For clients & security a WIP review is up, waiting for some validations-common to land:
    • https://review.rdoproject.org/r/c/rdo-jobs/+/36855 (WIP)

• Collections installation and usage in CI

  • Add additional collections to Zuul projects (merged)
  • Add required-projects to TripleO jobs for collections patch
  • Install collection deps for quickstart patch
  • Caching proxy for Ansible Galaxy in upstream infra!

2021-11-16 Community Call

Atendees: ysandeep, dviroel, rlandy, pojadhav

Agenda:

• Validations execution logs (VF around 6 minutes, total >3hrs)
  • http://pastebin.test.redhat.com/1007546
  • https://logserver.rdoproject.org/08/32408/14/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset001-component-master-validation/2d38d3a/
• Is it possible to make it any faster?

2021-11-09 Community Call

Atendees: chandan, marios, ysandeep, ananya, dviroel, matbu (VF), pojadhav

Agenda:

• Discussion with validation squad
  • What kind of multinode new jobs they want?
    • They are trying to create a new featureset/reuse fs010 + also adding ovb periodic jobs
    • https://review.rdoproject.org/r/c/rdo-jobs/+/36416/21/zuul.d/component-jobs-master.yaml
    • https://review.rdoproject.org/r/c/rdo-jobs/+/36384 <<< OVB 001

2021-11-02 Community Call

Atendees: akahat, chandan, ananya, marios, , rlandy, pojadhav, jfrancoa (Upgrades), holser (Upgrades)

Agenda:

• Updates/upgrades (FFU),
• undercloud-upgrade job in check/gate master (no xena so wallaby->master )
  • to add jobs back into upstream cos feedback loop is too big now.
    • aim: adding standalone ffu job from t->w (wallaby jobs)
    • not -readding the 'old' standalone-upgrade jobs to master/wallaby
  • https://bugs.launchpad.net/tripleo/+bug/1947959
  • Standalone upgrade role: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/roles/standalone-upgrade
  • aim to add 8 to 8 job for train to w cannot do 9 to 9 (cos implies 9 wallaby or being able to do leapp during the job from 8 to 9 ;)
  • Scenario upgrade jobs in component line (for future)