Try   HackMD

Deployers Team Meeting

Triage Query:
https://github.com/pulp/pulp-oci-images/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed
https://github.com/pulp/pulp-operator/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed

Oct 22

  • Mike to focus on documenting / polishing things over the next week or 2
  • Showed off Konflux CI 1 2 3

Oct 1

  • More CI improvements planned
    Image Not Showing Possible Reasons
    • The image file may be corrupted
    • The server hosting the image is unavailable
    • The image path is incorrect
    • The image format is not supported
    Learn More →

Sep 24

Sep 17

  • Just did triage

Jul 30th

Jul 16th

Jul 9th

Jul 2nd

  • FYI: Konflux CI opens up new possiblitiies
    • Haven't looked into how much RAM, etc is available
  • Demo on Konflux CI

Jun 25th

  • On the services, will use virtualenv to avoid /usr/lib being modified.
    • Might fix multiple python versions

June 18th

  • Python versions configurable: Reached out to Colin Walters, agreed to talk to PyGObject devs
  • pulp-oci-images links: Was advised by Pedro Possoa to use absolute links. Also can update relative links. I Cannot fix it with per-repo config.

June 11th

  • See June 4th item about pulp-operator GA
    • We agreed to release, but there's some final changes/checks 1st
  • Python versions other than el9 defaul (3.9) not meeting pulp-rpm
    • pulp-services will use 3.9
    • Mike will reach out to Colin Walters (ostree) or PyGObject about this
    • libmodulemd definitely not needed by services, will remove also in pulp-oci-images (pulp-rpm 3.19 stopped using it)

June 4th

  • One potential Pulp user asked about pulp operator going GA in the comment section: https://www.youtube.com/watch?v=LpgcdBd9xZI
    • We'll follow up with hyagi when he gets back
  • docs fix needs review
  • CI failing for plugins due to CentOS 8 Stream ("pulp_ci_centos" image) going EOL
    • Submitted PR, and backports, to use CentOS Vault in the base image that it is based on

Apr 23

  • No agenda items, just did triage

Apr 16

  • No agenda items, just did triage

Apr 9

  • No agenda items, just did triage

Apr 2

Mar 26

Mar 19

Mar 12

  • CI Change / Python versions PR
  • galaxy-operator now builds their own smaller images instead of using ours
    • TODO: Gerrod to find out if we can drop building galaxy images (altogether or just branches)

Mar 5, 2024

  • still need to write the blog post
  • In the middle of reviewing the big CI change / Python versions PR

Feb 27, 2024

  • still need to write the blog post

Feb 20, 2024

  • Outline for blog post on EL9 upgrade
    • Why we did it
      • Newer python versions
      • Other benefits?
      • We're developing upstream - latest and greatest
      • Python 3.8 EOL
    • image naming
    • image tags for python versions
    • plugin-template updated
    • remaining work:
      • Fix NSS output issue that breaks oci_env (Daniel working on this FYI)
      • EL8 images again
      • Make multiple EL9 python versions available
  • Old branches for pulpcore releases have not been created for the past ~20 releases

Feb 13, 2024

  • S6 upgrade seems to have fixed the issue with the fix-perms feature.
  • After fixing the postgres version detection properly, and some minor changes to the commits, we merged the EL9 upgrade!
    • Remaining work
      • EL8 builds also (Python 3.8)
      • Multiple EL9 python versions
  • Need to show off how I created the repos on dockerhub & quay

Feb 7, 2024

  • Fixed 2 Ci issues
    • Pulp images are now publishing
  • Helping Daniel with EL9 upgrade and the logic for fixing the permissions on the mounted folder
    • Upgraded S6 (worth seeing if it fixes it)
  • Design for python versions vs os versions
    • We previously agreed on tags
    • We previously agreed to create a new image, pulp_ci_centos9
      • Old el8 pulp_ci_centos image will be kept around for both the old image branches and the old plugin template
    • Python 3.8 is only on el8
    • Python 3.9 is on both el8 and el9
    • Python 3.11 is only on el9 (el9 needed for other reasons too)

Jan 30, 2024

  • pulp_installer bug reports and PRs
    • Agreed: Close them
    • Agreed: Archive the repository
  • User is experiencing a problem with pulp-worker pods getting restarted when the DB connection drops
    • The behavior is different from pulp-api pods
    • Changing the behavior of pulp-worker would not fix the underlying problem that the DB is becoming unavailable

Jan 23, 2024

  • I merged a feature request to pulp-oci-images

Jan 16, 2024

Jan 10, 2024

  • User requests for docker-compose with https
    • Point them to nginx container docs
  • Docs issue with navigation missing for a few pages

Dec 19, 2023

Dec 12, 2023

  • Multi-arch builds
    • Seems to be reliable now!
    • Mike will probably test on a Raspberry Pi 4 soon (forget if 2GB or 4GB)
  • Upgrading image to EL9 / python 3.9
    • We will check that no downstream products use python 3.8

Dec 5, 2023

  • Status of multi-arch builds
    • Merged
  • podman-compose up issue
    • [dkliban] will try to reproduce again
  • About the operator release cycle

Nov 28, 2023

  • [mikedep333] 2 things to review for pulp-operator
  • Status of multi-arch builds
    • [decko] had to revert this last week
    • mikedep333 to research multi-arch registries and review PR inbetween services work

Nov 14, 2023

  • [ttereshc] ansible branch for pulp-operator can be removed
    • a discourse post would be nice to have
    • [hyagi] We can replace it with a tag
      • [mikedep333] This will preserve git history
    • [mikedep333] I recommend renaming the branch to "deprecated-ansible" or "ansible-deprecated"
    • Agreed: Renamed branch or tag
  • [hyagi] ansible team is planning to use the galaxy-ng and galaxy-ui images in galaxy-operator, so, if they manage to make it work with these images, we can stop maintaining galaxy-web and galaxy-minimal
    • Agreed: Have discussions in #pulp-deployments matrix channel or in the #pulp slack channel
  • [dkliban] services team is putting on hold the request for the "watch all namespaces" feature
    • Looking into using pulp resources directly managed by clowder framework rather than clowder calling pulp-operator
    • [Ina] This would preclude the Open Operations Initiative
    • [dkliban] These services can be open, the example of console.fedoraproject.org (upstream for insights)
    • [dkliban] Clowder is specific to Red Hat currently, but could be used by other orgs.
    • [dkliban] As an example of stuff pulp-operator does do that we'd have to reimplement, is watching the config objects and then regenerating pulp settings (and applying them)
    • [Ina] There are people who test pulp-operator. Are they doing things to supplement it like how RH uses clowder?
  • [dkliban] services team is exploring switching the deployments on console.redhat.com to using Clowder https://docs.google.com/document/d/1I-ZI90_xEQpEl5GQdUt-Mcgs6PAwhjy88AHNHAQ_9T4/edit#heading=h.60zkfjl40xzp
  • [mikedep333] Status of fixing the compose data loss bug (PR 468)
    • probable issue with old podman & new podman-compose breaking the CI (the container is up)
    • So I tried upgrading podman, and now it's trying and failing to install old pulp-file for some weird pip dependency reason

October 31, 2023

  • Status of nested containers
    • Fixed an issue whereby the nested container needed 64K + 1 UIDs from the outer container (because UID 0 is already mapped in Podman but not Docker)
    • Hopefully the CI passes, we won't need to run "podman system migrate" in the inner container, and I can move onto the pulp-container PR.
  • weird CI breakage for pulp-ansible / pulpcore version
  • pulp-operator dual Pulp deployment check
  • Update on testing 2 instances of pulp-operator with kind (K8s on container instead of VM)
    • we are maxing the limits of GHA CI
    • We got the rudimentary check done

October 25, 2023

  • Status of nested containers
    • Explored alternatives thoroughly, but approach is require outer containers to support UIDs up to 75535 (65535 + 10000)
      • This avoids need for migration of existing containers
      • This avoids security issues and potential compatibility of supporting UIDs inside the nested container of upto 65534.
      • OpenShift is compatible for now because they do not run containers in user namespaces and therefore have 4G UIDs available
  • [tanya/hyagi] need clarification on impact on community because of "[HMS] Singleton operator - can watch all the namespaces"
    • what is the use case
    • at some point we moved from cluster-scoped to namespace-scoped operator, any concern in watching all namespaces now?
    • will we provide one option or all the combinaitons possible?
      • all
  • [hyagi] CVE for operator framework
  • [hyagi] Issue #1121
    • Does it affect pulp?
    • Should we warn users about "galaxy-operator" migration?
    • What is the correct approach to fix the error?
  • [hyagi] In a galaxy thread on slack the following question arose:
  • "Isn't it required to stop or scale down services before running migrations to prevent locks or conflicts?"*
    • [dkliban] Not required

October 17, 2023

  • Status of nested containers

    • Resumed investigating, the 3rd issue is with mounting /proc in the nested container

  • pulp-operator roadmap

    • We should have one.
    • At least next month or 2 timeframe.
    • Promote shared understanding to the team.

  • Suggested items

    • [dkliban][HMS] Add testing of dual pulp deployments in one namespace
    • [hyagi]bring some samples that we will use in pulpcon presentation up to date
    • [dkliban][HMS] fix how we release the operator bundle (wrong tag)
    • [decko for now] release process automation
    • [hyagi] CVE http/2
    • [collaborative effort?][HMS] Singleton operator - can watch all the namespaces
    • [good for newcomers] review and update the operator spec fields description/appearance in ocp
    • refactor https://docs.google.com/document/d/1a9j7pVi7fv_Fym6TBSxDNEvPyZVqexUNns3jdcjhIrs/edit#heading=h.8vjxjmi9nsre (see 'code improvements')
    • metadata signing needs to be fixed (dkliban to check if HMS is affected)

October 10, 2023

  • Resuming work on nested latest podman (for pulp_container CI)
    • 2 issues fixed, at least 1 remaining
  • Going to finally merge the data loss bug
    • 1st need to rebase
    • was delayed due to CI being red
  • pulp-operator testing
    • Modify an existing test to deploy 2 instances of the operator side-by-side, for image-builder + content-sources
    • We are considering making our operator global, watching all namespaces
    • (Another project) image is registered as a catalog source, as a bundle image. So they push these in CI, and their cluster-wide OLM operator knows to deploy a new version.
    • Releasing to the catalog from the main branch?

October 3, 2023

September 12, 2023

  • Need another pulp-operator release with the new entrypoint for pulpcore-api.

    • How can we avoid carrying a separate entry point in the operator?

  • Looks like we still have a single migration job running when we have two deployments of Pulp in one namespace.

Sept 5, 2023

  • backup of /var/lib/pulp/tmp content
    • can we ignore this folder during backup? does it have anything that is important?

Aug 29, 2023

  • pulp_container CI is failing with latest pulp-ci-centos image
    • There was a regression in the capabilities of newuidmap / newgidmap - easy to fix
    • I noticed that we never set VOLUMES for /var/lib/containers & /var/lib/pulp/.local/share/containers
      • Ask the pulp_container team about this.
    • Lots of other prescribed changes too that we never did
    • Agreed: Do not focus on all the other prescribed changes
    • Agreed: File an issue

Aug 22, 2023

  • Compose data loss bug
    • Fixing the migration of config files from multi-process container to compose
  • Helping jhutar with performance testing cluster
  • pulpcore-selinux 2.0.0 released for katello / satellite

Aug 15, 2023

  • oci_env issues that jtanner had
    • s6-rc: fatal: unable to take locks: Resource busy
      • Hard to research online
      • [mikedep333] Will look into if I run int it myself.
    • the next issue is that because of the chained restarts and config updates, we don't have the social auth postgres tables ... so something needs to run migrations again

Aug 8, 2023

  • Idea (not proposal) for pulpcon: demo HA pulp in k8s
    • Services team will be doing HA soon (part of production plan)
    • Let's do a joint presentation: mikedep333 & hyagi
    • [mikedep333] Please re-explain the pulp-operator deployment modified function check
  • Any changes with decko on leave?
    • dkliban and mikedep333 to maintain pulp-oci-images
  • [mikedep333] [services] pulp-smash config issue

Aug 1, 2023

  • Enabling multiple pulp operators to run in a namespace
    • Needed because content-sources and image-builder will each have a pulp operator running in a single namespace
    • Except for a few singletons, almost all k8s objects will have names like "content-sources-pulp-api-abcdef" and "image-builder-pulp-api-abcdef"

Jul 25, 2023

  • mikedep333 to continue following up with user support requests / bug reports for pulp_installer

Jul 19, 2023

  • CI dashboard will be helpful
  • for hosted-content: CI smoke tests with CJI IQE (requires writing plugin) or a custom bash test script that calls pytest?
    • Agreed: Write custom bash test script for now

Jul 12, 2023

  • We fixed pulp-oci-images CI last week
  • Did the release for fixed images for prior pulp versions
  • oci_env CI has the Ubuntu podman-compose fix now (from pulp-oci-images)

Jun 27, 2023

Jun 20, 2023

  • Haven't done the quick fix / release for AAP-11911 yet
  • Hosted Content: Deploying to staging is making slow but steady progress
  • Re-review please: https://github.com/pulp/pulp-oci-images/pull/508
  • We need to publish a pulp-web image to be used on consoledot.
    • Agreed: Create another dockerfile in pulp-clowder-deployments repo. Build in same script.

Jun 13, 2023

  • Was asked to fix AAP-11911
  • Hosted Content: Deploying to staging is making steady progress
  • 3.26 fixed release:
    • Not released yet, we just merged the PR to update the list of branches to build
    • Manually triggered a 3.26 CI run (pulp-oci-images CI) to build and push the 3.26 image

Jun 6, 2023

  • Hosted services: Mike working on deployment method
    • Plan to merge PR to pulp-clowder-deployments, and submit PR to app-interface today
  • Did CI fixes so that 3.25 image is built and pushed

May 30, 2023

  • Hosted services: Mike working on deployment method
    • Working on the dummy script, trying things out
  • A couple of open PRs / issues
    • Mike providing reviews, answering questions
  • Mike helped Decko with UBI image limitations
    • Some packages exist in repos like "RHEL base" but not "UBI base"
    • Adding CentOS Stream Repos for now
    • If there's breakage, we can try the rockylinux / almalinux repos. Not using them now because their 8.9 release will lag behind RHEL 8.9 release by a few weeks.

May 23, 2023

  • AAP-11911 is actually on hold (bmbouter relayed this on 5/17)
  • Work with the consulting team (Matt Dorn)?
    • No issues have come up
    • I think we wanted their review on this: https://github.com/pulp/pulp-operator/pull/898
      • Agreed: This is such a big review, we will focus as a team (with or without their help) on reviewing it when hyagi gets back, and establishes who needs this, and when.

May 17, 2023

Prior meetings:

tags: Minutes
Last changed by 
 
Pulp
Design/Discussion/Meeting minutes for https://pulpproject.org/
0
1267

Read more

RPM meeting

discuss adopting zero-downtime-migration strategy

Jun 16, 2025
Simpler Enterprise RBAC

The feedback on the SpiceDB approach received from an enterprise user of Pulp was:

Jun 12, 2025
Pulpcore team meeting

Core SME List

Jun 10, 2025
CLI Team

https://github.com/pulp/pulp-cli/pulls

Jun 4, 2025
Read more from Pulp
Published on HackMD