---
# System prepended metadata

title: Deployers Team Meeting
tags: [Minutes]

---

# Deployers Team Meeting

Triage Query:
https://github.com/pulp/pulp-oci-images/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed
https://github.com/pulp/pulp-operator/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed

# Oct 22
* Mike to focus on documenting / polishing things over the next week or 2 
* Showed off Konflux CI [1](https://github.com/pulp/pulp-service/blob/main/Dockerfile) [2](https://github.com/pulp/pulp-service/blob/main/.tekton/pulp-pull-request.yaml) [3](https://console.redhat.com/application-pipeline/workspaces/pulp-services/applications/pulp/activity/pipelineruns)

# Oct 1
* More CI improvements planned :smiley: 

# Sep 24
 * Working on more CI improvements
     * https://github.com/pulp/pulp-oci-images/pull/666
     * Want to speed it up, make it smarter about building images
 * Removing galaxy images
     * Only ansible core uses it in their CI, they use a specific tag: 4.7.1
 * Adding the UI to the Pulp (s6) images
     * https://github.com/pulp/pulp-oci-images/issues/664
     * Going to add support to the minimal images at a later date, then open an issue for operator support
 * Discussed user operator PR to make some secret fields no longer immutable
     * https://github.com/pulp/pulp-operator/pull/1355

# Sep 17
* Just did triage

# Jul 30th

# Jul 16th
* [New terraform logic](https://github.com/pulp/pulp-k8s-resources/pull/29), could use a review

# Jul 9th
* 

# Jul 2nd
* FYI: Konflux CI opens up new possiblitiies
    * Haven't looked into how much RAM, etc is available
* Demo on Konflux CI

# Jun 25th
* On the services, will use virtualenv to avoid /usr/lib being modified.
    * Might fix multiple python versions

# June 18th
* Python versions configurable: Reached out to Colin Walters, agreed to talk to PyGObject devs
* pulp-oci-images links: Was advised by Pedro Possoa to use absolute links. Also can update relative links. I Cannot fix it with per-repo config.


# June 11th
* See June 4th item about pulp-operator GA
    * We agreed to release, but there's some final changes/checks 1st
* Python versions other than el9 defaul (3.9) not meeting pulp-rpm 
    * pulp-services will use 3.9
    * Mike will reach out to Colin Walters (ostree) or PyGObject about this
    * libmodulemd definitely not needed by services, will remove also in pulp-oci-images (pulp-rpm 3.19 stopped using it)

# June 4th
* One potential Pulp user asked about pulp operator going GA in the comment section: https://www.youtube.com/watch?v=LpgcdBd9xZI
    * We'll follow up with hyagi when he gets back
* [docs fix needs review](https://github.com/pulp/pulp-oci-images/pull/633)
* CI failing for plugins due to CentOS 8 Stream ("pulp_ci_centos" image) going EOL
    * Submitted PR, and backports, to use CentOS Vault in the base image that it is based on
 
# Apr 23
* No agenda items, just did triage

# Apr 16
* No agenda items, just did triage

# Apr 9
* No agenda items, just did triage

# Apr 2
* Fix for configurable py version
    * https://github.com/pulp/pulp-oci-images/pull/622
* Need to create 3.49 branch

# Mar 26
* [Discourse post](https://discourse.pulpproject.org/t/containers-upgraded-to-centos-9/1165) on EL9


# Mar 19
* [CI Change / Python versions PR](https://github.com/pulp/pulp-oci-images/pull/606)

# Mar 12
* CI Change / Python versions PR
* galaxy-operator [now builds their own smaller images](https://github.com/ansible/galaxy-operator/issues/77#issuecomment-1979705823) instead of using ours
    * TODO: Gerrod to find out if we can drop building galaxy images (altogether or just branches)

# Mar 5, 2024
* still need to write the blog post
* In the middle of reviewing the big CI change / Python versions PR

# Feb 27, 2024
* still need to write the blog post

# Feb 20, 2024
* Outline for blog post on EL9 upgrade
    * Why we did it
        * Newer python versions
        * Other benefits?
        * We're developing upstream - latest and greatest
        * Python 3.8 EOL
    * image naming
    * image tags for python versions
    * plugin-template updated
    * remaining work:
        * Fix NSS output issue that breaks oci_env (Daniel working on this FYI)
        * EL8 images again
        * Make multiple EL9 python versions available
* Old branches for pulpcore releases have not been created for the past ~20 releases
    * https://github.com/pulp/pulp-oci-images/pull/597

# Feb 13, 2024
* S6 upgrade seems to have fixed the issue with the fix-perms feature.
* After fixing the postgres version detection properly, and some minor changes to the commits, we merged the EL9 upgrade!
    * Remaining work
        * EL8 builds also (Python 3.8)
        * Multiple EL9 python versions
* Need to show off how I created the repos on dockerhub & quay

# Feb 7, 2024
* Fixed 2 Ci issues
    * Pulp images are now publishing
* Helping Daniel with EL9 upgrade and the logic for fixing the permissions on the mounted folder
    * Upgraded S6 (worth seeing if it fixes it)
* Design for python versions vs os versions
    * We previously agreed on tags
    * We previously agreed to create a new image, pulp_ci_centos9
        * Old el8 pulp_ci_centos image will be kept around for both the old image branches and the old plugin template
    * Python 3.8 is only on el8
    * Python 3.9 is on both el8 and el9
    * Python 3.11 is only on el9 (el9 needed for other reasons too)

# Jan 30, 2024
* pulp_installer bug reports and PRs
    * Agreed: Close them
    * Agreed: *Archive* the repository
* User is experiencing a problem with pulp-worker pods getting restarted when the DB connection drops
    * The behavior is different from pulp-api pods
    * Changing the behavior of pulp-worker would not fix the underlying problem that the DB is becoming unavailable

# Jan 23, 2024
* I merged a feature request to pulp-oci-images
    * Now there's a [revert PR](https://github.com/pulp/pulp-oci-images/pull/581)

# Jan 16, 2024
* EL9 with Postgres version mismatch errors
    * https://github.com/pulp/pulp-oci-images/pull/575#issuecomment-1891036347
        * Mike to follow up with Daniel
* [Fixed links in docs](https://github.com/pulp/pulp-oci-images/pull/578), but wondering how to add indented/nested docs links on [navigation bar](https://docs.pulpproject.org/pulp_oci_images/) - [source](https://github.com/pulp/pulp-oci-images/blob/latest/mkdocs.yml#L29)

# Jan 10, 2024
* User requests for docker-compose with https
    * Point them to nginx container docs
* Docs issue with navigation missing for a few pages

# Dec 19, 2023
* 1 PR to review:
    * https://github.com/pulp/pulp-oci-images/pull/569
* We will not need to take turns to release the operator anymore because now we have a script to make the process easier

# Dec 12, 2023
* Multi-arch builds
    * Seems to be reliable now!
    * Mike will probably test on a Raspberry Pi 4 soon (forget if 2GB or 4GB)
* [Upgrading image to EL9 / python 3.9](https://github.com/pulp/pulp-oci-images/issues/537)
    * We will check that no downstream products use python 3.8

# Dec 5, 2023

* Status of multi-arch builds
    * Merged
* podman-compose up issue
    * [dkliban] will try to reproduce again 
* About the operator release cycle


# Nov 28, 2023
* [mikedep333] 2 things to review for pulp-operator
    * [pulp-operator HA](https://docs.google.com/document/d/1PDYFRUNiLf-AoF9xu9st5-mTchE39X-0wuCf7sBGUR8/edit#heading=h.zgoda9dbx81w)
    * [robot framework test code](https://github.com/git-hyagi/robot-framework-ocp-tests/tree/master/pulp)
    * mikedep333 provided initial feedback
    * We're confident in them, let's keep using them
* Status of multi-arch builds
    * [decko] had to revert this last week
    * mikedep333 to research multi-arch registries and review PR inbetween services work


# Nov 14, 2023

* [ttereshc] ansible branch for pulp-operator can be removed
    * a discourse post would be nice to have
    * [hyagi] We can replace it with a tag
        * [mikedep333] This will preserve git history
    * [mikedep333] I recommend renaming the branch to "deprecated-ansible" or "ansible-deprecated"
    * Agreed: Renamed branch or tag
* [hyagi] ansible team is planning to use the galaxy-ng and galaxy-ui images in galaxy-operator, so, if they manage to make it work with these images, we can stop maintaining galaxy-web and galaxy-minimal
    * Agreed: Have discussions in #pulp-deployments matrix channel or in the #pulp slack channel
* [dkliban] services team is putting on hold the request for the "watch all namespaces" feature
    * Looking into using pulp resources directly managed by clowder framework rather than clowder calling pulp-operator
    * [Ina] This would preclude the Open Operations Initiative
    * [dkliban] These services can be open, the example of console.fedoraproject.org (upstream for insights)
    * [dkliban] Clowder is specific to Red Hat currently, but could be used by other orgs.
    * [dkliban] As an example of stuff pulp-operator does do that we'd have to reimplement, is watching the config objects and then regenerating pulp settings (and applying them)
    * [Ina] There are people who test pulp-operator. Are they doing things to supplement it like how RH uses clowder?
* [dkliban] services team is exploring switching the deployments on console.redhat.com to using Clowder https://docs.google.com/document/d/1I-ZI90_xEQpEl5GQdUt-Mcgs6PAwhjy88AHNHAQ_9T4/edit#heading=h.60zkfjl40xzp
* [mikedep333] Status of fixing the compose data loss bug (PR 468)
    * [probable issue](https://github.com/mikedep333/pulp-oci-images/actions/runs/6791023058/job/18493443085#step:14:202) with old podman & new podman-compose breaking the CI (the container is up)
    * So I tried upgrading podman, and [now it's trying and failing](https://github.com/mikedep333/pulp-oci-images/actions/runs/6819758133/job/18547754631#step:10:205) to install old pulp-file for some weird pip dependency reason

# October 31, 2023
* Status of nested containers
    * Fixed an issue whereby the nested container needed 64K + 1 UIDs from the outer container (because UID 0 is already mapped in Podman but not Docker)
    * Hopefully [the CI](https://github.com/mikedep333/pulp-oci-images/actions/runs/6708668363) passes, we won't need to run "podman system migrate" in the inner container, and I can move onto the pulp-container PR. 
* weird [CI breakage](https://github.com/pulp/pulp-oci-images/actions/runs/6700817758/job/18207480438#step:10:265) for pulp-ansible / pulpcore version
* pulp-operator dual Pulp deployment check 
* Update on testing 2 instances of pulp-operator with kind (K8s on container instead of VM)
    * we are maxing the limits of GHA CI
    * We got the rudimentary check done

# October 25, 2023
* Status of nested containers
    * Explored alternatives thoroughly, but approach is require outer containers to support UIDs up to 75535 (65535 + 10000)
        * This avoids need for migration of existing containers
        * This avoids security issues and potential compatibility of supporting UIDs inside the nested container of upto 65534.
        * OpenShift is compatible [for now](https://issues.redhat.com/browse/RFE-3254) because they do not run containers in user namespaces and therefore have 4G UIDs available
* [tanya/hyagi] need clarification on impact on community because of "[HMS] Singleton operator - can watch all the namespaces"
    * what is the use case
    * at some point we moved from cluster-scoped to namespace-scoped operator, any concern in watching all namespaces now?
    * will we provide one option or all the combinaitons possible?
        * all
* [hyagi] CVE for operator framework
* [hyagi] Issue [#1121](https://github.com/pulp/pulp-operator/issues/1121)
    * Does it affect pulp?
    * Should we warn users about "galaxy-operator" migration?
    * What is the correct approach to fix the error?
* [hyagi] In a galaxy thread on slack the following question arose:
* "Isn't it required to stop or scale down services before running migrations to prevent locks or conflicts?"*
    * [dkliban] Not required

# October 17, 2023
* Status of nested containers
    * Resumed investigating, the 3rd issue is with mounting /proc in the nested container
* pulp-operator roadmap
    * We should have one.
    * At least next month or 2 timeframe.
    * Promote shared understanding to the team.

* Suggested items
    * [dkliban][HMS] Add testing of dual pulp deployments in one namespace
    * [hyagi]bring some samples that we will use in pulpcon presentation up to date
    * [dkliban][HMS] fix how we release the operator bundle (wrong tag)
    * [decko for now] release process automation
    * [hyagi] CVE http/2
    * [collaborative effort?][HMS] Singleton operator - can watch all the namespaces
    * [good for newcomers] review and update the operator spec fields description/appearance in ocp
    * refactor https://docs.google.com/document/d/1a9j7pVi7fv_Fym6TBSxDNEvPyZVqexUNns3jdcjhIrs/edit#heading=h.8vjxjmi9nsre (see 'code improvements')
    * metadata signing needs to be fixed  (dkliban to check if HMS is affected)


# October 10, 2023

* Resuming work on nested latest podman (for pulp_container CI)
    * 2 issues fixed, at least 1 remaining
* Going to finally merge the data loss bug
    * 1st need to rebase
    * was delayed due to CI being red
* pulp-operator testing
    * Modify an existing test to deploy 2 instances of the operator side-by-side, for image-builder + content-sources
    * We are considering making our operator global, watching all namespaces
    * (Another project) image is registered as a catalog source, as a bundle image. So they push these in CI, and their cluster-wide OLM operator knows to deploy a new version.
    * Releasing to the catalog from the main branch?

# October 3, 2023

* 


# September 12, 2023

* Need another pulp-operator release with the new entrypoint for pulpcore-api.
    * How can we avoid carrying a separate entry point in the operator?

* Looks like we still have a single migration job running when we have two deployments of Pulp in one namespace. 

# Sept 5, 2023

* backup of /var/lib/pulp/tmp content
    * can we ignore this folder during backup? does it have anything that is important?


# Aug 29, 2023
* pulp_container CI is failing with latest pulp-ci-centos image
    * There was a regression in the capabilities of newuidmap / newgidmap - easy to fix
    * I noticed that we never set VOLUMES for /var/lib/containers & /var/lib/pulp/.local/share/containers
        * Ask the pulp_container team about this.
    * [Lots of other prescribed changes](https://www.redhat.com/sysadmin/podman-inside-container) too that we never did
    * Agreed: Do not focus on all the other prescribed changes
    * Agreed: File an issue

# Aug 22, 2023
* Compose data loss bug
    * Fixing the migration of config files from multi-process container to compose
* Helping jhutar with performance testing cluster
* pulpcore-selinux 2.0.0 released for katello / satellite

# Aug 15, 2023

* oci_env issues that jtanner had
    * `s6-rc: fatal: unable to take locks: Resource busy`
        * Hard to research online
        * [mikedep333] Will look into if I run int it myself.
    * `the next issue is that because of the chained restarts and config updates, we don't have the social auth postgres tables ... so something needs to run migrations again`
        * they applied a workaround: https://github.com/ansible/galaxy_ng/pull/1838/files

# Aug 8, 2023

* Idea (not proposal) for pulpcon: demo HA pulp in k8s
    * Services team will be doing HA soon (part of production plan)
    * Let's do a joint presentation: mikedep333 & hyagi
* * [mikedep333] Please re-explain the pulp-operator deployment modified function check
* Any changes with decko on leave?
    * dkliban and mikedep333 to maintain pulp-oci-images
* [mikedep333] [services] pulp-smash config issue

# Aug 1, 2023
* Enabling multiple pulp operators to run in a namespace
    * Needed because content-sources and image-builder will each have a pulp operator running in a single namespace
    * Except for a few singletons, almost all k8s objects will have names like "content-sources-pulp-api-abcdef" and "image-builder-pulp-api-abcdef"


# Jul 25, 2023
* mikedep333 to continue following up with user support requests / bug reports for pulp_installer

# Jul 19, 2023
* [CI dashboard](https://github.com/pulp/pulp-ci/blob/master/README.md) will be helpful
* for hosted-content: CI smoke tests with CJI IQE (requires writing plugin) or a custom bash test script that calls pytest?
    * Agreed: Write custom bash test script for now

# Jul 12, 2023
* We fixed pulp-oci-images CI last week
* Did the release for fixed images for prior pulp versions
* oci_env CI has the Ubuntu podman-compose fix now (from pulp-oci-images)

# Jun 27, 2023
* CI Fixes
    * https://github.com/pulp/pulp-oci-images/actions/runs/5391036777
        * Looks like another one is needed
        * Needs backport
        * Broken CI may be sustaining this bug: https://github.com/pulp/pulp-oci-images/issues/510

# Jun 20, 2023
* Haven't done the quick fix / release for [AAP-11911](https://issues.redhat.com/browse/AAP-11911) yet
* Hosted Content: Deploying to staging is making slow but steady progress
* Re-review please: https://github.com/pulp/pulp-oci-images/pull/508
* We need to publish a pulp-web image to be used on consoledot.
    * Agreed: Create another dockerfile in pulp-clowder-deployments repo. Build in same script.


# Jun 13, 2023
* Was asked to fix [AAP-11911](https://issues.redhat.com/browse/AAP-11911)
* Hosted Content: Deploying to staging is making steady progress
* 3.26 fixed release:
    * Not released yet, we just merged the PR to update the list of branches to build
    * Manually triggered a 3.26 CI run (pulp-oci-images CI) to build and push the 3.26 image

# Jun 6, 2023
* Hosted services: Mike working on deployment method
    * Plan to merge PR to pulp-clowder-deployments, and submit PR to app-interface today
* Did CI fixes so that 3.25 image is built and pushed

# May 30, 2023
* Hosted services: Mike working on deployment method
    * Working on the dummy script, trying things out
* A couple of open PRs / issues
    * Mike providing reviews, answering questions
* Mike helped Decko with UBI image limitations
    * Some packages exist in repos like "RHEL base" but not "UBI base"
    * Adding CentOS Stream Repos for now
    * If there's breakage, we can try the rockylinux / almalinux repos. Not using them now because their 8.9 release will lag behind RHEL 8.9 release by a few weeks.

# May 23, 2023
* [AAP-11911](https://issues.redhat.com/browse/AAP-11911) is actually on hold (bmbouter relayed this on 5/17)
* Work with the consulting team (Matt Dorn)?
    * No issues have come up
    * I think we wanted their review on this: https://github.com/pulp/pulp-operator/pull/898
        * Agreed: This is such a big review, we will focus as a team (with or without their help) on reviewing it when hyagi gets back, and establishes who needs this, and when.

# May 17, 2023
* [AAP-11911: Sensitive information is exposed in clear text in setup files when no verbosity is set.](https://issues.redhat.com/browse/AAP-11911)
    * Agreed: Wait for info on when it is due
* CI status and compose data loss bug
    * Agreed: Wait for 3.25 releases of plugins, paticularly pulp_ansible, so that stable tests can pass.
    * Done: Closed and re-opened PR to refresh the GHA cache


# Prior meetings:
* [Jul 2022 - May 2023](https://hackmd.io/B41_0NSATPCMPwKafok4TA?edit)

###### tags: `Minutes`