# Deployers Team Meeting

Triage Query:
https://github.com/pulp/pulp-oci-images/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed
https://github.com/pulp/pulp-operator/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed

# Apr 23
* No agenda items, just did triage

# Apr 16
* No agenda items, just did triage

# Apr 9
* No agenda items, just did triage

# Apr 2
* Fix for configurable py version
* https://github.com/pulp/pulp-oci-images/pull/622
* Need to create 3.49 branch

# Mar 26
* [Discourse post](https://discourse.pulpproject.org/t/containers-upgraded-to-centos-9/1165) on EL9

# Mar 19
* [CI Change / Python versions PR](https://github.com/pulp/pulp-oci-images/pull/606)

# Mar 12
* CI Change / Python versions PR
* galaxy-operator [now builds their own smaller images](https://github.com/ansible/galaxy-operator/issues/77#issuecomment-1979705823) instead of using ours
* TODO: Gerrod to find out if we can drop building galaxy images (altogether or just branches)

# Mar 5, 2024
* still need to write the blog post
* In the middle of reviewing the big CI change / Python versions PR

# Feb 27, 2024
* still need to write the blog post

# Feb 20, 2024
* Outline for blog post on EL9 upgrade
* Why we did it
* Newer python versions
* Other benefits?
* We're developing upstream - latest and greatest
* Python 3.8 EOL
* image naming
* image tags for python versions
* plugin-template updated
* remaining work:
* Fix NSS output issue that breaks oci_env (Daniel working on this FYI)
* EL8 images again
* Make multiple EL9 python versions available
* Old branches for pulpcore releases have not been created for the past ~20 releases
* https://github.com/pulp/pulp-oci-images/pull/597

# Feb 13, 2024
* S6 upgrade seems to have fixed the issue with the fix-perms feature.
* After fixing the postgres version detection properly, and some minor changes to the commits, we merged the EL9 upgrade!
* Remaining work
* EL8 builds also (Python 3.8)
* Multiple EL9 python versions
* Need to show off how I created the repos on dockerhub & quay

# Feb 7, 2024
* Fixed 2 Ci issues
* Pulp images are now publishing
* Helping Daniel with EL9 upgrade and the logic for fixing the permissions on the mounted folder
* Upgraded S6 (worth seeing if it fixes it)
* Design for python versions vs os versions
* We previously agreed on tags
* We previously agreed to create a new image, pulp_ci_centos9
* Old el8 pulp_ci_centos image will be kept around for both the old image branches and the old plugin template
* Python 3.8 is only on el8
* Python 3.9 is on both el8 and el9
* Python 3.11 is only on el9 (el9 needed for other reasons too)

# Jan 30, 2024
* pulp_installer bug reports and PRs
* Agreed: Close them
* Agreed: *Archive* the repository
* User is experiencing a problem with pulp-worker pods getting restarted when the DB connection drops
* The behavior is different from pulp-api pods
* Changing the behavior of pulp-worker would not fix the underlying problem that the DB is becoming unavailable

# Jan 23, 2024
* I merged a feature request to pulp-oci-images
* Now there's a [revert PR](https://github.com/pulp/pulp-oci-images/pull/581)

# Jan 16, 2024
* EL9 with Postgres version mismatch errors
* https://github.com/pulp/pulp-oci-images/pull/575#issuecomment-1891036347
* Mike to follow up with Daniel
* [Fixed links in docs](https://github.com/pulp/pulp-oci-images/pull/578), but wondering how to add indented/nested docs links on [navigation bar](https://docs.pulpproject.org/pulp_oci_images/) - [source](https://github.com/pulp/pulp-oci-images/blob/latest/mkdocs.yml#L29)

# Jan 10, 2024
* User requests for docker-compose with https
* Point them to nginx container docs
* Docs issue with navigation missing for a few pages

# Dec 19, 2023
* 1 PR to review:
* https://github.com/pulp/pulp-oci-images/pull/569
* We will not need to take turns to release the operator anymore because now we have a script to make the process easier

# Dec 12, 2023
* Multi-arch builds
* Seems to be reliable now!
* Mike will probably test on a Raspberry Pi 4 soon (forget if 2GB or 4GB)
* [Upgrading image to EL9 / python 3.9](https://github.com/pulp/pulp-oci-images/issues/537)
* We will check that no downstream products use python 3.8

# Dec 5, 2023

* Status of multi-arch builds
* Merged
* podman-compose up issue
* [dkliban] will try to reproduce again
* About the operator release cycle

# Nov 28, 2023
* [mikedep333] 2 things to review for pulp-operator
* [pulp-operator HA](https://docs.google.com/document/d/1PDYFRUNiLf-AoF9xu9st5-mTchE39X-0wuCf7sBGUR8/edit#heading=h.zgoda9dbx81w)
* [robot framework test code](https://github.com/git-hyagi/robot-framework-ocp-tests/tree/master/pulp)
* mikedep333 provided initial feedback
* We're confident in them, let's keep using them
* Status of multi-arch builds
* [decko] had to revert this last week
* mikedep333 to research multi-arch registries and review PR inbetween services work

# Nov 14, 2023

* [ttereshc] ansible branch for pulp-operator can be removed
* a discourse post would be nice to have
* [hyagi] We can replace it with a tag
* [mikedep333] This will preserve git history
* [mikedep333] I recommend renaming the branch to "deprecated-ansible" or "ansible-deprecated"
* Agreed: Renamed branch or tag
* [hyagi] ansible team is planning to use the galaxy-ng and galaxy-ui images in galaxy-operator, so, if they manage to make it work with these images, we can stop maintaining galaxy-web and galaxy-minimal
* Agreed: Have discussions in #pulp-deployments matrix channel or in the #pulp slack channel
* [dkliban] services team is putting on hold the request for the "watch all namespaces" feature
* Looking into using pulp resources directly managed by clowder framework rather than clowder calling pulp-operator
* [Ina] This would preclude the Open Operations Initiative
* [dkliban] These services can be open, the example of console.fedoraproject.org (upstream for insights)
* [dkliban] Clowder is specific to Red Hat currently, but could be used by other orgs.
* [dkliban] As an example of stuff pulp-operator does do that we'd have to reimplement, is watching the config objects and then regenerating pulp settings (and applying them)
* [Ina] There are people who test pulp-operator. Are they doing things to supplement it like how RH uses clowder?
* [dkliban] services team is exploring switching the deployments on console.redhat.com to using Clowder https://docs.google.com/document/d/1I-ZI90_xEQpEl5GQdUt-Mcgs6PAwhjy88AHNHAQ_9T4/edit#heading=h.60zkfjl40xzp
* [mikedep333] Status of fixing the compose data loss bug (PR 468)
* [probable issue](https://github.com/mikedep333/pulp-oci-images/actions/runs/6791023058/job/18493443085#step:14:202) with old podman & new podman-compose breaking the CI (the container is up)
* So I tried upgrading podman, and [now it's trying and failing](https://github.com/mikedep333/pulp-oci-images/actions/runs/6819758133/job/18547754631#step:10:205) to install old pulp-file for some weird pip dependency reason

# October 31, 2023
* Status of nested containers
* Fixed an issue whereby the nested container needed 64K + 1 UIDs from the outer container (because UID 0 is already mapped in Podman but not Docker)
* Hopefully [the CI](https://github.com/mikedep333/pulp-oci-images/actions/runs/6708668363) passes, we won't need to run "podman system migrate" in the inner container, and I can move onto the pulp-container PR.
* weird [CI breakage](https://github.com/pulp/pulp-oci-images/actions/runs/6700817758/job/18207480438#step:10:265) for pulp-ansible / pulpcore version
* pulp-operator dual Pulp deployment check
* Update on testing 2 instances of pulp-operator with kind (K8s on container instead of VM)
* we are maxing the limits of GHA CI
* We got the rudimentary check done

# October 25, 2023
* Status of nested containers
* Explored alternatives thoroughly, but approach is require outer containers to support UIDs up to 75535 (65535 + 10000)
* This avoids need for migration of existing containers
* This avoids security issues and potential compatibility of supporting UIDs inside the nested container of upto 65534.
* OpenShift is compatible [for now](https://issues.redhat.com/browse/RFE-3254) because they do not run containers in user namespaces and therefore have 4G UIDs available
* [tanya/hyagi] need clarification on impact on community because of "[HMS] Singleton operator - can watch all the namespaces"
* what is the use case
* at some point we moved from cluster-scoped to namespace-scoped operator, any concern in watching all namespaces now?
* will we provide one option or all the combinaitons possible?
* all
* [hyagi] CVE for operator framework
* [hyagi] Issue [#1121](https://github.com/pulp/pulp-operator/issues/1121)
* Does it affect pulp?
* Should we warn users about "galaxy-operator" migration?
* What is the correct approach to fix the error?
* [hyagi] In a galaxy thread on slack the following question arose:
* "Isn't it required to stop or scale down services before running migrations to prevent locks or conflicts?"*
* [dkliban] Not required

# October 17, 2023
* Status of nested containers
* Resumed investigating, the 3rd issue is with mounting /proc in the nested container
* pulp-operator roadmap
* We should have one.
* At least next month or 2 timeframe.
* Promote shared understanding to the team.

* Suggested items
* [dkliban][HMS] Add testing of dual pulp deployments in one namespace
* [hyagi]bring some samples that we will use in pulpcon presentation up to date
* [dkliban][HMS] fix how we release the operator bundle (wrong tag)
* [decko for now] release process automation
* [hyagi] CVE http/2
* [collaborative effort?][HMS] Singleton operator - can watch all the namespaces
* [good for newcomers] review and update the operator spec fields description/appearance in ocp
* refactor https://docs.google.com/document/d/1a9j7pVi7fv_Fym6TBSxDNEvPyZVqexUNns3jdcjhIrs/edit#heading=h.8vjxjmi9nsre (see 'code improvements')
* metadata signing needs to be fixed (dkliban to check if HMS is affected)

# October 10, 2023

* Resuming work on nested latest podman (for pulp_container CI)
* 2 issues fixed, at least 1 remaining
* Going to finally merge the data loss bug
* 1st need to rebase
* was delayed due to CI being red
* pulp-operator testing
* Modify an existing test to deploy 2 instances of the operator side-by-side, for image-builder + content-sources
* We are considering making our operator global, watching all namespaces
* (Another project) image is registered as a catalog source, as a bundle image. So they push these in CI, and their cluster-wide OLM operator knows to deploy a new version.
* Releasing to the catalog from the main branch?

# October 3, 2023

*

# September 12, 2023

* Need another pulp-operator release with the new entrypoint for pulpcore-api.
* How can we avoid carrying a separate entry point in the operator?

* Looks like we still have a single migration job running when we have two deployments of Pulp in one namespace.

# Sept 5, 2023

* backup of /var/lib/pulp/tmp content
* can we ignore this folder during backup? does it have anything that is important?

# Aug 29, 2023
* pulp_container CI is failing with latest pulp-ci-centos image
* There was a regression in the capabilities of newuidmap / newgidmap - easy to fix
* I noticed that we never set VOLUMES for /var/lib/containers & /var/lib/pulp/.local/share/containers
* Ask the pulp_container team about this.
* [Lots of other prescribed changes](https://www.redhat.com/sysadmin/podman-inside-container) too that we never did
* Agreed: Do not focus on all the other prescribed changes
* Agreed: File an issue

# Aug 22, 2023
* Compose data loss bug
* Fixing the migration of config files from multi-process container to compose
* Helping jhutar with performance testing cluster
* pulpcore-selinux 2.0.0 released for katello / satellite

# Aug 15, 2023

* oci_env issues that jtanner had
* `s6-rc: fatal: unable to take locks: Resource busy`
* Hard to research online
* [mikedep333] Will look into if I run int it myself.
* `the next issue is that because of the chained restarts and config updates, we don't have the social auth postgres tables ... so something needs to run migrations again`
* they applied a workaround: https://github.com/ansible/galaxy_ng/pull/1838/files

# Aug 8, 2023

* Idea (not proposal) for pulpcon: demo HA pulp in k8s
* Services team will be doing HA soon (part of production plan)
* Let's do a joint presentation: mikedep333 & hyagi
* * [mikedep333] Please re-explain the pulp-operator deployment modified function check
* Any changes with decko on leave?
* dkliban and mikedep333 to maintain pulp-oci-images
* [mikedep333] [services] pulp-smash config issue

# Aug 1, 2023
* Enabling multiple pulp operators to run in a namespace
* Needed because content-sources and image-builder will each have a pulp operator running in a single namespace
* Except for a few singletons, almost all k8s objects will have names like "content-sources-pulp-api-abcdef" and "image-builder-pulp-api-abcdef"

# Jul 25, 2023
* mikedep333 to continue following up with user support requests / bug reports for pulp_installer

# Jul 19, 2023
* [CI dashboard](https://github.com/pulp/pulp-ci/blob/master/README.md) will be helpful
* for hosted-content: CI smoke tests with CJI IQE (requires writing plugin) or a custom bash test script that calls pytest?
* Agreed: Write custom bash test script for now

# Jul 12, 2023
* We fixed pulp-oci-images CI last week
* Did the release for fixed images for prior pulp versions
* oci_env CI has the Ubuntu podman-compose fix now (from pulp-oci-images)

# Jun 27, 2023
* CI Fixes
* https://github.com/pulp/pulp-oci-images/actions/runs/5391036777
* Looks like another one is needed
* Needs backport
* Broken CI may be sustaining this bug: https://github.com/pulp/pulp-oci-images/issues/510

# Jun 20, 2023
* Haven't done the quick fix / release for [AAP-11911](https://issues.redhat.com/browse/AAP-11911) yet
* Hosted Content: Deploying to staging is making slow but steady progress
* Re-review please: https://github.com/pulp/pulp-oci-images/pull/508
* We need to publish a pulp-web image to be used on consoledot.
* Agreed: Create another dockerfile in pulp-clowder-deployments repo. Build in same script.

# Jun 13, 2023
* Was asked to fix [AAP-11911](https://issues.redhat.com/browse/AAP-11911)
* Hosted Content: Deploying to staging is making steady progress
* 3.26 fixed release:
* Not released yet, we just merged the PR to update the list of branches to build
* Manually triggered a 3.26 CI run (pulp-oci-images CI) to build and push the 3.26 image

# Jun 6, 2023
* Hosted services: Mike working on deployment method
* Plan to merge PR to pulp-clowder-deployments, and submit PR to app-interface today
* Did CI fixes so that 3.25 image is built and pushed

# May 30, 2023
* Hosted services: Mike working on deployment method
* Working on the dummy script, trying things out
* A couple of open PRs / issues
* Mike providing reviews, answering questions
* Mike helped Decko with UBI image limitations
* Some packages exist in repos like "RHEL base" but not "UBI base"
* Adding CentOS Stream Repos for now
* If there's breakage, we can try the rockylinux / almalinux repos. Not using them now because their 8.9 release will lag behind RHEL 8.9 release by a few weeks.

# May 23, 2023
* [AAP-11911](https://issues.redhat.com/browse/AAP-11911) is actually on hold (bmbouter relayed this on 5/17)
* Work with the consulting team (Matt Dorn)?
* No issues have come up
* I think we wanted their review on this: https://github.com/pulp/pulp-operator/pull/898
* Agreed: This is such a big review, we will focus as a team (with or without their help) on reviewing it when hyagi gets back, and establishes who needs this, and when.

# May 17, 2023
* [AAP-11911: Sensitive information is exposed in clear text in setup files when no verbosity is set.](https://issues.redhat.com/browse/AAP-11911)
* Agreed: Wait for info on when it is due
* CI status and compose data loss bug
* Agreed: Wait for 3.25 releases of plugins, paticularly pulp_ansible, so that stable tests can pass.
* Done: Closed and re-opened PR to refresh the GHA cache

# Prior meetings:
* [Jul 2022 - May 2023](https://hackmd.io/B41_0NSATPCMPwKafok4TA?edit)

###### tags: `Minutes`