# Deployers Team Meeting Triage Query: https://github.com/pulp/pulp-oci-images/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed https://github.com/pulp/pulp-operator/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-Needed # Feb 27, 2024 * still need to write the blog post # Feb 20, 2024 * Outline for blog post on EL9 upgrade * Why we did it * Newer python versions * Other benefits? * We're developing upstream - latest and greatest * Python 3.8 EOL * image naming * image tags for python versions * plugin-template updated * remaining work: * Fix NSS output issue that breaks oci_env (Daniel working on this FYI) * EL8 images again * Make multiple EL9 python versions available * Old branches for pulpcore releases have not been created for the past ~20 releases * https://github.com/pulp/pulp-oci-images/pull/597 # Feb 13, 2024 * S6 upgrade seems to have fixed the issue with the fix-perms feature. * After fixing the postgres version detection properly, and some minor changes to the commits, we merged the EL9 upgrade! * Remaining work * EL8 builds also (Python 3.8) * Multiple EL9 python versions * Need to show off how I created the repos on dockerhub & quay # Feb 7, 2024 * Fixed 2 Ci issues * Pulp images are now publishing * Helping Daniel with EL9 upgrade and the logic for fixing the permissions on the mounted folder * Upgraded S6 (worth seeing if it fixes it) * Design for python versions vs os versions * We previously agreed on tags * We previously agreed to create a new image, pulp_ci_centos9 * Old el8 pulp_ci_centos image will be kept around for both the old image branches and the old plugin template * Python 3.8 is only on el8 * Python 3.9 is on both el8 and el9 * Python 3.11 is only on el9 (el9 needed for other reasons too) # Jan 30, 2024 * pulp_installer bug reports and PRs * Agreed: Close them * Agreed: *Archive* the repository * User is experiencing a problem with pulp-worker pods getting restarted when the DB connection drops * The behavior is different from pulp-api pods * Changing the behavior of pulp-worker would not fix the underlying problem that the DB is becoming unavailable # Jan 23, 2024 * I merged a feature request to pulp-oci-images * Now there's a [revert PR](https://github.com/pulp/pulp-oci-images/pull/581) # Jan 16, 2024 * EL9 with Postgres version mismatch errors * https://github.com/pulp/pulp-oci-images/pull/575#issuecomment-1891036347 * Mike to follow up with Daniel * [Fixed links in docs](https://github.com/pulp/pulp-oci-images/pull/578), but wondering how to add indented/nested docs links on [navigation bar](https://docs.pulpproject.org/pulp_oci_images/) - [source](https://github.com/pulp/pulp-oci-images/blob/latest/mkdocs.yml#L29) # Jan 10, 2024 * User requests for docker-compose with https * Point them to nginx container docs * Docs issue with navigation missing for a few pages # Dec 19, 2023 * 1 PR to review: * https://github.com/pulp/pulp-oci-images/pull/569 * We will not need to take turns to release the operator anymore because now we have a script to make the process easier # Dec 12, 2023 * Multi-arch builds * Seems to be reliable now! * Mike will probably test on a Raspberry Pi 4 soon (forget if 2GB or 4GB) * [Upgrading image to EL9 / python 3.9](https://github.com/pulp/pulp-oci-images/issues/537) * We will check that no downstream products use python 3.8 # Dec 5, 2023 * Status of multi-arch builds * Merged * podman-compose up issue * [dkliban] will try to reproduce again * About the operator release cycle # Nov 28, 2023 * [mikedep333] 2 things to review for pulp-operator * [pulp-operator HA](https://docs.google.com/document/d/1PDYFRUNiLf-AoF9xu9st5-mTchE39X-0wuCf7sBGUR8/edit#heading=h.zgoda9dbx81w) * [robot framework test code](https://github.com/git-hyagi/robot-framework-ocp-tests/tree/master/pulp) * mikedep333 provided initial feedback * We're confident in them, let's keep using them * Status of multi-arch builds * [decko] had to revert this last week * mikedep333 to research multi-arch registries and review PR inbetween services work # Nov 14, 2023 * [ttereshc] ansible branch for pulp-operator can be removed * a discourse post would be nice to have * [hyagi] We can replace it with a tag * [mikedep333] This will preserve git history * [mikedep333] I recommend renaming the branch to "deprecated-ansible" or "ansible-deprecated" * Agreed: Renamed branch or tag * [hyagi] ansible team is planning to use the galaxy-ng and galaxy-ui images in galaxy-operator, so, if they manage to make it work with these images, we can stop maintaining galaxy-web and galaxy-minimal * Agreed: Have discussions in #pulp-deployments matrix channel or in the #pulp slack channel * [dkliban] services team is putting on hold the request for the "watch all namespaces" feature * Looking into using pulp resources directly managed by clowder framework rather than clowder calling pulp-operator * [Ina] This would preclude the Open Operations Initiative * [dkliban] These services can be open, the example of console.fedoraproject.org (upstream for insights) * [dkliban] Clowder is specific to Red Hat currently, but could be used by other orgs. * [dkliban] As an example of stuff pulp-operator does do that we'd have to reimplement, is watching the config objects and then regenerating pulp settings (and applying them) * [Ina] There are people who test pulp-operator. Are they doing things to supplement it like how RH uses clowder? * [dkliban] services team is exploring switching the deployments on console.redhat.com to using Clowder https://docs.google.com/document/d/1I-ZI90_xEQpEl5GQdUt-Mcgs6PAwhjy88AHNHAQ_9T4/edit#heading=h.60zkfjl40xzp * [mikedep333] Status of fixing the compose data loss bug (PR 468) * [probable issue](https://github.com/mikedep333/pulp-oci-images/actions/runs/6791023058/job/18493443085#step:14:202) with old podman & new podman-compose breaking the CI (the container is up) * So I tried upgrading podman, and [now it's trying and failing](https://github.com/mikedep333/pulp-oci-images/actions/runs/6819758133/job/18547754631#step:10:205) to install old pulp-file for some weird pip dependency reason # October 31, 2023 * Status of nested containers * Fixed an issue whereby the nested container needed 64K + 1 UIDs from the outer container (because UID 0 is already mapped in Podman but not Docker) * Hopefully [the CI](https://github.com/mikedep333/pulp-oci-images/actions/runs/6708668363) passes, we won't need to run "podman system migrate" in the inner container, and I can move onto the pulp-container PR. * weird [CI breakage](https://github.com/pulp/pulp-oci-images/actions/runs/6700817758/job/18207480438#step:10:265) for pulp-ansible / pulpcore version * pulp-operator dual Pulp deployment check * Update on testing 2 instances of pulp-operator with kind (K8s on container instead of VM) * we are maxing the limits of GHA CI * We got the rudimentary check done # October 25, 2023 * Status of nested containers * Explored alternatives thoroughly, but approach is require outer containers to support UIDs up to 75535 (65535 + 10000) * This avoids need for migration of existing containers * This avoids security issues and potential compatibility of supporting UIDs inside the nested container of upto 65534. * OpenShift is compatible [for now](https://issues.redhat.com/browse/RFE-3254) because they do not run containers in user namespaces and therefore have 4G UIDs available * [tanya/hyagi] need clarification on impact on community because of "[HMS] Singleton operator - can watch all the namespaces" * what is the use case * at some point we moved from cluster-scoped to namespace-scoped operator, any concern in watching all namespaces now? * will we provide one option or all the combinaitons possible? * all * [hyagi] CVE for operator framework * [hyagi] Issue [#1121](https://github.com/pulp/pulp-operator/issues/1121) * Does it affect pulp? * Should we warn users about "galaxy-operator" migration? * What is the correct approach to fix the error? * [hyagi] In a galaxy thread on slack the following question arose: * "Isn't it required to stop or scale down services before running migrations to prevent locks or conflicts?"* * [dkliban] Not required # October 17, 2023 * Status of nested containers * Resumed investigating, the 3rd issue is with mounting /proc in the nested container * pulp-operator roadmap * We should have one. * At least next month or 2 timeframe. * Promote shared understanding to the team. * Suggested items * [dkliban][HMS] Add testing of dual pulp deployments in one namespace * [hyagi]bring some samples that we will use in pulpcon presentation up to date * [dkliban][HMS] fix how we release the operator bundle (wrong tag) * [decko for now] release process automation * [hyagi] CVE http/2 * [collaborative effort?][HMS] Singleton operator - can watch all the namespaces * [good for newcomers] review and update the operator spec fields description/appearance in ocp * refactor https://docs.google.com/document/d/1a9j7pVi7fv_Fym6TBSxDNEvPyZVqexUNns3jdcjhIrs/edit#heading=h.8vjxjmi9nsre (see 'code improvements') * metadata signing needs to be fixed (dkliban to check if HMS is affected) # October 10, 2023 * Resuming work on nested latest podman (for pulp_container CI) * 2 issues fixed, at least 1 remaining * Going to finally merge the data loss bug * 1st need to rebase * was delayed due to CI being red * pulp-operator testing * Modify an existing test to deploy 2 instances of the operator side-by-side, for image-builder + content-sources * We are considering making our operator global, watching all namespaces * (Another project) image is registered as a catalog source, as a bundle image. So they push these in CI, and their cluster-wide OLM operator knows to deploy a new version. * Releasing to the catalog from the main branch? # October 3, 2023 * # September 12, 2023 * Need another pulp-operator release with the new entrypoint for pulpcore-api. * How can we avoid carrying a separate entry point in the operator? * Looks like we still have a single migration job running when we have two deployments of Pulp in one namespace. # Sept 5, 2023 * backup of /var/lib/pulp/tmp content * can we ignore this folder during backup? does it have anything that is important? # Aug 29, 2023 * pulp_container CI is failing with latest pulp-ci-centos image * There was a regression in the capabilities of newuidmap / newgidmap - easy to fix * I noticed that we never set VOLUMES for /var/lib/containers & /var/lib/pulp/.local/share/containers * Ask the pulp_container team about this. * [Lots of other prescribed changes](https://www.redhat.com/sysadmin/podman-inside-container) too that we never did * Agreed: Do not focus on all the other prescribed changes * Agreed: File an issue # Aug 22, 2023 * Compose data loss bug * Fixing the migration of config files from multi-process container to compose * Helping jhutar with performance testing cluster * pulpcore-selinux 2.0.0 released for katello / satellite # Aug 15, 2023 * oci_env issues that jtanner had * `s6-rc: fatal: unable to take locks: Resource busy` * Hard to research online * [mikedep333] Will look into if I run int it myself. * `the next issue is that because of the chained restarts and config updates, we don't have the social auth postgres tables ... so something needs to run migrations again` * they applied a workaround: https://github.com/ansible/galaxy_ng/pull/1838/files # Aug 8, 2023 * Idea (not proposal) for pulpcon: demo HA pulp in k8s * Services team will be doing HA soon (part of production plan) * Let's do a joint presentation: mikedep333 & hyagi * * [mikedep333] Please re-explain the pulp-operator deployment modified function check * Any changes with decko on leave? * dkliban and mikedep333 to maintain pulp-oci-images * [mikedep333] [services] pulp-smash config issue # Aug 1, 2023 * Enabling multiple pulp operators to run in a namespace * Needed because content-sources and image-builder will each have a pulp operator running in a single namespace * Except for a few singletons, almost all k8s objects will have names like "content-sources-pulp-api-abcdef" and "image-builder-pulp-api-abcdef" # Jul 25, 2023 * mikedep333 to continue following up with user support requests / bug reports for pulp_installer # Jul 19, 2023 * [CI dashboard](https://github.com/pulp/pulp-ci/blob/master/README.md) will be helpful * for hosted-content: CI smoke tests with CJI IQE (requires writing plugin) or a custom bash test script that calls pytest? * Agreed: Write custom bash test script for now # Jul 12, 2023 * We fixed pulp-oci-images CI last week * Did the release for fixed images for prior pulp versions * oci_env CI has the Ubuntu podman-compose fix now (from pulp-oci-images) # Jun 27, 2023 * CI Fixes * https://github.com/pulp/pulp-oci-images/actions/runs/5391036777 * Looks like another one is needed * Needs backport * Broken CI may be sustaining this bug: https://github.com/pulp/pulp-oci-images/issues/510 # Jun 20, 2023 * Haven't done the quick fix / release for [AAP-11911](https://issues.redhat.com/browse/AAP-11911) yet * Hosted Content: Deploying to staging is making slow but steady progress * Re-review please: https://github.com/pulp/pulp-oci-images/pull/508 * We need to publish a pulp-web image to be used on consoledot. * Agreed: Create another dockerfile in pulp-clowder-deployments repo. Build in same script. # Jun 13, 2023 * Was asked to fix [AAP-11911](https://issues.redhat.com/browse/AAP-11911) * Hosted Content: Deploying to staging is making steady progress * 3.26 fixed release: * Not released yet, we just merged the PR to update the list of branches to build * Manually triggered a 3.26 CI run (pulp-oci-images CI) to build and push the 3.26 image # Jun 6, 2023 * Hosted services: Mike working on deployment method * Plan to merge PR to pulp-clowder-deployments, and submit PR to app-interface today * Did CI fixes so that 3.25 image is built and pushed # May 30, 2023 * Hosted services: Mike working on deployment method * Working on the dummy script, trying things out * A couple of open PRs / issues * Mike providing reviews, answering questions * Mike helped Decko with UBI image limitations * Some packages exist in repos like "RHEL base" but not "UBI base" * Adding CentOS Stream Repos for now * If there's breakage, we can try the rockylinux / almalinux repos. Not using them now because their 8.9 release will lag behind RHEL 8.9 release by a few weeks. # May 23, 2023 * [AAP-11911](https://issues.redhat.com/browse/AAP-11911) is actually on hold (bmbouter relayed this on 5/17) * Work with the consulting team (Matt Dorn)? * No issues have come up * I think we wanted their review on this: https://github.com/pulp/pulp-operator/pull/898 * Agreed: This is such a big review, we will focus as a team (with or without their help) on reviewing it when hyagi gets back, and establishes who needs this, and when. # May 17, 2023 * [AAP-11911: Sensitive information is exposed in clear text in setup files when no verbosity is set.](https://issues.redhat.com/browse/AAP-11911) * Agreed: Wait for info on when it is due * CI status and compose data loss bug * Agreed: Wait for 3.25 releases of plugins, paticularly pulp_ansible, so that stable tests can pass. * Done: Closed and re-opened PR to refresh the GHA cache # Prior meetings: * [Jul 2022 - May 2023](https://hackmd.io/B41_0NSATPCMPwKafok4TA?edit) ###### tags: `Minutes`