Pulpcore team meeting

# Pulpcore team meeting ## Overview * [Core SME List](https://hackmd.io/@pulp/core_sme) * Release rotation (2 months) * RELEASING: https://github.com/pulp/pulpcore/blob/main/releasing.md * CURRENT: dalley - November, December * mdellweg ipanova ggainey decko gerrod dkliban dalley * Meeting lead (2 months) * CURRENT: dkliban - November, December * ggainey gerrod mdellweg decko ipanova dalley * Github issues in the last 7 days * https://github.com/pulp/pulpcore/issues?q=is%3Aissue+is%3Aopen+created%3A%3E2023-01-17 * Security Alerts: - https://github.com/pulp/pulpcore/security/code-scanning ## Meeting Agenda * Pending-AI review * Upcoming items * Open PR review * [core non-draft open](https://github.com/pulp/pulpcore/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse+) * Collect Action Items, copy to Pending-AI ## Pending AI * AI: pypi, project-ownership, and organizations * ggainey - review/prune pypi projects * 20-NOV-2024 all versions of 20 projects yanked. DONE * ~~Q: Yank, or DELETE?~~ * ~~Current plan is to do one of the above to anything that hasn't been updated after 2022~~ * ~~consensus - do the yank~~ * ~~AI: ggainey to this by 29-OCT~~ * ~~or later in 2024..~~ * https://github.com/pulp/pulpcore/issues/5931 * AI: bmbouter/services-team will See What Happens if CONTENT_ORIGIN isn't set # Upcoming * 3.70 is very soon * Think about deleting old migrations in Pulpcore. # November 12, 2024 * https://github.com/pulp/pulpcore-selinux * does *anybody* have *any* selinux expertise? * 4 (ignored) PRs waiting * escalate to mgt * can we get an selinux-expert from Elsewhere, to agree to 'own' this? * contingency: Merge And Hope * https://github.com/pulp/pulpcore/issues/5931 * multiple FQDN gateways * what happens if we just allow CONTENT_ORIGIN to be empty? * can we get an experiment to see how this works? * just return relative-urls? * AI: bmbouter/services-team will See What Happens * can we build "absolute URLs" based on incoming request? * maaaaaybe - but, it is (or can be) Very Complicated * RFE: multiple URLs on a remote * https://issues.redhat.com/browse/SAT-26317 * not RPM - would be core * or, does katello manage multiple-remotes for the repo, and retry-on-failures? * does order matter? how is it maintained? * how does this interact with RemoteArtifacts? * RAs have their own set of...suboptimality, already * how does this interact with ACS? * Can the remote-repos have diff characteristics? (eg, auth?) * net of discussion: * several possible avenues * all have pitfalls * How important is this, Really? * it's a really specific edge-case * escalate to Satellite PM for prioritization # October 29, 2024 * Squashing Migrations * We will continue un this (maybe in a Plugin first) * We will require plugins to rebase their migration dependencies on a not too old pulpcore migration with the next breaking change release. * OTEL update, it's brokeness, and what we can do about it * auto-instrumentation is broken * the problems are near endless * we're replacing it with a bit of our own middleware and the SDK * will simplify oci-images which required special handling for OTEL auto-inst * Add 3.63 to supported branches? # October 22, 2024 * Merging migrations redux * we have A LOT of migrations pre-3.20 * last supported branch is 3.21 * released Sep-2022 * can pick something before 3.21 if we like * prob w/ squashing is at upgrade-time * makes pre-to-post-squash take two steps * would want to squash plugins first, if/when we do this * due to plugins dependent on core migrations * teach plugins to be dependent on "last core migration in oldest compatible core LB" at next breaking change point? * *absolutely* have to consider downstream implementations * need to have conversation with downstream because this affects upgrade instructions * downstream *already* deals with this to handle OS-upgrade-transitions * what is (if any) the restriction on multi-release-upgrades for downstream? * Look at Pulp 4 planning hackmd written by ggainey & give him feedback https://hackmd.io/@ggainey/pulp4_proposal * Pulp4 proposal discussion * packaging? * AI: [all] any review comments to ggainey by next core-team mtg # October 15, 2024 * core/3.70 (breaking change) is coming - announce it, decide on next breaking-change? (3.85?) * do we even have "breaking" plugin-api changes? can't think of any? * PRN changes? - it's just a new field, shouldn't break anything, and is already merged * Pulp4 presentation draft - available no later than 22-OCT, ggainey to send draft-link to core-team list as soon as there is something to send # October 8, 2024 - oci-env failing recently with "pip not found" - some problems seem to be resolved by switching the binary from podman to docker - Pulp 4 discussion - we probably do want to provide a v4 API in parallel with v3 in Pulp 3 - plan: - /v4/ "shows up" in core/3.X line, as tech-preview - /v3/ and /v4/ co-exist - limited changes (prn/domain/acess-policies cfg-based only/remove-artifact) - at "some point", a core/4.0 is released - /v4/ is now out of preview - /v3/ still exists *but is unmaintained* - at "some point" we release a core/5.0 - *at that point* the /v3/ API is *removed* - /v4/ moves to "exists but no longer maintained" - AI: [ggainey] agreed to own slides/presentation at PulpCON for this, and demo at some core-team pre-pulpcon # October 1, 2024 - OpenPGP PR - https://github.com/pulp/pulpcore/pull/5133 - OTEL metrics reporting - https://github.com/pulp/pulpcore/issues/5845 - Conclusion: add additional metrics to a ScheduledTask, but add a comment that the task otel reporting needs to be left in the worker code # September 17, 2024 - [content in repo version set calculation is slow](https://github.com/pulp/pulpcore/issues/5783) - what experiments can we run here to evaluate any PoC against? - example: - script to populate a repository with "many" entities - remove each entity (creating a new version) till empty, then add them all back one at a time - at each step, measure "amount of existing content" and "time to answer the question" - graph and consider the implications :) - another example: as above. but with a "broader" instance (ie, more repos and hence larger table sizes) - Telemetry worker - https://github.com/pulp/pulpcore/pull/5761 - Common patterns - Where the queries and telemetry related stuff should run? Time-based. - There are things we rely on middlewares. Events. # September 10, 2024 - https://github.com/pulp/plugin_template/pull/910 - users are asking if there is a place they can contribute scripts they've written https://github.com/pulp/pulp_container/issues/1749#issuecomment-2315236062 - could just go in a gist from the user? have a place for links? - discourse? - docs-section? - we don't want to assume support for such things - consensus: - users can post to discourse first - have a section 'somewhere' ("Utility Scripts"?) in docs for "list of user script links" - we can pull things in to such a section as they show up? - pulp_labels work is progressing - who would be interested in commenting on a POC based on discussions from last week? - bmbouters, mdellweg, pbrochado already involved - need one reviewer - mdellweg - Q: All entities? Or just Content? - for now - just Content - comment: The nature of shared-content makes this...suboptimal. There are still objections to adding to Content - but we're moving forward, even tho there are downsides - Reproducible builds may conflict with this design. - comment: Run the draft-PR API past COPR before committing - comment: need to be able to find all the repo-versions "containing labelled content"? - also "all repositories by label" - Q-filter ways of doing this? - comment: the "search-and-destroy" problem is *separate* from this feature - PulpCon 2024 CfP - anything to add? - see draft in [planning doc](https://hackmd.io/@pulp/PulpCon_2024) - ggainey to post to discourse by 1200 EDT - if you think of any themes before then, add directly to draft in planning doc # August 27 - [replicate] q filter https://github.com/pulp/pulpcore/pull/5733#discussion_r1731383753 - Yes, it's techpreview. - We want to replace the old filter right away. - We need to think about the DB changes however: - ZDU is a "do it right everytime." # August 20 * https://github.com/pulp/pulpcore/issues/5087 * would like to get attention on this if/as people have some time # August 13th * High memory use during repository delete - https://github.com/pulp/pulpcore/issues/5048 * draft PR https://github.com/pulp/pulpcore/pull/5049 * some table-relationship-edge-cases to be evaluated * to get a 90% fix out quickly - maybe bundle up the odd/hypothetical-edge-cases into throwing "this isn't handled" exception, to be caught (if ever) in testing * can we get this tested in Staging? - work with services on that * want to make sure this delete-work doesn't inadvertently leave around "unreachable" entities post-delete * pulp_label_select improvement for replicate - https://github.com/pulp/pulpcore/issues/5087 * existing Q-filter should be available to selecting-labels already * BUT - upstream-replicant needs an API that *allows* specifying to a q-filter * maybe add a new API call to alow downstream to take advantage? * Add a q_select parameter to the UpstreamPulp API. This parameter will be used to filter distributions. It will have the same syntax as the q filter everywhere else in the API. * https://github.com/pulp/pulpcore/issues/3338#issuecomment-2255898417 COPR would like labels-for-content * can we answer this by labelling Content? * collides with RBAC "ownership" * but is MUCH simpler, both for implementation and "user expectation" of a class of Pulp users (ie, single-user Pulp installs) * or do we need to add labels to RepositoryContent? * or should it be per-repository-version? * fails in the "deleting versions collapses content-added into the remaining repo-version" * is this a core-feature, or should it be plugin-specific (ie, in rpm) * ansible has "marks" * more general, more resource-hungry, not-currently-in-active-use * AI: [ggainey] summarize discussion into the above issue * https://github.com/pulp/pulpcore/issues/3338#issuecomment-2286363401 * issue with post-migrate commands when there aren't any actual migrations * do we need to explicitly say "any package upgrade needs to be followed by 'migrate'"? * that's kind of the current "assumption" # August 6th * Feedback request: Domain RBAC for Services * See this doc for background: https://docs.google.com/document/d/1U3PK642P3kz4TVgyDFul2Wktf641v2T5KseGA7TTpTc/edit * Is there an alternative that does not require writing any code? * What alternatives have we not considered? * What is the right way to store the Organization ID? A label on a domain? A new table that is in the downstream? * What to do about the remaining non-domain-namespaced endpoints? # July 30th * RBAC for content - is that a thing? * based on repository-permissions * should answer dkliban/lzap's needs * ostree isn't using SingleArtifactContentSerializer...somewhere? (in import-all) * Labels for Content * COPR request - user added to [issue #3338](https://github.com/pulp/pulpcore/issues/3338) * Q: why is pulp_labels not a Global Thing? * initial implementation was generic-relationship on basemodel * horribly unperformant * refactored to use HStore * discussion about Content * shared-content causes shared-labels * this was before-domains * maybe it makes sense to do it now? * how about following the pulp_ansible content-mark content-type? * what about labels on repositorycontent model? * also keeps labels specific to repository * We could add pulp_label to repository_version * Not conflicting the other solutions, but accompany them. * think about how/if this violates our "repoversion is **IMMUTABLE**" rule - do we even care? * would also make dedup/RBAC a moot point - you'd have to have repo-RBAC-access already anyway # July 23rd * https://github.com/pulp/pulp-openapi-generator/issues/85 * we def need to version that repo * what versioning-scheme do we need? * do we need to make versioned-fixes? * probably yes - to be able to fix things for older-supported-releases * how do we match a specified core version, to the "correct" openapi version? * how does this interact with "plugins can be associated with different versions of pulpcore"? * how about using calendar-versioning (ie YYYYMMDD.0) * easier to map "when" a thing happened than, say, hash-versioning which requires an extra trip to the GH repo * semver doesn't actually give us much here - we're not really in control of how-much-changed in openapi * use a .0 even if we're only going to branch/support if we really have to - that way tools already admit to the possibility of a "release" * this spotlights (again) that bindings should be created, on-demand, to match the actual installed versions in the specific pulp-instance they are talking to * see ["Stop publishing bindings"](https://discourse.pulpproject.org/t/we-will-stop-publishing-bindings-soon/1240/23) thread on discourse * Enabling SHA1 digests on an existing deployment. What should happen? * discussion ensues # July 16th * discussion around template-fix * ggainey to sheperd backports as needed (3.21/22) * discussion around dependabot Weirdness * commit that broke us has been reverted (for now) * mdellweg to get currently-open-pulpcore-PRs recreated * all: pay attention to how to do this, and do it for "your" plugin * HOWEVER - we need to be aware of potentially being broken "again" at some point * "widen" currently shows a Scary Warning when used * being explicit about that now, *may* teach us to ignore Scary Error Messages (which is suboptimal) # July 9th * pyPI project ownership * current state is...a mess * some projects only have one owner * some have at least one owner whose pyPI login does NOT have 2FA enabled * some projects under pulp-login are not owned by the pulp login (?!?) * every project under pulp needs to have pulp as one of its owners * need to have at least 3 people, with pyPI logins, that are 2FA enabled, be added as Owner to *every single project* owned by pulp * anyone whose login is *not* 2FA-protected, needs to be not-an-owner * there are 62 projects involved * ggainey volunteers to do the tedious work * proposed minimum-set of owners: ggainey, mdellweg, dalley? * anyone else who wishes can be added - but see above RE 'pyPI login, 2FA turned on' * Conclusions: * ggainey to investigate pyPI Organizations' suitability for Pulp * https://blog.pypi.org/posts/2023-04-23-introducing-pypi-organizations/ * ggainey to drive the below * identify pyPI Ownership Team * mdellweg, dalley, one services-mini-team member (bmbouters to take back to team) * that team prunes list of projects * ggainey updates pyPI ownership of remaining projects * https://hackmd.io/@ggainey/task_throttling_thoughts * issue https://github.com/pulp/pulpcore/issues/5552 # July 02 * pyPI Next updates * Grant needs to find some folk with 2FA working. * mdellweg gonna help with it * ggainey's POC for pulp-cli * https://github.com/pulp/pulp-cli/pull/1003 * discussions had, working on cleanup and finalizing 1003 * mdellweg - dependabot PRs messing with the dependencies' lower bounds * couldn't find any docs to configure this aspect * previous dependabot PR's didn't touched the lower bounds * let's ignore this update and wait for a week # June 25 - Ruby bindings issue? https://github.com/pulp/pulp_rpm/issues/3639 - Repository gets created, but then it throws an `uninitialized constant` error from Ruby - Has anyone seen this? - Can it be transferred? - pulp is doing The Right Thing, but something in the response is confusing the bindings-code - what's changed? - what version of openapi-gen? (looks like "no") - what version of pulpcore are the bindings generated against? - target to investigate: is it the "new puilp_rpm/old-pulpcore" mismatch causing us grief? - may be only rpm-plugin-version that changed - someone needs to sit down w/ iballou and pair-program our way through debugging - see potential problems w/ "us" generating bindings [here](https://discourse.pulpproject.org/t/we-will-stop-publishing-bindings-soon/1240) - as of "now" it is Way Easier to self-gen than when that thread was opened - discussion around Artifact upload in console.redhat.com - currently, there *is* a bug where any-user can upload a naked Artifact - "who can see naked artifacts" - only the admin (actually any user can, same behavior as upload) - what needs to happen to allow a user to upload into-a-repository - you need to have upload-creator-role to allow chunked uploads - there *was* an issue with ostree-uploads - but it's fixed - multiple-files-uploaded to create one-repo-version discussion - proposal: upload into a temp-repo, then modify target-repo with a single modify (or rpm/copy) command - "modify" can require a large list of specified content-units - "copy" (available plugin-specific, e.g. rpm) lets one specify "everything from Repo-A into Repo-B" - proposal: add orphaned artifacts/content to repo - "naked" content can currently only be seen by an admin - content is immutable and shared, so no one user can "own" - how do you resolve permissions/protection on content in order to preserve "ownership" of content? - discussion around [saving task diagnostics](https://github.com/pulp/pulpcore/issues/5422) - lots of collaboration - open question: how do we report "artifact saved" to the user - log it? - makes self-service hard/impossible - save as created-resource? - not in a repo, just "naked" content - unexplored territory for Pulp - see 2024-06-18 meeting minutes for a LOT of previous discussion - break up the problem? - Step 1: create/report (logging only) the created artifact via UUID - Step 2: provide an API that could provide access to that UUID (assuming not cleaned up) - what if we just provide logging option *first*? - AI: bmbouter to post a summary in ticket # June 18 - Requests issue is "solved". - Do not install requests 2.32.{0,1,2,3}. - see https://github.com/psf/requests/issues/6730 - *do not* "just accept" any dependabot-requests-update - CI is failing all over the place. - mostly docs-related - pulp_deb changelog issue - fixed now - not the only issue - take a sec to look into https://github.com/pulp/pulp-ci and see if you can fix something - Saving task diagnostics as an artifact - https://github.com/pulp/pulpcore/issues/5422 - we need "a good story on how we would retrieve the resulting artifact" - maybe a new REST API? - make sure it only works for admin - already an upstream-request for "download an artifact" - https://github.com/pulp/pulpcore/issues/5003 - some otel links that they're at least considering - https://opentelemetry.io/blog/2024/profiling/ - https://thenewstack.io/metrics-traces-logs-and-now-opentelemetry-profile-data/ - https://github.com/open-telemetry/oteps/blob/main/text/profiles/0239-profiles-data-model.md - are these ready-for-use? are they fit-for-purpose? - not quite there yet - [dalley] but neither is our current solution * leaks memory by collecting all call stacks in memory, this is significant after a few minutes https://github.com/joerick/pyinstrument/issues/169 * doesn't handle sync->async transitions gracefully - profiling the "wrong kinds of things" can run **the entire pulp instance** out of memory and crash it - specifically, long-running outliers (e.g., large imports) - significant concern that needs to be resolved **prior to** addressing this - thought: can current-diagnostics-output be handed to otel as-is? (rather than keeping in pulp or using not-quite-ready otel-diagnostics?) - core/3.55 release concerns - can't go into one-container unless/until all installed plugins have had compatibility releases - let's get core/3.55 out "today" - plugins should update core/UB to <3.70 - [ggainey] add release-notification/reminder to existing discourse thread - [dkliban] reach out to R-plugin authors - [ggainey] poke ATIX folks # June 11 - CI duckup - Fun with self-signed certs/azure/s3/pulp-replication setups - should we drop back to http:? - should we not-validate TLS? - (note: **in CI**) - see https://github.com/pulp/pulpcore/pull/5466/files for an experiment - pulp-python ONLY accepts SSL-remotes - syncing between test-pulp-instances is affected - is this an EL9 container issue? - pulp-container inheriting host-certs isn't trusted (suddenly) - "something" changed externally to make the workaround no longer work - "requests" update? - where do the certs come from? - self-signed openssl calls - container-startup generates one - trustme library? - not the current issue - we may not have a current expert in this area (Fabricio?) - need to get this addressed one way or another before we can move forward on (any) releases - AI: can we organize a few people to really dig into solving this problem please? - ggainey, mdellweg - Do we need any other blocker on the Milestone for the next Breaking Change release (3.55)??? - https://github.com/pulp/pulpcore/milestone/10 - **breaking** changes - Next breaking change release? - last time we went 15 releases - that seems to have worked well for us? - consensus: 3.70 will be the next breaking-change release - [name=ipanova] Orphan cleanup is opened only to the admin. This might be inconvenient in the domain enabled deployment where owner of the domain cannot free-up his storage bucket - orphan-cleanup already is per-domain - BUT, only instance-admin can run it - should a "domain admin" be able to run orphan cleanup on "their" domain? - consensus: yes please - AI: [dalley] open an issue and start design discussion there - larger question around "should orphan cleanup just be a Thing pulp does "for you" instead of user-intervention - discussion to be had - [name=ipanova] Task-group-issued tasks should be limited in workers usage. Certain actions like core import, replica, file ACS rpm prune create a task group that can potentially dispatch hundreds of tasks. We should add an option/setting to cap the % usage of the workers. - limit concurrent-tasks-per-user would be useful - but not **this** problem - limit concurrent-tasks-per-domain - similarly - worker-categorization/tagging possibility (eg, "only workers 1 and 5 know how to sign things") - **also** not this problem, but is something we should be considering as we expand the multi-user exposure of a Pulp instance - "cheap"/"expensive" workers concept? - consensus: let's solve the immediate task-group-ad-hockery inside of task-group, and then refactor the ad-hoc users to take advantage of that - consensus: the "related issues" above **need to be solved** - but design/architecture/brainstorming needs to happen first - Do we want to backport this? - https://github.com/pulp/pulpcore/pull/5463 - "create api.,json without needing a running pulp instance" - drastically simplifies the CI - should help packaging teams as well - if we're willing to ignore the "hey! you're backporting a feature!" warnings, we can backport - mark as .misc? - maybe just hand-name it as .misc **in the backport branches** - consensus: yes please, figure out details at backport time # June 04 - Discussion about db indices - https://github.com/pulp/pulpcore/pull/5368 - We want to wait for Grant to discuss this again - https://github.com/pulp/pulpcore/pull/5370 - seems to have made the unblocked_at-problem MUCH better - API worker timesout waiting on an advisory lock to dispatch a task - https://github.com/pulp/pulpcore/issues/5390 - discussion ensues - Reviewed some blocking tasks for the 3.55 release # May 28 - Discussion about db indices - Waiting on Grant - Do we want to stop publishing bindings? - AI: Publish a Discouse topic about this - Generate the bindings is very unique to each usage - Different plugins, domains enabled or not. - All those things make the bindings unusabled between different configurations - Service pulp users are generating their own bindings with very specific plugins - Which users would be affected by not publishing the bindings anymore? - Kept it on the milestone unless it's the only thing blocking the new breaking release # May 21 - Use AWS STS for S3 credentials instead of saving them in Pulp - https://github.com/pulp/pulpcore/issues/5346 - Should it be handled by django-storages, if it's possible. - There's a scenario where the application retrive the token using boto, the retrieve of the token can be very specific - Sources, a Red Hat Service, interacts with AWS STS. - We need to pass through Sources, to aquire a session token. - Have an internal pulp plugin with a new storage class to deal with it. Few code as possible on upstream pulpcore. - Make domains extensible # May 14 - Kafka integration for the tasking system - https://github.com/pulp/pulpcore/issues/5337 - Kevin Howell is going to work on this feature - several/many pulp users have expressed interest in such a facility - Kafka is popular - is it the best option for us? - is it still amqp under the covers? (if so, we'd then work w/ any amqp broker) - payload standard is...exciting - websocket/webhook support? (for, e.g., pulp-cli?, galaxy_ng?) - works well in 1-client/1-server models, not so much in Pulp's multiple-actor-model - one kind-of architecture is, service notifies to kafka/amqp Thing, and there exists a listener/shim to implement webhook/websocket layer - django-channels discussion - whole app would need to be asgi instead of wsgi - future thoughts: - who is it for? - how configurable is/should this be? - is it like setting up a remote? - consensus: Hell No :) - or is it an admin-setup? - if admin - what settings belong to Pulp, vs Kafka admin/cfg? - server-url/cnx info - topic - what happens if you can't talk to msg-queue? - several options on how "fatal" not being able to send/queue msgs could be - Improve replication - https://github.com/pulp/pulpcore/issues/5214#issuecomment-2104590545 - key changes are: - replicate labels downstream - only replicate label-affected repos - do not clear-out repos not identified by labels - opens up window for collisions opn repo-names - creating a subset of repos without clearing out non-matching-repos in the downstream? - can this be flag-controlled? - needs discussion - collision needs to not-500, but to fail Obviously To The User - https://github.com/pulp/pulpcore/issues/4271 - lack of this is swamping upstream/replication process - easy win - needs someone to pick this up - what if we limit number-of-sync-tasks-at-once? - we have a pattern for this in import/export already - consensus: use this ticket to handle multi-sync-limit - Add ability to migrate S3 storage credentials - It's an important upstream feature, and we could use some input on testing it - https://github.com/pulp/pulpcore/pull/4298 - needs some not-gerrod, not-services-team testing, if at all possible - can this be testable in CI? - multi-bucket-testing requires commands to backend - current tests do file -> minio and back - that sounds good - clarification around default-domain is biggest thing keeping it in draft - what do we do when storage-settings-in-file no longer match storage-settings-in-db? - where do we check? - see SHA-SUM check as an example - issue discovered regarding dflt-storage not honoring env-var (see dkliban for details) # May 6, 2024 - ~~q filter still tech preview? https://github.com/pulp/pulpcore/issues/5319~~ - django-storages dependabot discussion - 1.14.3 adds some env-vars that domains should know about - decision: hold off on these updates till gerrod has a chance to add that support, then merge gerrod's PR and these updates simultaneously - AI: decko to revert the django-storages[google] update before releasing today - discussion around API_ROOT work - let's put an end-date on waiting for feedback? - ie, "We'd really like to get this released to the world, but want to make sure we aren't missing anything you need"? # April 30, 2024 - Using docker compose in our CI [dkliban] - podman-compose maybe? - not at feature-parity w/ docker-compose - problems due to old images/versions used in CI - whatever we do needs to be the same in the dev-env as is used in CI - currently ansible setup *does* do that? - maybe we can clean that up? - current setup is inherited from Travis - we don't have to do it That Way - should be able to massively simplify? - plugins still need to have control at pre-install-time? - AI: [dkliban] will file an epic and work w/ x9c4 to flesh out specific tasks. ggainey to learn, decko also. - core/3.53 releasing today - breaking change is coming, check deprecations please - ok to stop publishing changelogs on old docs now? - start publishing to new-place - redirect old-changelogs to new-location? - OR, note in old-changelogs pointing to new-location # April 23, 2024 - Sentry / GlitchTip integration - https://github.com/pulp/pulpcore/pull/5296 - (lots of) discussion ensues - consensus: - don't create specific pulp-setting - dkliban to investigate "make this useful in workers" - discussion RE releasing - currently problem building/releasing (old-style) docs - holding off on releases until we get green-CI (sez ggainey who is release-nanny this month :) ) - pedro/lmjachky investigating "removing possibly-unused dependency" that is causing our problem - consensus: - let's Just Do It - if something currently-unknown breaks the release-process, ggainey promises not to yell at anyone # April 16, 2024 - an API to use with the livenessProbe in kubernetes - https://github.com/pulp/pulpcore/issues/5243 - do it The Kubernetes Way - provide liveness-prob-api on content-app - database heartbeat? - cmd to run to check worker-liveness - api-liveness will also test db-liveness and connectivity? - let's not correlate these two answers - heartbeats all end up in the database - check that instead? - no, doesn't tell us "you can get to pulp from outside" - The Point here, is to let k8s decide "do I need to restart the pods" - and current approach (/status) sometimes takes Too Long - Conclusions: - YES, do a simple /healthz for api-app - YES, something similar for content-app - YES, **cmd** to check worker liveness - ADDITIONAL: worker that can't talk to DB "three" times in a row **should die** - updating timestamps seems to be taking minutes during a sync - "touch" bites us again - let's get an EXPLAIN/ANALYZE on the query - might be domain-index-related Thing? - can we get debug-tooling in staging env? # April 9, 2024 * [pedro] Discuss "Temporary files are an inefficient use of cloud storage" * issue: https://github.com/pulp/pulpcore/issues/3827 (added meeting key takes there) * board: https://miro.com/app/board/uXjVKYvswJg=/?share_link_id=853630394310 # April 2, 2024 * [pedro] Discuss OpenPG support draft PR (might interest COPR): * https://github.com/pulp/pulpcore/pull/5133 * summary: generally accepted, discuss further * Take a look at supported branches list * https://github.com/pulp/pulpcore/blob/main/template_config.yml#L84 * discussion ensues * the more branches we have, the more "in danger" the oci-image-creation is due to size limitations * discussion: we can't enforce this - but we *can* help/suggest/guide our downstreams to try and be mindful of the "hidden costs" of choosing a pulpcore * can we try dropping release branches? * we stop backporting dropped branches, which can cause all sorts of Fun * trying to do a lot of, say, CI updates, all at once, is...problematic * [decko] PR for ThirdPartyAuthenticationScheme * https://github.com/pulp/pulpcore/pull/5193 # March 26, 2024 * Remote Authentication, Pulp behind API Gateways and OpenAPI schema * OpenAPI advertises only Basic/Session auth * info put into OAPI via drf-spectacular * auth-backend being used in services is Different - looks for a third-party header * need to augment OAPI in a way to "know about" this * i.e., advertise the *gateway's* auth-url/workflow * https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.1.0.md#implicit-oauth2-sample * Fun With Pulp-CLI will then ensue :) * as long as the pulp admin sets this up following the OAPI Standard schema for a security-workflow, pulp-cli should be able to consume it * AI: [dkliban/decko] get an issue up for pulpcore-side work, docs * rubygems.org MFA * proposal: * all current users of the 'pulp' login at rubygems.org get a 'personal' rg.org account * ggainey, dkliban, bbouterse, dalley * all such accounts turn on MFA * THEN, all such accounts are marked as 'owners' of pulp's gems * THEN, pulp login turns on MFA * AI: ggainey to setup login, MFA, ownership and explore API access # March 19, 2024 * For Satellite, Bugzilla is being phased out in ~2 months, do we have any existing Jira integrations which we can adapt? * Otherwise: we need to create one * https://github.com/pulp/pulp-ci/actions/runs/8343502322 * we need someone to look into replication feature request [4637](https://github.com/pulp/pulpcore/issues/4637) * re-rooting content-app discussion * https://github.com/pulp/pulp-oci-images/issues/605 * https://github.com/pulp/pulpcore/pull/5011 fails w/out a fix for 605 * get 5100 backported/released for 3.49 * discussion RE [PR 4761](https://github.com/pulp/pulpcore/pull/4761) - [Stable task-resource identifiers](https://github.com/pulp/pulpcore/issues/4315) * will be a 3.55 breaking change * AI: mdellweg will review this week * discussion RE handling old-tasks and filtering * need to be able to convert between HREF to PRN * old-filters are still used in MANY PLACES * does this change the plugin API? -no, we're good * this either blocks the 3.55 release, *or* we carry the deprecated code until the breaking-chg-after-3.55 - fun! # March 12, 2024 * [lmjachky] migrating large amounts of data in pulp-container * artifacts store JSON data, we want to get rid of the artifacts and have the data stored in the Model instead * migration vs django-command (cost–benefit analysis) * considering downtime (pulp/Katello/Satellite) * transaction timeout?; migration can run once, if it fails, it fails * storage far-away can/will increase time * what if artifacts missing? * pulp_gem process * new-model * migration added new-model * mgt-command to populate new-model * old-model would continue being available until all-new-models available * what if: * migration adds field * code knows both old/new way * django-admin then migrates the data * "agreed solution": * 1 migration to add the field, raise warning to run the management command * add codepaths to accept both versions * management command to migrate the data and report on missing artifacts (can be rerun) * Later release remove extra codepaths and management command (force that the new field is not None in the migration or migrate the rest) and add the migration that forces the "migration" * Document zero downtime upgrade procedure * [dkliban] Replication feature request * https://github.com/pulp/pulpcore/issues/4637 * 4637 will be treated as a bug (because it's eminently backportable), dkliban to open an issue/RFE "handle making direct modifications to a Replicant, and then re-replicating" # March 5, 2024 * django-lifecycle update breaks us * [this](https://github.com/rsinger86/django-lifecycle/pull/151/files#diff-ccd0462c3332b1460bdeefa901260186d8b9de888e6acd16c6d63a0fbbbe2c27R85) change doesn't work if hashmap included * AI: mdellweg - teach dependabot to ignore 1.2+ changes (DONE) * AI: ggainey to open issue w/ django-lifecycle project * https://github.com/rsinger86/django-lifecycle/issues/155 * AI: all - if there are breaking-changes desired, we have 4 weeks to get them in * 3.55 is Coming!! * nice to haves RE release-process: * changes to requirements.txt being noted as "releaseable" * AI: [ggainey] to investigate updating check_release to look for these * what about test-infra? * don't need changelog entries (or at best .misc) * "I need a release of a fixture that my plugin needs to use" is the reason for a z-release with just-test-update * teach towncrier/release-script to auto-create a changelog entry for all req.txt changes since last Z * maybe keep this in the bnack of our heads as "something to automate better" when we have some free cycles # February 27, 2024 - chunked uploads of big file are slow because of default nginx client_body_max_size is 1mb - task unblocked_at is ready for reviews # February 20, 2024 * pulp_last_updated field * https://github.com/pulp/pulpcore/issues/5033 * https://github.com/pulp/pulpcore/pull/5036 * ipanova will summarize the outcome - expose everywhere just document * task created_by - there may be a timing window w/ task dispatch? * mdellweg to open an issue w/ discussion therein # February 13, 2024 * Github PulpOrganization/Core membership * help! * what about pypi/rubygems.org? * more coverage needed * we need to figure out how to handle 2FA for these sites * more volunteers needed * bitwarden locker? * need to fix the immediate need (not enough coverage) *now* * long-term, maybe discuss Better Ways * dalley, mdellweg, ggainey, dkliban volunteers for pypi access-keys * mdellweg, ggainey, dalley, bmbouter volunteers for RubyGems.org * ggainey wants to be added to the pulp org in github * we should audit permissions "regularly" (once/yr) * memory spike fix discussion https://github.com/pulp/pulpcore/pull/5049 * affects large/mature Pulp installations, sometimes quite badly * large change - more eyes please, once it's out of draft * "supported branches" and "lower bounds tests" * pulp_deb had, as a lower-bounds, a not-in-supported-branches pulpcore * examples: pytest<8 and black<24 and pulp-smash-removal issues * all LBs "should" match a supported-branch * Probably not solvable in a generic way. We'll fix this when it comes up. # February, 6, 2024 * Python 3.9 and CentOS 9 * katello wants py3.11 "now" * configurable py-vers and updated base-image - how do we make sure they don't conflict? * we already know how to tell branches to use a specific-image-tag 9eg, we have older branches that need py3.6) * centos9 doesn't "have" py3.8 * do we want to bump main-compat to 3.9+ ? * yes please * 3.8 is EOL "soon" (upstream) * downstream(s) are on 3.9 already * to do this: * images to centos9 (ie py3.9) * *then* configurable-py-version * older-supported-branches? * new image gets anew name * current branches continue to use existing name * plugin template needs to enable specifying image-by-name (it may already) * AFTER this all works, review supported branches and decide which ones need to "move up" py-vers-images * need images for diff py-vers (oci-images already discussing making this happen, and making it selectable) * https://github.com/pulp/pulp-oci-images/pull/575 is the in-progress work * currently stalled (s6 deprecated functionality) * AI: gerrod to take over PR, gerrod/dalley to sit down w/ mikedep to resolve current issue * dkliban is a resource for finding the right s6 folks to talk to * Repository deletion memory spike https://github.com/pulp/pulp_rpm/issues/2280#issuecomment-1928769513 * problem affects both up- and downstream * publishedartifact "seems to be" the current culprit * see possible approaches in links mentioned in 2280 * this prob needs to be addressed at pulpcore level * let's handle this at a generic/consistent level (see Hap's blog post) * poke Hap and ask if he wants author-credit if we expand on his examples * dalley to experiment in pulp_rpm, and move to pulpcore once we have something working # January 30, 2024 * pytest<8 * core, deb, rpm, certguard are addressed * core needs backports * AI: ggainey to sheperd those backports * every plugin needs to update their test-requirements * do we want to install into base-image? * pytest shouldn't be in production image * if installed in the image, we might not "notice" problems until *much* later * plugins have their own test-requirements * shows us (again) upperbounds-required protects us * should we have something in plugin_template to address this as part of ci-update-next? * AI: mdellweg to prepare such a thing * do we really need to have an upperbound on Everything? * lint/test reqs are safer for products to not-require-UB? * dependabot would yell at us when updates come up * does it? - yes * would allow us to *plan* accepting updates * yes please? * AI: [all] plugin leads need to open an issue and update their test-requirements to "reasonably pin" upperbounds * AI: [ggainey] add to discourse-discussion with an explanation for this * note on "why we don't use pip-compile" * Dependencies on release branches. * is there some way to evaluate dependencies for release-branches? * https://github.com/pyupio/pyup? * https://safetycli.com/ * https://pypi.org/project/safety/ * changing deps to allow "newer" *can* introduce issues * new deps *can* be incompatible with other entangled deps * if just relax UB, shouldn't be an issue * discussion around pip-compile and "pin exact versions always" * prob not a Good Idea for us * if we were using Pulp to curate our own content, we would maybe have not been affected by this? * better to break our CI than our upstream * "our" scale is significantly different than "pulp users" scales * Pulpcore certguard docs still not able to publish. * https://github.com/pulp/pulpcore/actions/runs/7641605467/job/21019289431 * still an ssh-key/permission problem * waiting for a run * AI: [dkliban] to try and fix manually * core/3.22.X to be released (jinja update) * backport releases are often done by not-the-release-sheperd * core docs conversion update * Pedro/Dennis have a draft PR for core-migration! * https://staging-docs.pulpproject.org/ for progress! # January 23, 2024 * orphan_for filter ready for review: https://github.com/pulp/pulpcore/pull/4668 * LTS branches * what is The Process? * can't support more than 5 +/- 3 branches at a time * OTOH - product-lifecycle suggests "support for 3 years" * can we add a "list of supported versions" to the spreadsheet/matrix? Can we make that happen with Spreadsheet Magic? * downstream/Pulp communication is a fertile ground for missed communications * stakeholder/pulp teams need to be Very Explicit about this * what if we choose "breaking + 2" as "always a supported release"? * any new feature request/migration issues would mitigate against this * what *can* we do/control? * no drawing to an inside straight, for stakeholders :) * we can be more explicit about "these are the branches we currently allow backports to" * lead-to-stakeholder communication * "look at template.cfg", and/or support matrix * each core-team-mtg, review currently-supported branches and decide if something has gone EOL * problem: stakeholder moving to a new branch wants no-regressions * if you backport to "A" supported branch, you need to backport to all supported branches in between * if stakeholder moves to "not latest but newer than current-lastest-supported", then they might be missing backports * AI: stakeholders need to stay in close comms w/ stakeholder leads RE supported versions * AI: stakeholders cannot move to "random" branches * AI: backports have to happen "in supported branch order" (No Regressions) * AI: support-matrix is Controlling - be aware of it! * AI: let's think about how we actually *retire* old branches? * always a challenge * AI: [mdellweg] open issue in plugin-template to separate "latest release" and "stakeholder supported" branch-lists for us * mdellweg to do next release * certguard merge blocking 3.45 - let's get it removed # January 9, 2024 * Today's release may have some Issues due to pypi API access changes * mdellweg continues as release-nanny for now * Do we want to bump minimum Python version to 3.9? * oci-images (PR draft?), plugin-template prob need changes * PR bumping oci-images to EL9 and Python 3.9 https://github.com/pulp/pulp-oci-images/pull/554 * what does 3.9 get us? * still needs monkeypatch to enable TLS-proxy support * need 3.11 to not-do-that * bumping in oci-images means we no longer test on 3.8 * can/should we get signoff from stakeholder leads? * RHUI/Satellite on same rpms (3.9 for current-support) * what if oci-images had multiple python-version images, and branches used "oldest supported"? * what's the proposed "long term" python-version to be supported? * maybe whatever is in F40? * 3.12 on F39 * maybe pick 3.11? * conclusion: we're not ready to do this yet * bumping aiohttp to 3.9.0+ and python-cryptography to 41.0.6+ * stakeholders want bugfixes * need to think about which branches need them * need to make sure branch-upper-bound *allows* fixed version(s) to be installed (main branch already ok) * mdellweg to pick this up * Orphan cleanup: https://github.com/pulp/pulpcore/pull/4841 * Can't find a way to be compatible with orphan_for filter: https://github.com/pulp/pulpcore/pull/4668 * maybe close 4841 in pref for 4668, and then change tests in https://github.com/pulp/pulp_rpm/pull/3313 to take advantage * conclusion: yes please * Note: if pulp stopped allowing content outside of repositories, the need for this goes away * implications on uploads (mostly) * impacts "what is the change-diff" (can address in other ways) * see also container's "pending blobs" solution * last-touched and sync implications * PyPI credentials update ___ # Archive * [2023 meeting notes](https://hackmd.io/@pulp/core_2023) * [2022 meeting notes](https://hackmd.io/@pulp/core_2022) * [2021 meeting notes](https://hackmd.io/@pulp/core_2021) * [2020 meeting notes](https://hackmd.io/@pulp/core_2020) ###### tags: `pulpcore`

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.