RPM meeting - HackMD

# RPM meeting  ## Pending action items: * AI: [ggainey] stand up an rpm/deb/ansible install and experiment, to get hard details on the actual behavior (CopySerializer issue) * discuss adopting zero-downtime-migration strategy * https://github.com/pulp/pulpcore/blob/main/docs/plugin_dev/plugin-writer/concepts/index.rst#zero-downtime-upgrades * last copr issue https://github.com/pulp/pulp_rpm/issues/2271 ## Agenda template ``` ### Month DD, 2023 Discussion: Open, non-Draft PRs: * https://github.com/pulp/pulp_rpm/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ``` ## 2024 ## Upcoming ### July 18, 2024 * CI failures update * nightly - performance tests failing * fixed by a PR that Pedro merged to the template * https://github.com/pulp/plugin_template/pull/890 * ci update job failing * https://github.com/pulp/plugin_template/issues/895 * 3.18 - build is failing * why are we trying to build the wheels here? * https://github.com/pulp/pulp_rpm/actions/runs/9942408703/job/27538226968#step:7:265 ### June 27, 2024 * discussion RE nightly-failure * AI: [ggainey] Raise pulp_rpm LB to core/3.49 in main * https://github.com/pulp/pulp-cli/issues/994 * discussion ensues * ggainey to continue reporting back to team as progress is made * ### June 20, 2024 * core/3.55 support needs to be released * https://github.com/pulp/pulp_rpm/pull/3621 * lots of deprecated test-fixtures went away * some discovered-issues with pulp-cli openapi (see below) * discussion: can we do this as a Z-release or should/must we do a new-Y for core/3.55? * Z, please * AI: [ggainey] to Think Hard for a few minutes * prune and advanced-copy now covered in pulp-cli * JSONField has a bug in both django-spectacular and cli/openapi validation * https://github.com/pulp/pulp-cli/pull/995 * fixed and released in 0.25 * means we can lose more httpie from rpm docs * AI: [ggainey] https://github.com/pulp/pulp_rpm/issues/3622 * copy-serializer in REST API is currently colliding with copies in ansible and deb * https://github.com/pulp/pulpcore/issues/5494 * AI: [ggainey] stand up an rpm/deb/ansible install and experiment, to get hard details on the actual behavior * Kickstart tree spec vs. reality * https://github.com/pulp/pulp_rpm/issues/3594#issuecomment-2179672007 * everyone is sad * would it be a good idea to have a Pulp/Anaconda meetup? * there are other issues than this one where Pulp and the spec 'disagree', but nobody's every noticed? * if someone (/me waves at pbrochado) is interested in fixing the fixture and seeing if it breaks us - go4it! * https://github.com/pulp/pulp_rpm/pull/3525 (get rid of is_modular) * stuck in Draft - katello relies on this field (as imperfect as it is) * need to discuss w/ iballou on how to make that better? * https://github.com/pulp/pulp_rpm/issues/3452 (remote-repo with **only** .treeinfo) * discussion ensues * see PR for some Pain :'( * https://github.com/pulp/pulp_rpm/pull/3523 (Test Jira PR) * can we close? ### June 13, 2024 Discussion: - "out of order metadata" error - discussion in Slack #pulp w/ jfont - Release a pulpcore<3.70 version "now"? - do this in rpm/3.26.1? - cut 3.27? - consensus: let's update requirements to core<3.70 and release rpm/3.27 Fri/Monday [dalley to pick up] - pulp-cli support for prune_packages and copy coming soon - would be nice to have rpm/3.27 released - ggainey: finish pulp rpm copy impl and then update rpm-docs to no longer need httpie - note RE adv-copy: operation-id is "copy_content", instead of something like "rpm_copy_copy_content" - not currently "a problem", but definitely NOT STANDARD - `pulp debug openapi operation-ids` - Jira tickets out of sync (just be aware, check whether BZ was closed) - 50 open BZs, 100 open Jiras - some that were closed/BZ are NEW/Jira - dalley trying to unsnarl - kickstart metadata issue: - https://github.com/pulp/pulp_rpm/issues/2292#issuecomment-2163270958 - discussion ensues, comment added to issue Open, non-Draft PRs: * https://github.com/pulp/pulp_rpm/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse * CI failures * azure-fix not applied yet - close and re-run after next CI update * lowerbounds failing due to libcomps-fix not backported far enough * dalley to look into fixing Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### June 07, 2024 Discussion: * Fun with Hao and advisory.resolve_conflicts(): * https://redhat-internal.slack.com/archives/C031713779N/p1717674742538369 * Grant has a Sad * This broke (supposedly) when Katello refactored their Pulp-indexing code * Hao has a workaround for Katello * Pulp should, in theory, be providing unique Errata, but this apparently isn't the case * special case: see thread * Pulp should, probably, try to stop merging Errata and do away with as many special cases as possible * and hopefully not rediscover all the reasons why it was done this way originally * other possibility: restructure Errata merge to be dramatically more simple * each errata being independent documents, modeled more simply, theoretically the logic could be simpler * (grant disagrees w/ simpler) * *can* we not-merge? even tho createrepo_c allows dup-advisories - is that *really* the answer? Discussion to be had * [prune](https://github.com/pulp/pulp_rpm/pull/3536) discussion * ggainey modified to not-expose concurrency to user * show as second commit to improve review-ability * if acceptable - squash and accept and merge Open, non-Draft PRs: * https://github.com/pulp/pulp_rpm/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### May 16, 2024 - Don't merge the pruning PR before the release - Grant on PTO for 2 weeks - https://github.com/pulp/pulp_rpm/pull/3425, merge it or not? - merge it, tech preview label - yes there's limitations, but we can document them - needs some doc-love before undrafting - https://github.com/pulp/pulp_rpm/issues/3510 - on the context of the Modulemd upload API, the modulemd linking to their packages should happen on RepoVersion creation time (not upload time). - open questions: - snippet upload: string or file? - ask whether microsoft needs or is using module upload functionality: done ### May 9, 2024 - about build workflow: - couldn't repositories be used to hold builds? this would solve same-nevra-in-repo and give better control over packages belonging to a specific build. - the short answer is "yes" - the long answer is, it would nice if the end-user doesn't have to maintain a lot of workflow to accomplish the goal COPR has (and other build-systems have) - need to be informed by the "Pending" concept pulp-container already implemented - since that's partway into this space already - More Brainstorming Needed - Let's *not* write Pulp4 - let's think of what can be done with the current Pulp3 architecture/entities - modulemd creation API: - issue: https://github.com/pulp/pulp_rpm/issues/3510 - the "packages" api param (the list of hrefs) should be a derived attribute from snippet, because we use the snippets for creating the yaml on the publishing task. - https://github.com/pulp/pulp_rpm/blob/a1fae15654e5958c184bcdb0b3266a629ea2afd0/pulp_rpm/app/tasks/publishing.py#L587-L607 - all the "required" fields are also defined by the snippet, and the snippet should "win" - agreed? yes - when do we decide to tell the user "your module-snippet lists pkg-NEVRAs that are not in this repository, and so it is **broken**" - at modulmd-create, or at publication-time? - could use more thinking - maybe fail-at-upload, accompanied by docs saying "you have to create/upload all Packages **first**, before adding a modulemd that requires them" is the better choice for "when to fail" - Modulemd.packages are required also in modify API (copy, remove) Random observation: The concept of "a build" of many RPMs is basically a generalization of a module - a grouping of packages with attached metadata, handled as a single unit. ### May 7, 2024 * AI: [ggainey] before next RPM mtg, archive prev years' minutes into separate docs * [2023 minutes](https://hackmd.io/@pulp/rpm_meeting_2023) * [2022 minutes](https://hackmd.io/@pulp/rpm_meeting_2022) * [2021 minutes](https://hackmd.io/@pulp/rpm_meeting_2021) Discussion: - https://github.com/pulp/pulp_rpm/issues/2271 - collision between "delivering content to clients" and "slowly creating a deliverable repository" workflow - discussion about diff between rpm-build-time and pulp.repositorycontent.pulp_created - new_version.finalize_version() already de-dups same-NEVRA-diff-csum by choosing latest-added(?) - discussion: can we actually meet this requirement? - discussion: how can Pulp meet the needs of "users who want draft/partial repo creation" workflows? - use-cases for a build-system are qualitatively different then use-case of content-delivery - https://blogs.gnome.org/alexl/2019/03/19/introducing-flat-manager/ - use-cases might "just" need refinement on what "publish" means - ie diff "kinds of" publications? - or "publish" gets smarter? - it already is, in many cases - NEVRA+location_href is a current constraint - multiple RPMs/same-spec-file can't be treated independently - must operate on "entire build", not at package level - so management of that has to understand builds - which Pulp doesn't - how do we proceed from here? How can we make COPR successful? - notes on pulp_container: - "Pending" entities are A Thing - need to have an atomic operation at push/commit-time of blobs/manifest/tags/etc - can "Pending" be a...Thing? associated with Repository? - If COPR uses Pulp "as a service", won't be able to rely on COPR admins having access to pulpcore-manager commands Summary: - Pulp doesn't track builds together, if you "remove a build" COPR will need to track what packages are involved - If COPR is doing that, then adding them back isn't such a problem, as per [issue 2271](https://github.com/pulp/pulp_rpm/issues/2271#issuecomment-1416034867) - Pulp is a repository management solution not a bucket of builds - you can't just put multiple duplicate NEVRA packages in the same repo as a holding area - Cleanup - there is a mismatch between the way orphan cleanup works and the way COPR needs to be able to keep historical builds alive for some period of time. Where do you keep those builds if not in a repository? And how do you keep builds together (again)? - "Pending" a la container plugin? ### May 3, 2024 Discussion: * https://discourse.pulpproject.org/t/workflow-with-rpm-modules-and-immutability/1209 * AI: Daniel needs to file an issue about removing `is_modular` like he said he would * last copr issue https://github.com/pulp/pulp_rpm/issues/2271 Open, non-Draft PRs: * https://github.com/pulp/pulp_rpm/pulls?q=is%3Apr+is%3Aopen+draft%3Afalse Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### April 18, 2024 Discussion: * fun with redhat-uep CA * discussion about pulp_rpm and tutorials * discussion about signing options * pass pubkey at upload? * would be nicer if the destination-repo just "knows its own pubkey)" * add pubkey as an attribute of Repository? * this seems to be a good . Pulp-ansible already has such field https://github.com/pulp/pulp_ansible/blob/main/pulp_ansible/app/models.py#L519 * str-attr to match other cert-attributes (e.g. see Remotes) * have PubKey be its own Model? * is this really useful, if we're not tracking signatures? - prob not * Track Artifact signature? * consensus: let's not - seems a lot of work for not a lot of end-user usefulness Open PRs: * https://github.com/pulp/pulp_rpm/pulls Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### April 12 2024 Discussion: * What are the primary benefits to COPR of using Pulp? * staging versions of repos * handling payload in S3 * can help get COPR away from "ridiculous disk array" monster * 40TB (!!) of data(ish) * expect 50+TB in 2 yrs * not considering containers * decouple from worrying about storage * Do "super-old repos" ever get deleted? * yes, but the workflow to the user is Complicated * currently, outdated rawhide packages "stay forever" * deletes, when they happen, happen in batches * poss 1000s at once, when it happens * COPR currently has priority queues * Pulp/COPR will consider details of workflow to make sure the workflow works for them * What is COPR general timeline and plan for the migration? * COPR finishing up research/staging tasks * incremental approach is The Plan * switch some dev-projects to pulp while "prod" continues Current Process * https://pavel.raiskup.cz/blog/copr-and-pulp-content-hosting.html * want to have Pulp in production this calendar year with "some" projects moved/taking advantage of it * poss move All Data To Pulp next calendar year? * Running pulp in Fedora infra * Q4? * maybe Fedora COPR could be "one of" the users? * *very* interesting to COPR team - removes a lot of maintenance burden * worried about installation/moving to the container-world * mostly because "we dont' do it that way now" * lack of pulp_installer support may be an issue * lots of experience debugging performance issues in the traditional unix world, might be a challenge adapting to openshift #### COPR and signing General: - Is signing part of the Pulp features that bring the most benefit for COPR? - not at the moment - can stay "for a while" w/ current mechanism - COPR's current plan is to upload-signed-RPMs - need key-management workflow before moving to letting pulp do the signing - current mechanism works - obs-sign offloads heavy work out from the client-tool - COPR has been following the PR discussion - Its ok if post-upload signing is slow - Resigning is a disaster case, it's not done frequently - Packages are always signed in the COPR workflow. We shouldn't have unsigned packages in pulp Implementation specific: - Signing on upload: - Implementation detail: when using workers to process a tmpfile, we need to send it to a shared permanent storage (temporarily) so any worker can access it. - The extra 2 roundtrips are not paid in the same region (brian) - The extra 2 roundtrips are each better than O(n) in time, as S3 transfers can be optimized w/ parallelization (improves workers availability) - https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html#boto3.s3.transfer.TransferConfig - Post-Upload signing or re-signing: - How COPR wants to handle managing keyring-repo-content relations? - a) Pulp should know what key to use to sign a package that belongs to repo. It should know if a package is signed and its key expiration date. - b) Pulp doesn't need to know these, and COPR will be able to perform the appropriate calls using their own info. - Implementation detail: when (re-)signing a package inside Pulp, we can't immediately replace it with the signed one. The deletion of the old is done through an orphan clanup process, which involved unlinking the old package from immutable objects. - Orphan cleanup scheduling should adjust to the rate of signing packages. #### COPR and RPMs * can work w/ theforeman RPMs currently Open PRs: * https://github.com/pulp/pulp_rpm/pulls Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### March 28 2024 * discussion around [3485](https://github.com/pulp/pulp_rpm/pull/3485) * backport to latest (3.25) * drafting that backport pending some further testing * discussion around issue [3482](https://github.com/pulp/pulp_rpm/issues/3482) * would like to prioritize this doc section (ggainey) * discussion around [3425](https://github.com/pulp/pulp_rpm/pull/3425) * waiting for feedback/discussion from mdellweg, copr folk * fun with cats :) ### March 14 2024 * discuss RPM builds for COPR * https://github.com/pulp/pulpcore/issues/5128 * Katello only builds/uses/supports certain versions long-term, what do you want your upgrade cadence to look like? * Are the [current for-theforeman RPMs](http://yum.theforeman.org/pulpcore/) acceptable? * Do they have to be published directly in Fedora? * Would we need to rebuild for a different buildroot, as these packages are currently built against enterprise linux * How should migrations be handled? * Katello/Satellite/RHUI have external tools/scripts that manage upgrades, but standalone RPMs in a repo won't have this * Does it happen automatically? Does the user need to trigger it? * discussion RE https://github.com/pulp/pulp_rpm/issues/2986#issuecomment-1995335073 * slower-but-correct is better than poss-faster-but-less-reliable * leans us towards "sign in worker" instead of in-API-worker ### February 16 2024 ### February 8, 2024 * signing discussion * would be very helpful to have signing-on-upload * extend later for signing/re-signing-existing-content * keyring/priv-key-per-repo * add to upload file/repo/signing-service * discussion: repo defines signing-service * discussion: have a boolean, at upload, to ask for uploads to be signed * discussion: for existing content, let's add a new api for asking for existing-rpm to be signed * discussion: should repo have an "always sign on upload", to control/override the sign-on-upload parameter? - yes * discussion: impact on the modify/copy APIs * would it happen atomically as part-of-the-copy? * or require user to call sign-everything seperately * implications of object-storage needs more thought * this is all a phase-later Thing * discussion: does Package need to know its signing-key-id? so we can know if a given package is signed with a specified key? * is this a one-to-many Package-to-signature relationship? * needs further discussion - but, leaning towards "keep sig info a separate Thing from Package" * signature contains timestamp - but that can be specified, in order to support reproduceable builds * discussion: signing changes the csum/pkgid * multiple signatures allowed? * https://ftp.osuosl.org/pub/rpm/max-rpm/s1-rpm-pgp-signing-packages.html * https://github.com/rpm-software-management/rpm/issues/189 * discussion on https://github.com/pulp/pulp_rpm/pull/3426 * https://github.com/pulp/pulp_rpm/issues/2678#issuecomment-1714738077 * make sure sync doesn't fail when upstream repo is...suboptimal * discussion RE COPR implications? * https://github.com/pulp/pulp_rpm/issues/2271 ### January 25, 2024 Discussion: * backport labels * we need 3.23 and "latest release" * AI: dalley to update labels and ci-supported branches to 3.18, 3.19, 3.23, and 3.25 * what is going on with https://github.com/pulp/pulp_rpm/issues/3286 ?? * Suggestion: have stakeholder(s) open an issue when they want a new branch to be "supported" * turn on automerge? * yes! done * more discussion on [2253381](https://bugzilla.redhat.com/show_bug.cgi?id=2253381) Open PRs: * https://github.com/pulp/pulp_rpm/pulls Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ### Jan 18, 2024 Discussion: * Things Being Worked On * ggainey: https://bugzilla.redhat.com/show_bug.cgi?id=2253381 (import/advisory-merge UQ) * (dalley) https://github.com/pulp/pulp_rpm/issues/3368 - migration issue * COPR issues : https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3Acopr+ * (ggainey) https://github.com/pulp/pulp_rpm/issues/2909 (purge by datetime) * maybe leave obs-signing for last (since it's marked "nice to have") * discussion w/ pbrochado on the user/katello/pulp workflow involving filters Open PRs: * https://github.com/pulp/pulp_rpm/pulls Un-triaged bugs: * https://github.com/pulp/pulp_rpm/issues?q=is%3Aissue+is%3Aopen+label%3ATriage-needed CI status check * https://github.com/pulp/pulp_rpm/actions/workflows/nightly.yml?query=workflow%3A%22Rpm+Nightly+CI%2FCD%22 ## [2023 minutes](https://hackmd.io/@pulp/rpm_meeting_2023) ## [2022 minutes](https://hackmd.io/@pulp/rpm_meeting_2022) ## [2021 minutes](https://hackmd.io/@pulp/rpm_meeting_2021) ###### tags: `RPM`, `Minutes`

Read more

Pulpcore team meeting

CLI Team

Deployers Team Meeting

Open Floor Agenda