owned this note changed 6 days ago
Published Linked with GitHub

Pulpcore team meeting

Overview

Meeting Agenda

  • Pending-AI review
  • Upcoming items
  • Open PR review
  • Collect Action Items, copy to Pending-AI

Pending AI

  • downstream-team: get together and pair-program on capsule sync connection issue
    • discussion happened, 3.70 issues intervened
  • AI: bmbouter wants to discuss COPR Cloudfront setup w/ Somebody On COPR, will report back
  • AI: [ggainey] to coord w/ ATIX on core/3.70 work
    • 1232
    • no current progress - "might start this week"
  • AI: ggainey to revisit postgres plans w/ katello
    • added to agenda for 19-FEB Katello Integration mtg
  • AI: ggainey to work w/ Tanya on some high-level plans for pulpcore before next mtg

Upcoming

Feb 18, 2025

  • Enhance Replicate delete policy
    • https://github.com/pulp/pulpcore/pull/6247
    • PR allows user to customize what/how objects get deleted on replication
    • protect "unlabeled" entities in case of a naming conflict?
    • def need some explanation in docs about "expected behavior"
      • prob for "how replicate makes decisions"
    • rename to just "policy" - since it covers more than just "delete"
  • Fun With AWS
  • What do we need to do to move away from Postgres-13?
    • downstream has current-versions that use it
      • RHEL9 base
    • pgres-13 is EOL in November
    • we stay on 13 to make sure we don't start relying on newer-version-features for bugfixes we want to backport
    • to change, need to write code in the images to handle the postgres-migration needed to move from 13 to "whatever we decide to move to"
    • do we support both?
      • please no
    • what are downstream's plans to move to newer versions?
      • Satellite
      • Automation Hub
      • RHUI
    • AI: ggainey to revisit plans w/ katello
  • new plugin from Evgeni
  • ttereshc needs some help on where to file some high-level Jira requests for pulpcore
    • AI: ggainey to work w/ Tanya

Feb 11, 2025

  • Discussion: With Pulp4 remove CONTENT_ORIGIN (maybe?)

    • https://github.com/pulp/pulpcore/issues/6239
    • there are standards for "building URLs" - we should prob follow/use them
    • CONTENT_ORIGIN is designed to be the content URL, as separate from the API URL
      • removal would prevent split-fqdn deployments
      • AI: [mdellweg] add some discussion to issue on this
    • can we take advantage of at least some of this in Pulp3?
      • discussion ensues
      • all of django lives on request/response - some of Pulp, does not
  • CloudFront Support

    • https://github.com/pulp/pulpcore/issues/6258
    • there is a formal request for this support
    • geo-located support would be Really Nice for non-US-East deployments
    • dalley: COPR already takes advantage of this
      • Pulp-origin, backed by S3 (or something), and CloudFront cache "in front of" Pulp
      • cache updates from Pulp
      • clients "talk to" cache-layer
    • botocore's CloudFrontSigner
    • Services needs/wants to be able to set up storage/redirect/cloudfront info per domain
    • "timebombed" URLs need to work correctly in this instance as well
    • django-storage support already exists maybe?
    • serializer needs to be updated to "understand" these new fields when present
    • medium-level urgency
    • testing: How?
      • do we need cloudfront actually involved?
      • credentials etc?
      • unit-test the serializer - we shouldn't be in the business of testing django-storages
      • AI: [mdellweg] let's get this discussion into the issue
    • Services willing to take on building this, given the unit-test-only testing decision
    • Consideration: how hard is it to add Cloudfront support to an existing S3?
      • under discussion
    • AI: bmbouter wants to discuss COPR setup w/ Somebody On COPR, will report back
  • where are we on stabilizing plugins for core/3.70?

    • upgraded the bindings generator for 3.70, which added pydantic to the models
    • compatibility doc
    • gem, python, maven, ostree done/released
    • ansible needs work
      • AI: [ggainey] will take a stab at this this week
    • deb needs work
      • There's a PR to bump version, but no other work
      • AI: [ggainey] to coord w/ ATIX on this work
    • rpm work still in-progress (Pedro et al)
    • container in progress (Gerrod)
      • why getting rid of pulp-smash?
        • it breaks the client bindings
    • npm in-progress
      • plugin is out-of-date and needs work to get to supportable state
    • documentation for our community needs to be prioritized
    • what were the effects on the Ruby bindings of this update?
      • mdellweg talked to iballou/evgeni about self-gen bindings
      • downstream has/will-have some of the same issues w/ bindings we do, over the course of maintenance
    • we prob should de-prioritize "pushing the bindings" for talking to Pulp3
      • use pulp-glue (python only)
      • talk requests() directly
      • something "like" glue for your own language
  • Pulp4 timelines

    • stakeholders are Very Interested
    • "not before Q3" is our current messaging
    • maybe align w/ 3.85? (unlikely)
    • milestone-based approach
      • all plugins to domains
        • deb, container, npm, maven need to be done
      • domain-enabled oci-images
      • Pulp4 Milestone in GH w/ required activities
      • don't forget that /v4/ APIs can/will/must show up in Pulp3!

Feb 4, 2025

  • Core 3.70 is out!
    • All blockers (e.g. multiple FQDN support / CONTENT_ORIGIN) closed
    • Plugin compatibility?
      • None yet, ansible and RPM need to merge fixes, other plugins just need to bump the minimum bound (AFAWK)
  • Services: update on investigation into Task retry on worker exist RFE
    • Patched worker shutdown to be really long (hour) - that resolves our needs
    • Issue closed

January 28 2025

  • Can we upgrade the Postgres version for Pulp 4? (Also the Python version?)
    • gerrod: upstream comment on Postgres13 (single container, compose)
    • dalley: pgres17 is most recent
    • bmbouter: drivers
      • broad compat (esp for downstream support)
      • newer pgres can have significant new features
      • ship vs test-against? (min vs single-container)
    • dalley: downstream ships/uses pgres13
      • let's ask downstream about upgrade plans?
    • bmbouter: let's separate this from Pulp4
    • bmbouter: can we build pgres13/pgres17 images? and test both?
      • gerrod: more images is Not Great, given our constraints
    • dalley: services?
      • AWS pgres16 + extensions
    • bmbouter: prob should stay w/ what downstream tests/ships against
    • AI: ggainey to check w/ Sat on upgrade plans and bring back to this mtg
      • from ehelms: "RHEL 9 currently has PG 13 as it's full-lifecycle version of postgres, the other versions require modularity which we are trying to avoid. We would like to handle any PG updates once we get to a containerized version to avoid modularity."
    • gerrod: Next: Python version
      • we test against 3.9, which is EOL "soon"
        • which causes our third-party-libs to raise their LBs
      • downstreams are on 3.11 (we think)
      • when can we do this? how "breaking" is this?
        • plugins need to line up "acceptable" versions
        • install-from-source requires you to upgrade your python
      • what if we plan for core/3.85?
      • do we want to hold 3.70?
        • ggainey: late in the cycle, katello would like to branch "soon"
    • bmbouter: core/3.85 milestone including an "upgrade python to 3.11" blocker
    • consensus: we want both of these, "soon"(ish) - but not blocking on 3.70, can be done before Pulp4
  • Task retry on worker exist RFE
    • services running into some issues
    • discussion in the issue, please
    • services to experiment, will come back to us later
  • Discuss a plan for capsule sync connection issue
    • Downstream really wants some answer here
    • dalley: "somewhere in aiohttp" Things Go Awry
      • very difficult to reproduce
        • requires a very specific kind-of network failure
      • needs more eyes/rubber-ducks on this
    • consensus: downstream-team should get together on this
    • ttereshc: ask for support/QE help on reproducing?
  • labels for domains RFE
    • would help a services use-case
    • there is a current workaround
    • would follow the model set by Repository/Remote/Distribution labelling
    • ggainey: prob straightforward?
  • CONTENT_ORIGIN discussion
    • discussion and results in the PR
    • HTTP_X_FORWARDED_HOST doesn't have what we expect/need (alas)
    • HTTP_X_RH_EDGE_HOST does? (but is nonstandard)
    • add new setting to tell pulp which header to look for?
    • pulp_rpm - can we get access to global-request-object from content_handler()?
    • even if we take this route - the header that we would like to use, is NOT standard, and we'll have to adapt to it not existing regardless
    • we really need to make a decision here (one way or another)
    • "most" users will not be affected
      • for those who are - some degraded/missing functionality per plugin?
      • OR, we can re-enable w/ some new settings (as noted above) - but still possibly unreliable
      • NOTE : not all plugins are adversely affected
        • pulp_rpm/config.repo is affected
        • pulp_ansible/preauth-URLs is affected
          • is there a workaround possible?
    • consensus: this kind of sucks, but for 3.70, pursue "CONTENT_ORIGIN is null", possible solutions rely on non-standard headers
      • AI: ggainey - let's get this on the PR

January 21 2025

  • go/nogo for everything on 3.70 blocker list
    • purge migrations - accept or no?
      • are plugins ready?
      • are we willing to Just Break Everything?
      • consensus: let's remove from blocker list - it's not Required
    • content-origin - even if it merges today, do we want to release immediately?
  • Propose a new policy for adding dependencies:
    • We can only pick up a dependency if it's packaged (and not deprecated) in fedora.
      • outsource the decision about good and bad dependencies.
      • outsource the work to "find the better alternative"
      • relieve the pressure on the build team (assuming they never need to repackage those dependencies)
      • discussion ensues
        • reducing build-team-work is a Good Idea
        • requiring only-fedora can be limiting
        • Being Intentional about dependencies is a Really Good Idea
        • transitive dependencies are Painful
        • can/should we review current dependencies? do we need them all?
        • discussion about orphaned-content-process
        • a strict policy may not be a realistic expectation
      • consensus: "be reluctant to add dependencies, but if you do, something already-packaged makes that call easier"
      • discussion: optional dependencies
        • we don't have mechanism in place to warn users about missing "optional" dependencies
        • however - still a good idea to make things optional when that makes sense (e.g., otel deps)
        • see kafka-decision-process
        • would require docs-work/startup-checks to explain, to the user, what they need to do to get access to an "optional" feature
  • Instead collaborating on one feature branch should be handled by special permissions on the (personal) fork (limited to that branch if possible) not the main repository.
    • services has its own set of timing/requirements, and handles PRs differently as a result
    • Do we want/need a generally-accepted "team norm" for not-services?
    • How often does this come up?
    • transferring work from one person to another is one thing, multiple people working together is a different workflow
    • consensus:
      • services' workflow works, and we're not touching it
      • outside of services - this comes up infrequently enough that we'll deal with it if/when it happens
      • Overarching Goal: Keep all the work/discussion for a given bug/feature in one PR if at all possible
        • if not - start new PR w/ summary and pointer to previous/original
  • content-origin discussion
    • COPR needs it from us (among other things)
    • consensus : def "needs to be in 3.70"
    • how do we deal with plugins that need the full URL, not the relative path?
    • discussion
      • are API/Content on the same host?
      • are there multiple gateways by which one can access?
      • if "deployed seperately" AND "multiple gateways" - then Things Will Break
      • do we wait until we know that plugins aren't Hopelessly Broken by this?
      • it "probably" have been a Better Idea to do everything using relative-URLs in the first place
      • discussion around pulp-rpm's config.repo:baseurl problem
      • discussion around gateways/cert-auth/VPN-only-access
        • some gateways need certs, some need tokens
      • discussion: should we return null for dist-base-url if content-origin not set
        • don't set it to an empty string
      • redirect-call is involved in get_artifact
    • decision
      • each plugin needs to do "whatever the plugin decides is Good"
        • replication: added a new field
        • rpm: can't have CO-null and as to auto-gen config.repo
        • python: don't rely on the default if CO-null
        • ansible: ummmmmm
      • let's get this PR accepted and merged and 3.70 released
      • plugins will adapt as they need
        • plugins can fail-immediately if they decide "I can't work in this environment"
      • [NOT DISCUSSED YET] https://github.com/pulp/pulpcore/blob/main/pulpcore/app/util.py#L499 this function should probably add a relative_path_ok=False parameter to catch all the unknown assumtions in plugins using it.

January 14 2025

  • Random weekly cleanup question:
    • Can we delete pulp-smash from pypi?
    • was needed out there for "a while", but we do versionless/source-install now
    • archive all the versions? or delete?
    • we still use pulp-smash in "a few" places
      • pulp_rpm, pulp_ansible
    • delete or archive?
      • consensus: delete
        • upload empty package into the namespace after?
      • AI: ggainey to delete
    • AI: issues in pulp_rpm/pulp_ansible to finish the pulp-smash removal
  • Should this be archived?
    • https://github.com/pulp/pulpproject.org
    • discussion ensues
      • pmbrochado: had/has plans to use it for common docs
        • pulp-docs was/is intended to be "just a tool", not the content
      • AI: archive, it can always be undone if we want to
  • introductions :)
  • go/nogo for everything on 3.70 blocker list
    • still waiting on 2
    • consensus: wait on these to release 3.70
  • Propose a new policy for adding dependencies:
    • We can only pick up a dependency if it's packaged (and not deprecated) in fedora.
      • outsource the decision about good and bad dependencies.
      • outsource the work to "find the better alternative"
      • relieve the pressure on the build team (assuming they never need to repackage those dependencies)
  • Think about format for this mtg/release-mtg-lead rotations for next time
    • If you have Strong Opinions - bring to mtg!
    • dkliban: finds this mtg/format Very Useful
    • dalley: this is the only (mostly) team-wide technical mtg we have
    • dkliban: this is The Place to do teamwide brainstorming
    • consensus: this is def a well-spent hour!
    • mtg-lead/release-lead changes?
      • consensus: no changes needed
  • adding hyagi to core team
    • NOTE: done, but insufficient - also made him an "Owner" of the org (last person who wasn't that)
    • we had to close PR unecessarily because hyagi coudn't push to dkliban's PR
    • ggainey volunteers
    • how abouteverybody?
    • this is actually separate from the "taking over someone else's branch"
    • This is the wrong reason to do the right thing.
      • Instead collaborating on one feature branch should be handled by special permissions on the (personal) fork (limited to that branch if possible) not the main repository.
      • services has its own set of timing/requirements, and handles PRs differently as a result
  • ggainey AI: archive 2024 notes

January 7 2025

  • new RFE for services https://github.com/pulp/pulpcore/issues/6154

    • we did it in services here
      • already tested/deployed and in use
    • how should we proceed with an upstream contribution?
    • Is there consensus on "just do it"
      • ggainey/'mdellweg' will be assigned to review
    • How will this work on domain-disabled systems?
      • Does this endpoint even exist on domain-disabled?
    • Let's have dev-discussion on the PR when opened, not here
  • multi-gateway PR is ready see this PR

    • audit usage of the CONTENT_ORIGIN like this
    • REMINDER: this is a 3.70 plugin-breaking change
      • plugins that "assume" CONTENT_ORIGIN is always set, will fail if it's not-set
      • ansible, container, maven, python are prob affected, may be others - LOOK CAREFULLY
    • plugins will need a "CONTENT_ORIGIN is not set" test added
    • mdellweg to add "can we have a "it's not set" test added, to the PR
      • work w/ hyagi on details
    • plugin teams are/will be respoinsible for addressing this change - NOT the services team - in the plugins
  • core/3.70 - are we at go/no-go?

    • django deprecation status (pbrochad):
      • https://github.com/pulp/pulpcore/pull/6058
      • Problem: users can't update their deployment settings before the removal release
      • Approach: hack to allow settings update between \([3.70,3.85)\) and schedule removal to 3.85.
      • AI: apply fixes for hack side-effects on plugins
        • ready to be reviewed
      • discussion:
        • plugins would like to be able to "cross the boundary" - ie be able to run on <3.70 (so know old-way) and on 3.70+ 9and therefore know the new-way as well
        • plugins MUST adapt to the change to be 3.70-compatible
        • can pedro organize
        • hyagi: what about oci_env and pulp-operator impacts?
          • hyagi volunteers for operator
          • oci-env needs eyes before 3.85 releases
            • sooner is better
    • we need to socialize the potential work that plugins need to do
      • CONTENT_ORIGIN
      • django STORAGES change
    • ruby bindings bump is almost there: https://github.com/pulp/pulp-openapi-generator/pull/108
      • not being tested in CI because we don't use nightly
      • being tested locally
      • comes back to "we would really like to not publish bindings"
    • python bindings updated by LOTS of changes
      • it is likely that plugin-tests are going to break
      • see 6138
    • blocker milestone
      • clearly no release today
  • backport migration (field add) and patch alternative (pbrochad)

    • backporting migrations isa problem
    • fixing the "missing remote artifact problem" PR
  • adding hyagi to core team

    • we had to close PR unecessarily because hyagi coudn't push to dkliban's PR
    • ggainey volunteers
    • how abouteverybody?
      • this is actually separate from the "taking over someone else's branch"
  • Think about format for this mtg/release-mtg-lead rotations for next time

    • If you have Strong Opinions - bring to mtg!

Archives

tags: pulpcore
Select a repo