# Pulpcore/katello/pulp_deb integration meeting, 2023 archive

## 2023-12-05 1000-1030 GMT-5
### Attendees: ggainey, mbucher, hstct, sjham iballou, dalley
### Regrets: quba42
### Agenda:
* Previous AIs:
* Meeting Review/Resched for 2024!
* Still a useful mtg?
* Still a useful cadence?
* Still a useful time?
* Yes to all
* AI: ggainey to sched for 9-JAN, and then 5 1st-Tues into 2024
* Question for Katello QE (if present): How do you test for N-1 proxy compatibility?
* a lot of hand-testing
* once basic smoke tests pass, move to N-1
* sync upstream all content-type repos, then SmartProxy sync the results
* QE uses robotello tests, will be getting enhanced
* there's a response "somewhere" in the foreman-community on this
* Katello PR feedback:
* Add SmartProxy content repair button (see last meeting's agenda)
https://github.com/Katello/katello/pull/10802
* internal discussions happening on when/how to run repair
* global-repair is very time-consuming/Fun
* lots of container-repair issues addressed, need to be examined
* RE global repair - does it continue if one repo fails?
* AI: [iballou] let's get the discussion into the PR see people can see/contribute
* Fix Incremental Update concurrency
https://github.com/Katello/katello/pull/10781
* Inconsistent repo publication
https://github.com/Katello/katello/pull/10776
* already on internal-katello board/to-do
* hstct: publish-time for deb-repos got really long
* due to on-demand/RemoteArtifact lack of indexing
* https://github.com/pulp/pulpcore/pull/4836 adds an index
* publish-time grew from 8 hours to 20+ at large installation
* AI: hstct to report back on customer experience once applied
* may cause specific issues in pulp-deb due to its code-path
* specifically, affects installations w/ large amounts of on_demand content and publishing
* https://github.com/pulp/pulp_rpm/issues/3254
* need createrepo update as well?
* yes - need 1.something
* pulp_rpm patch updates createrepo_c dep as well
* backported to rpm/3.18, .19, and .22

### Action Items:
* ggainey: schedule mtgs for 2024
* iballou: summarize discussion RE global-repair into PR
* iballou: Poke People to get other katello PRs reviewed
* hstct to report back on customer experience once RemoteArtifact indices applied
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-11-14
### Attendees: quba42, iballou, mbucher, hstct, vsedmik, ggainey
### Regrets:
### Agenda:
* Previous AIs:
* quba42: Not yet any progress on structured APT (I got consumed in firefighting...)
* Heal failed publications PR: https://github.com/Katello/katello/pull/10776
* can we get some katello eyes to understand the failing test (and understand/approve)
* Regarding: https://community.theforeman.org/t/slow-content-proxy-sync/35162/38
* Is there a conversation about pulpcore and Foreman/Katello syncing up postgresql versions?
* See also: https://community.theforeman.org/t/does-foreman-support-postgresql-13/35639
* katello can have platform-limitation (ie EL9) issues to resolve
* module-streams - can they help here?
* Does Katello provide any features to trigger pulpcore's artifact repair? Should it? (I am thinking a rake task or similar, or perhaps even a UI button?)
* See: https://docs.pulpproject.org/pulpcore/workflows/repairing-pulp.html
* Things to consider:
* Repair on server and repair on capsule?
* How to report corrupt artifacts that could not be re-downloaded? (capsule vs server)
* katello-current supports just the repair-by-repo
* katello *could* support the "fix everything" endpoint - RFE please
* see https://projects.theforeman.org/issues/36803
* verify-checksum sync on repo does this (from katello)
* there have been similar asks from other users - you are not alone!
* pulp_rpm publishes or syncs wrong modules.yaml:
* We have a foreman/katello that regularily synchs Alma/RHEL/Oracle/Rocky 8
* From time to time it happens that one published Appstream-repo has the modules.yaml from another distro (e.g. Appstream RHEL has Oracle's modules.yaml)
* It cannot be fixed by re-publishing, only resynching fixes it.
* In detail pulp debugging has yet to be done.
* is this https://github.com/pulp/pulp_rpm/issues/3241 ?
* UI performance:
* We are getting customer feedback that it "feels like" things are growing slower and slower with each new version.
* Note that these reports are mostly about Foreman 3.3, Katello 4.5
* Question: Have you heard similar complaints? Did anything change (like system requirements, optimization confg)?
* we have had specific performance issues, mostly on Really Large Instances
* (list of closed perf-related issues here)
* All hosts page: According to Bernhard Suttner a lot of data (including a lot of Katello related stuff) is loaded that is not even necessarily displayed, can this be optimized. (Examples: content_facets / concerns / host_managed_extensions)
* from katello:
* applicability has had some perf updates recently
* host-queries can be really slow
* RE katello/4.5: can't think of any current complaints?
* can we get specifics? please?
* welcome to vsedmik (katello QE)
### Action Items:
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39
* quba42 to update OR-4242

## 2023-10-03 1000-1030 GMT-4
### Attendees: sjha, quba42
### Regrets: iballou, ggainey, hstct
### Agenda:
* Previous AIs:
* RE structured-deb-content
* AI: [quba42] get forklift-env set up w/ POC uip and running
* still in-progress
* pulp_deb/3.0 gated us - but it has been released!!
* AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
* quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
* Remaining pulp_deb 3.0 packaging PRs:
* Merged to nightly, but not for Katello 4.10.
* https://github.com/theforeman/foreman-packaging/pull/9777
* https://github.com/theforeman/foreman-packaging/pull/9810
* Import/Export nil vs empty string issues. Has anyone else encountered this as a problem?
* sjha: Question regarding smart proxy syncs for deb content:
* Working on a new feature to display content counts for smart proxy syncs.
* Noticed that Debian appeared to have two contents on the smart proxy for every one content on the server CV/Lcenv.
* It appears every .deb package is duplicated on the smart proxy with two alternate relative paths.
* Is this a bug?
* quba42: This is an unintended design consequence of having both structured and simple publication mode on the server.
* This will no longer be the case if we eventually transition to using only structued APT.
* For now, we will just have to live with the fact that the content counts for .deb content will be double for the proxy.
* Side note: For synced content the content count on the proxy should be exactly twice what it is on the server, however this may not be true for uploaded packages.

### Action Items:
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-09-05 1000-1030 GMT-4
### Attendees: quba42, sjha, iballou, ggainey
### Regrets:
### Agenda:
* Previous AIs:
* RE structured-deb-content
* AI: [quba42] get forklift-env set up w/ POC uip and running
* still in-progress
* pulp_deb/3.0 gated us - but it has been released!!
* AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
* AI: [iballou] nag more folk to review the structured-content-doc
* lots of responses already in thread
* [pulp_deb 3.0 is released](https://discourse.pulpproject.org/t/pulp-deb-3-0-0-has-been-released/989).
* What Katello version should this be added to? (long running DB migration!)
* requires matching client-gem packaging, implies a potentially long-running migration
* prob too late for katello/4.10
* let's shoot for katello/4.11?
* how does pulp_deb make sure this ends up with the right katello/foreman packaging dir?
* foreman-installer will handle updating if/when pulp_deb/3.0 "magically shows up" in the katello/4.10-pulpcore/3.28 repo-structure
* how to handle N-1 smartproxy setup?
* can a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
* this needs to be flattened first
* quba42 to work on investigation here
* pulp_deb bugfix releases, e.g.: https://github.com/theforeman/pulpcore-packaging/pull/713
* should "just work"
* add required PRs to approproiaite release-branches
* What exactly is the state of pulp_rpm dependency solving?
* https://projects.theforeman.org/issues/35568
* https://bugzilla.redhat.com/show_bug.cgi?id=2132472
* https://github.com/pulp/pulp_rpm/issues/2824
* "dependency solving is harder than you think it is, and probably doesn't do what you wish it did"
* pulp_rpm has it, will continue to have it, it works for a large number of use-cases but continues to have Painful Edge Cases
* would be Really Nifty to have pulp_deb participation in foreman-community-demos
* every 3 weeks on Thurs?
* this is a fine, fine idea
### Action Items:
* RE structured-deb-content
* AI: [quba42] get forklift-env set up w/ POC uip and running
* still in-progress
* pulp_deb/3.0 gated us - but it has been released!!
* AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
* quba42: investigate whether a pulp_deb/3.0 client-gem talk to a pulp_deb/2.21.X system?
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-08-01 1000-1030 GMT-5
### Attendees: ggainey, quba42, hstct, iballou, sjha
### Regrets:
### Agenda:
* Previous AIs:
* DONE AI: [ggainey] nag iballou RE [katello RFE](https://community.theforeman.org/t/auto-select-errata-which-is-solved-by-installing-another-erratum/31615), pulp_deb/3.0 migration issue
* DONE AI: [quba42] ask ewoud about theforeman-reviewers for PR
* DONE AI: [quba42] to finally write that RFC on "structured APT"
* https://community.theforeman.org/t/rfc-transitioning-katello-to-structured-apt-deb-content/34262
* My apologies for the amount of text.
* We had a strange edge case where a Pulp publication failed because of full disk. (The Pulp sync was successfull) This means the repo was synced and all the packages showed up in Katello, but the content wasn't actually being served in the CV. Re-syncing did nothing, because the Pulp re-sync reported no new packages, so Katello did not bother to re-publish. Ultimately we solved it by doing a "republish repository metadata" using hammer.
* Are there plans for "republish repository metadata" to come back to the UI, or will it remain hammer only?
* The above showed that if sync succeeds, but publish fails, regular Katello usage will never repair this situation, can this be improved?
* there are lots of ways metadata might get corrupted
* katello is restoring "republish" to the UI for non-mirror-complete repositories
* there is a push to make it globally available in spite of ways that might Go Worng - discussion in progress
* review of structured-apt writeup
* will be in pulp_deb 3.0
* 2 katello PRs, 1 subscription-manager PR
* how to make forklist use unreleased versions of "things"?
* iballou: usually just do a pip-install of source
* orcharino test-setup - see quba42 for details
* still [edge-cases](https://community.theforeman.org/t/rfc-transitioning-katello-to-structured-apt-deb-content/34262#remaining-challenges-and-limitations-5) that need to be solved
* prob need to get together w/ someone on candlepin to iron out some details
* who should quba42 work with/demo to, to work thru the pulp_deb/katello/candlepin pieces?
* paji/jlentz/croberts?
* AI: [quba42] get forklift-env set up w/ POC uip and running
* AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
* AI: [iballou] nag more folk to review the structured-content-doc
* is this something katello/4.10 has-to have?
* is there a pulp_deb compatible with core/3.28?
* yes - 2.22 (we think)
* is that the right version for katello/4.10? If not, tell iballou "now"
* there exists a long data-migration to get to pulp_deb/3
* notes from chat
* Samir Jha
1. This deb release will be compatible with pulpcore >= 3.25 ?
2. Is there any migration involved of existing deb repositories?
3. We can move those settings to installer and possibly options there like we do for --enable-deb?
* hstct responds
1. right now it is compatible with it.
2. yes there is I believe it this one here: https://github.com/pulp/pulp_deb/blob/main/pulp_deb/app/migrations/0025_merge_colliding_structure_content.py
### Action Items:
* RE structured-deb-content
* AI: [quba42] get forklift-env set up w/ POC uip and running
* AI: [quba42/iballou] work together to get a meeting together w/ Appropriate Knowledgeable Folk to go over the POC
* AI: [iballou] nag more folk to review the structured-content-doc
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-07-11 1000-1030 GMT-5
### Attendees: quba42, hstct, bernhards, ggainey, mbucher
### Regrets: all of katello
### Agenda:
* Previous AIs:
* RFC: https://projects.theforeman.org/issues/35778
* AI: [ggainey] poke iballou to respond to this thread!
* "sometimes" the requested behavior works
* applying N+1 can make N unapplicable?
* get that fact into the thread!
* AI: [ggainey] add thoughts to discourse thread [DONE]
* More of a Foreman deployment (rather than a Katello topic), but whom to ask (?) about review for: https://github.com/theforeman/foreman/pull/9318
* ggainey does not know - maybe ask Ewoud in Matrix?
* Transitioning Katello to “structured APT”:
* Tracker: https://projects.theforeman.org/issues/35959
* Draft PR: https://github.com/Katello/katello/pull/10420
* We have encountered a challenge: cp2_content always seems to be updated simultaneously across all LCENVs, as soon as the corresponding katello_root_repository is updated. Would it be possible to have different cp2_content associated with different LCENV based on the katello_repository in use for that LCENV?
* add this to the "upcoming" RFC
* Structured upload PR: https://github.com/Katello/katello/pull/10639
* community RFC: quba42 has not yet written this.
* Will include instructions how to test.
* next pulp_deb release will be called 3.0
* big db-migration incoming
* may take A Long Time to run the migration - needs to be in the CHANGELOG
* prob need a blog-post...somewhere?
* maybe start with ATIX blog
* can this be included in katello/4.10 timeframe?
* AI: [ggainey] bring to iballou's attention

### Action Items:
* AI: [ggainey] nag iballou RE [katello RFE](https://community.theforeman.org/t/auto-select-errata-which-is-solved-by-installing-another-erratum/31615), pulp_deb/3.0 migration issue [DONE]
* AI: [quba42] ask ewoud about theforeman-reviewers for PR
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-06-06 1000-1030 GMT-5
### Attendees: hstct, sjha, quba42, iballou
### Regrets: ggainey
### Agenda:
* iballou to host due to ggainey out for an appt
* Thanks Ian!
* Previous AIs:
* AI: [ggainey] is there a list of core-releases we backport stuff to?
* eg, "what releases does katello depend on"
* https://docs.pulpproject.org/pulpcore/release_process.html#active-branches-as-of-2023-05-16
* docs get out of date - CI testing is canonical
* https://github.com/pulp/pulpcore/blob/main/template_config.yml#L21
* Transitioning Katello to "structured APT"
* Existing PR: https://github.com/Katello/katello/pull/10420
* Can only really be merged once the whole feature is ready...
* Needs to be introduced step by step using settings
* Need to go into some background...
* Our internal test branch uses the following in `/etc/foreman/plugins/katello.yml`:

        :deb:
          :use_structured_content_for_clients: true
          :publish_simple_content: false 

* AI: quba42 to open a Foreman tracker and write a community RFC

### Action Items:
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-05-02 1000-1030 GMT-5
### Attendees: hstct, sjha, quba42, ggainey, iballou
### Regrets:
### Agenda:
* Previous AIs:
* core/3.24 releasing TODAY
* core/3.25 #soon - mid/end of May?
* https://github.com/pulp/pulpcore/milestone/8
* django-4.2
* psycopg3
* any deprecation warnings? fix em now!
* next breaking-release: core/3.40
* see https://hackmd.io/XfI9DxymSAu6UJnblh7prQ#March-7-2023 for discussion
* discussion
* packaging?
* where do plugins look to find dperecation-warnings?
* quba42 to ask in OpenFloor
* AI: [ggainey] is there a list of core-releases we backport stuff to?
* write this down somewhere
* eg, "what releases does katello depend on"
* pulp_deb release: https://discourse.pulpproject.org/t/pulp-deb-2-20-2-has-been-released/833
* bindings change implied?
* makes katello-packaging harder
* serializer-change - almost certainly yes
* can this be ready for katello/4.9 release?
* should be possible
* quba42 to investigate/do/poke katello if more info needed
* SLES advisory export/import problem
* https://github.com/pulp/pulp_rpm/issues/3127
* https://github.com/pulp/pulp_rpm/pull/3128
* Need to talk to pulp_rpm about this, did not have time to address it here.

* Working with the Katello upload implementation:
* https://github.com/Katello/katello/blob/master/app/lib/actions/pulp3/orchestration/repository/upload_content.rb
* Katello is open to using the `repository` parameter on content endpoints instead of separately calling the `modify` API after creating the content.
* If the upload workflow needs to change for plugins that implement "create or retireve" logic on the Pulp side, there are two options:
* A flag in the existing workflow (if the two workflows are not very different)
* A separate workflow that is called for some plugins
* The goal should be to ultimately move all plugins to the new workflow and move "create or retrieve" logic from Katello to Pulp.

### Action Items:
* AI: [ggainey] is there a list of core-releases we backport stuff to?
* write this down somewhere
* eg, "what releases does katello depend on"
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-04-04 1000-1030 GMT-5
### Attendees: quba42, hstct, sjha, ggainey
### Regrets:
### Agenda:
* Previous AIs:

* Possibility of releasing pulp_deb orphan cleanup fix for released versions of Katello?
* See: https://community.theforeman.org/t/katello-4-8-0-rc1-feedback/33039/4
* fix includes a migration
* it's the only "new" migration since release
* should be "safe" in all respects (idempotent, no other possibly-skipped migrations "in the way")
* which version backporting to?
* pulp_deb/2.20.1, for katello/4.7 users
* Any plans to drop Puppet 6 support? If so, with what Foreman/Katello version?
* See also: https://docs.theforeman.org/3.6/Release_Notes/index-katello.html#_puppet_6_is_eol
* Foreman won't "drop" Puppet/6 support "yet"
* further plans - talk to TheForeman gang
* Adding a mix of synced and uploaded content in a single repository
* ATIX opinion:
* It is a best practice to use separate repos for the two use cases. (Especially in the case of deb/APT content)
* Katello should be opinionated about enforcing best practice here.
* Pulp probably does not need to be as opinionated about this as Katello should be.
* Pulp opinion: Pulp **should not/cannot** be opinionated in this area
* See also: https://discourse.pulpproject.org/t/rfc-separating-sync-and-upload-workflows-in-pulp-deb-and-others/787/9
* can we get a discussion opened on theforeman discourse to gather user-feedback?
* At a minimum, can we get the katello UI to give some guidance to keep users from *accidentally* shoot themselves in the foot
### Action Items:
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-03-07 1000-1030 GMT-5
### Attendees: ggainey, iballou, sjha, quba42, hstct
### Regrets:
### Agenda:
* Previous AIs:
* AI: [ggainey] schedule 6 months starting first tues in March
* AI: [ggainey] clean up mtg-doc
* https://hackmd.io/@pulp/pulp-deb-katello-integration_2022
* https://hackmd.io/@pulp/pulp-deb-katello-integration_2021
* pulp_rpm to get rid of dependency solving?! (https://discourse.pulpproject.org/t/katello-pulp3-integration-meeting/37/73)
* "fun" discussion ensues
* no, pulp_rpm isn't dropping depsolve any time soon
* How important a feature is ACS for pulp_deb?
* My gut feeling is that people can live without this feature, and other priorities should take precedence. However, I am wondering if there are potential pitfalls in pulp_deb does not implementing this?
* ACS will allow smart-proxies to sync directly from upstream-deb-repos
* not 'necessary' to implement, totally optional
* some RH users are excited about it, but we don't have any metrics on actual uptake
* S3-storage for pulp-artifacts in katello/foreman-proxy-content? *Out-of-scope*, *on the roadmap*, or *waiting for someone to implement it*?
* https://github.com/pulp/pulpcore/issues/3358
* Pulp implements this
* needs installer-support to turn on the right cfg-options
* katello has related issue
* "switching to S3 broke things" - to be reviewed/fixed
* Notes from Pulp Gang at CfgMgtCamp:
* several community members were asking for RBAC in pulp_deb (Magnus and Alexander?)
* ttereshc will ping Magnus to open an issue so y'all can talk directly
* https://github.com/pulp/pulp_deb/issues/392
* ATIX met w/ ddavis' team
* may invite their team to this mtg?
* not as interested in pulp_deb/katello integration
* What version of katello is ATIX using?
* 4.1 katello, branch using 4.3
* sync up katello/foreman versions? - not currently
* fun with LEAPP is happening at ATIX
* be aware of the postgres-reindex-issue!
* https://bugzilla.redhat.com/show_bug.cgi?id=2161929
* https://community.theforeman.org/t/possible-to-use-leapp-for-oracle-7-oracle-8-migration/32273/34
* Q: is ATIX working on new-host-details page?
* prob become important post-EL7-to-EL8 upgrade/katello-4.3
* Discussion around pulpcore-version-release-process upcoming changes
* will be discussed in/on/discourse/matrix

### Action Items:
* AI: ggainey to look at "can one decide to move content to a diff backend storage"
* not yet - https://github.com/pulp/pulpcore/issues/3358
* ggainey to add minutes to https://discourse.pulpproject.org/t/katello-pulp-community-integration-meeting-minutes/39

## 2023-01-10 1000-1030 GMT-5
### Attendees: ggainey, iballou, sjha, quba42, hstct
### Regrets:
### Agenda:
* Previous AIs:
* AI: [iballou] to look at SCA-thread, add debugging commentary if/as useful
* Resulted in https://projects.theforeman.org/issues/35713
* AI: [ATIX] to start a thread on import/export with "katello" tag, maybe ping iballou/sjha/paji in #foreman-dev
* https://community.theforeman.org/t/support-importing-content-views-with-deb-content-with-hammer-content-import/31178
* AI: [ATIX] someone at ATIX will poke at issue [#35362](https://projects.theforeman.org/issues/35362)
* We believe it is fixed with [these pulp_deb bugfix releases](https://discourse.pulpproject.org/t/pulp-deb-2-20-1-2-19-3-2-18-3-have-been-released/716)
* Requires this packaging PR: https://github.com/theforeman/pulpcore-packaging/pull/603

* New Items
* What is the deprecation timeline for the old host details page (what Foreman version switches by default, what version drops the old one completely?)
* Thinking about https://projects.theforeman.org/issues/35713
* discussion has happened, we think we know the answer
* When will "legacy" page get totally removed?
* not before katello/4.8, most likely katello/4.9
* katello/4.5 allows choosing which you want
* katello/4.7 makes new-page The Default
* katello will add a "debian" tag to the content-page (see #35713 above)
* Does somebody remember https://github.com/pulp/pulpcore/issues/2250 ?
* Following links eventually led me to https://access.redhat.com/solutions/6977656, but I am still unsure about the full scope/background of the problem.
* Is there a pulpcore version, that completely solves this issue, or is it a case of "performance has improved over time to the point where it is no longer a critical issue"?
* we think we're happy w/ the kbase described above
* lots of performance/memory fixes going into pulpcore - stay up to date(er)?
* What else is needed to make [this](https://github.com/pulp/pulpcore/pull/3038) work for Katello 4.3.1/pulpcore 3.16.12?
* may actually be working for ATIX - needs new-year-neurons to look at it
* How should this meeting proceed in 2023?
* nag paji as needed
* 1st Tues works
* Feb is to be skipped - cfgmgt camp and lots of year-startup happening
* AI: [ggainey] schedule 6 months starting first tues in March
* Proposal: Archive 2021/2022 minutes into a separate doc
* no major objection
* AI: [ggainey] clean up mtg-doc