HackMD
Pulpcore 2022 archive
December 20, 2022
- "stream" test is really unreliable
- we need to fix this problem, or disable the unreliable tests - they teach us to "ignore red"
- [ipanova] to my knowledge Michal was working on its fix, not trivial
- this https://github.com/pulp/pulpcore/issues/3336 ?
- if it's not going to happen Soon, then ggainey would like the test disabled for now
- [ipanova] to my knowledge Michal was working on its fix, not trivial
- we need to fix this problem, or disable the unreliable tests - they teach us to "ignore red"
- Upstream high-value tickets, should we prio-list them?
- Why? – Came from user feedback sessions
- not much of a quorum - move to Upcoming again
- FWIW, ggainey and decko in favor
- discussion RE putting more focus on community asks
- treat upstream-community as a formal Stakeholder
- def a good thing
- think about 3-month-planning improvement
- ggainey out 27-DEC - cancel the mtg next week? or appoint a facilitator?
AI: ggainey to cancel next week
- gerrod becomes facilitator for Jan-Feb
- prob should not be "optional" for those months :)
AI: ggainey to voluntold gerrod :)
December 13, 2022
- [decko] reviewed/approved 3427 (HStore)
- Zero-downtime next steps
- https://github.com/pulp/pulpcore/pull/3448
- all pulpcore and pulp_file migrations need to be zero-downtime compatible
- all tasks need to be backwards compatible
- AI: review , ggainey added as reviewer
- 3190 (domains)
- discussion ensues
- AI: [gerrod] run all/many plugins' CI tests against the PR
- who are the SMEs?
- gerrod
- RTFM group?
- gerrod's presentation makes it possible for everyone to understand
- until we have confident LG2M from confident SMEs, we should maybe release core/3.22 sans domains, then 3.23 w/ this
- core/3.22 discussion around probs caused by merged content-app pr
- affects python and container
- do we revert? wait for a fix? hold off 3.22?
- need to resolve prior to a core/3.22 release
- needs to be fixed/addressed in pulpcore - issue?
- "fixing it" in container/python
- fails at core/3.22 release if older plugins are installed
- what happens if we "fix" this in plugins and backport?
- container tinks this is supportable
- how many active z-streams? (2 for container)
- CI is currently broken because of this problem
- about 1/3 tests in container
- "some" tests broken in python
- reverting the two commits fixes CI - but removes the major drivers for core/3.22 being date-driven
- revert-commit-msg needs to explain WHY we're reverting
- need to have something in place for core/3.22 and pulp_rpm
- we need to expand on pulpcore's interface in this area, in a way that plugins can reliably take advantage of it - for core/3.25
- AI: [bmbouter] to open an issue, for this issue
- AI: this needs A LOT of discussion at OpenFloor
December 6, 2022
- Proposal: have pulpcore adopt zero-downtime migration policy
- to do this we need to write plugin-writer docs first
- https://discourse.pulpproject.org/t/support-zero-downtime-updates/645/18
- https://hackmd.io/dxm3JG0bSEizAJeM6E9HUw
- start with pulpcore and pulp_file and see How It All Works, starting with 3.23
- THIS WILL BE AN EXPERIMENT - no guarantees stated or implied!
- AI: [bmbouter] open a doc-issue
- AI: [bmbouter] write up plugin docs for this
- Proposal: give decko write-perms to pulpcore
- previously, generally waited a year/major feature addition as SME to add
- feels like we need to give it decko now to allow reviews for pytest work to "count"
AI: [dkliban] add decko to pulpcore team
- remove ppicka invitation/commit bit since January
- AI: [bmbouter] to remove from mtg invite
- AI: [team] move ppicka to Contributor
- Should settings be semver controlled?
- dkliban: leaning yes
- bmbouter: users are asking for it, we should give it to them
- still can do "emergency" changes (e.g. security issues) if we need to under semver
- x9c4: can we get away with not-defining and deal w/ it case-by-case
- since semver allows for emergency fixes, prob ok
- ggainey: also leaning yes
- mdellweg: we can decide for pulpcore, but not for Everyone
- maybe we should put up in discourse for a week for discussion
- bmbouter: transparency is good
- AI: [x9c4] open thread in discourse
- AI: [all] revisit and confirm at next mtg
- Labels PR is ready and needs review/merge: https://github.com/pulp/pulpcore/pull/3427
- do we want this in 3.22?
- it's a bugfix
- merge it - "waiting in main doesn't actually help"
- 3.22 is 12-DEC - so review ASAP
- AI: [team] get 3427 reviewed so we can merge no later than Fri
- do we want this in 3.22?
- Q: how easy is it to create a new release?
- Proposal: we should do releases more often
- plugins that depend on core/main can't release until that main becomes a 'real' release
- Proposal: let's start by releasing quickly (two weeks? three?)
- also - no go/nogo
- strictly time-based
- Y-release ONLY if there are features
- required-PR has Implications
- what about backport policy?
- prob OK, for multiple reasons
- https://github.com/pulp/pulpcore/issues/3368
- can we make this a prio-list? - yes
November 29, 2022
- Are we going to actually pay attention to the CodeQL output?
- decko can't view https://github.com/pulp/pulpcore/security/code-scanning ?
- proposal: if we're not going to use, turn it off
- proposal: if we are going to use, then open issues and fix the reports
- [davidd] Pulp's policy around settings changes https://github.com/pulp/pulpcore/issues/3417#issuecomment-1329659503
- We set a number of settings and are concerned about settings changes
- If settings are not covered by semver, could there be a documented policy/deprecation cycle?
- proposal: settings should follow same semver policy as REST API
- i.e.: can deprecate, but not remove
- each rename/removal is Going To Be A Pain - "please don't" is the generic dev policy
- proposal: doc settings.local more in docs?
- pulp_labels
- https://github.com/pulp/pulpcore/pull/3427
- currently exposed for distributions, repositories, remotes (so far)
- jsonb is overkill
- would need to remove generic-relation
- exposes some of the functionality in the API
- breaking change?
- plugin API affected
- need to research if anyone is actively "aware of" the current implementation
- marked tech-preview - so (technically) "can do anything"
- OTOH - we' guarantee that plugin-breaking-changes won't happen until breaking-change releases (e.g. 3.25)
- Base model introduction would require Very Complicated Migration
- REST API unaffected
- poss bug in current REST
- pulpcore 3.22 GA is postponed
- all thumbs-up
- tech-preview discussion
- we have a tech-preview problem
- just because we say "we can break this", doesn't make it even remotely a good idea
- how long can/should something say "tech-preview"
- can we move to "how much is it used"?
- analytics to the rescue!
November 22, 2022
- Review ggainey's rearranging of this doc to see if it's what the team wants
November 15, 2022
- 3.22 release schedule
- ipanova is the release shepherd
- We're at 2 months since 3.21
- tentative GA date is Nov 30
- 16 and 23 of Nov go/no-go meetings
- what should go in?
- domains
- needs issue, add to 3.22 blockers [AI: gerrod]
- blockers
- domains
- django4/core-3.25 discussion
- django-4 happens in April
- 3.25 needs more time due to likelihood of breaking-changes
- don't want to get downstream products "stuck" on older versions that will EOL
- "when should 3.23 and 3.24 happen"
- IF 3.25 is going to be "django-4" - prob want to have significant burn-in
- 3.23 - prob end-of Jan, to get in anything we want to have available for "conference season" in Feb
- 3.24 - prob end-of March, for 2-month-cadence/pre-django-4
- 3.25/django-4 linkage is not required - let's "see what happens" w/ django and re-evaluate in April
- AI: ggainey to post to discourse [DONE]
- AI: ggainey to engage in "creative rearranging" of team doc [DONE]
November 1, 2022
- Failing test related to repo version delete
- dkliban to investigate
October 25, 2022
- Filter-process appears to need some more refactoring
- x9c4 is willing to do, but needs some buy-in for quick review turnarounds
- will fix a couple of openapi-schema bugs
- discussion ensues RE opportunity costs
- what about new content-types?
- how can we best prioritize competing priorities from groups that are not in agreement?
- let's get folk from "emerging" opportunities come talk to us directly
October 18, 2022
- CI failures on 3.16
- failure on the local filesystem runner
- proposal: skip the single test because it's preventing us from running all tests
- observation: this pulp_rpm has a similar failure https://github.com/pulp/pulp_rpm/pull/2838
- actually it's different and it needs an actual code change in pulp_rpm test to fix so it's unrelated
- deadlock PR desperately needs another review(er) please
- pulp_file needs some TLC
- 6 PRs open
- performance tests constantly failing - why? looks like it is fixed now :)
- design discussion for: https://github.com/pulp/pulpcore/issues/2851
- bmbouter scheduling the zero downtime working group
October 11, 2022
- https://github.com/pulp/pulp-manifest/ needs an owner.
- probably the pulpcore team
- mention the existence of the script/repository in the pulp_file documentation
- design discussion for: https://github.com/pulp/pulpcore/issues/2851
- zero-downtime working group call for participation:
- telemetry updates
- see the recent data: https://analytics.pulpproject.org/
October 4, 2022
- request for zero-downtime upgrades
- https://discourse.pulpproject.org/t/support-zero-downtime-updates/645/
- this would be a project-wide change that all plugins would have to adhere to
- advertising/discussing here but real discussion happens on discourse
- [bmbouter proposal] let's create a ticket and PoC it, we also need to ask for feedback at open floor
- [ipanova] this will most likely be a breaking change that will not come in earlier than 3.25. We should then pick when the time is right for the PoC so the code does not get old.
- 3.21 release highlights for AH, needed or not?
- Hide manifest files https://github.com/pulp/pulp_file/issues/837
- Would appreciate some feedback
- S3 storage docs https://issues.redhat.com/browse/AAH-1944
September 27, 2022
- https://github.com/pulp/pulpcore/issues/3212
- will have a dedicated meeting for this topic
- idea 1
- duplicate content between repositories
- create a content unit for each remote that syncs that content
- idea 2
- add foreign key to RemoteArtifact on ContentArtifact
- What are the right branches for the plugin template to be opened up against?
- We call them hot branches. We can still release on others, just not that fast.
- This plugin_template PR changes how this works: https://github.com/pulp/plugin_template/pull/693
- 3.7(AH), 3.14(Sat), 3.15(AH), 3.16(Sat), 3.18(AH, Sat), 3.21(AH, Sat)
- Do we still need 3.7, 3.14?
- 3.16, 3.18, 3.21 make a PR
- FYI the dependencies are now in-place for pulpcore, we'll watch dependabot do the rest
- FYI analytics.pulpproject.org is flowing
- Need to do some pulpcore-content peformance testing, e.g. 500K req / sec
September 20, 2022
- There's an ask for more opinions
- Some kanban reflections
- Outbound: How do issues move off the Kanban board? Lots of "done" issues currently, clutters the per-repo view
- https://github.com/orgs/pulp/projects/8/views/2?filterQuery=status%3A
- it is not implemented yet, but Tanya's idea was manually dispatching a workflow to clean the done tasks at the beginning of the planning
- Inbound: many plugins have nothing on the board, is that an accurate portrayal of priority or a process issue? Do we need to add items more frequently?
- Only 2 "free to take" issues outside of the RPM plugin
- Challenge: How do we manage the prio-list label with full inclusivity in an async environment?
- Idea: When adding/removing/modifying the prio-list label leave a comment with @pulp/core
- We lack hard limits on the in progress/review columns
- Items with reviews must move back to "in progress" if not ready to merge.
- What to do with PRs that never had 'prio-list' ( community contributions, some old issue before kanban era, )
- Outbound: How do issues move off the Kanban board? Lots of "done" issues currently, clutters the per-repo view
September 13, 2022
- who is going to take care of pulp2 eol annoucement https://discourse.pulpproject.org/t/archiving-pulp-2-plugin-code/582/5
- versioning dependencies
- components sme list
September 6, 2022
- Update list of SME
- content-type change affects a pulp_ansible test
- I am currently trying to get an answer whether it is an issue.
- https://github.com/pulp/pulp_ansible/pull/1229
- Discussion on https://github.com/pulp/pulpcore/pull/3136
- Suggestion: instead of using uuid directly, provide a pulp_uuid function which we can change the implementation of without needing future migrations
- Can be done independently of whether we adopt UUIDv7 right now
- Discuss further at 3.21 go-no-go meeting
- Suggestion: instead of using uuid directly, provide a pulp_uuid function which we can change the implementation of without needing future migrations
August 30, 2022
- pulpcore 3.21 release plans?
- https://discourse.pulpproject.org/t/pulpcore-3-21-release-schedule/573/
- go/no-go today, mdellweg will run
- need someone to regularly go to katello integration call and report for pulpcore
- dkliban to take over the reporting role for pulpcore
- We need to have rules for approvals.
- proposal: Require one Approval from SME and one non-necessarily-SME.
- rationale: We do not have enough people to find two SME for each PR. Admin-merge is not a good solution (and an even harder requirement). Each pulpcore committer should be able to find code smells.
- We're supposed to have two SMEs for each component, so couldn't we have two SME reviewers?
- even when we do - if one (of the two) SMEs submits the PR, we're still short a reviewer
- problem: we don't have the SMEs for pulpcore components written down somewhere
- solution: mdellweg to write up SME components and list
- We're supposed to have two SMEs for each component, so couldn't we have two SME reviewers?
- FYI: pulp-smash proposal to be put into read-only mode on Sept 5th
- set prio-list label
- to review asynchronously and propose 3 items via pulp-dev, then we will review next week
- added prio-list query to the top of this hackmd https://github.com/pulp/pulpcore/issues?q=is%3Aopen+is%3Aissue+label%3Aprio-list
- review/unblock any assigned tickets, so folks have no more than 2 items assigned (or find a good reason to increase the limit)
- backport discussion
- still doing updating automated CI updates for every release branch back to 3.14
- suggestion: have a list of explicit "supported releases"
- AI for dkliban to write up and implement
- suggestion: after a product release drops active support, we do stop eager backports
- e.g. Once Satellite 6.N+1 comes out, "active" support stops and they only patch critical issues, z-streams not planned in advance
- still doing updating automated CI updates for every release branch back to 3.14
August 23th, 2022
- Discussing the Squashing Migrations plan
- https://discourse.pulpproject.org/t/squashing-migrations/571
- AI: discuss on discourse please
- pulpcore 3.21 release plans?
- any stakeholders waiting for it and blockers?
- 1 blocker: https://github.com/pulp/pulpcore/milestone/4
- Satellite and AH might both use 3.21 if released on time for their schedule
- any stakeholders waiting for it and blockers?
- Proposal: enabling anonymous telemetry submission for end users in 3.21.0
- https://github.com/pulp/pulpcore/pull/3116
- [bmbouter] Telemetry process needs to be in our docs
- Are we already using the new label?
- prio-list
- Releases are currently broken
August 16th, 2022
- When to upgrade to Django 4?
- django timelines for deprecation https://www.djangoproject.com/download/
August 9th, 2022
-
[lmjachky - PTO] Please, take a look at the current implementation of the remotes removal. We decided to implement an option where a user needs to specify the "force=True" header to delete the remote if it is being used by existing content artifacts. Otherwise, an HTTP 406 error is returned by Pulp:
https://discourse.pulpproject.org/t/when-to-allow-removal-of-remotes-used-for-on-demand-syncing/489/16 [ipanova] will comment- we can't use force due to it being a DELETE call
- let's not create a force now because we'd have to make an entirely new endpoint
- let's have a great error message returned
- we need tests
- [AI] @ipanova to represent ^ changes on the discourse and the PR reviews
-
PoC pulp-operator in Golang https://docs.google.com/document/d/1j6eanLgVz4HboEslzhjhOGKK4HF-jSSuC2wSC34eVpA/edit#heading=h.mdfz93aqubxv (pros and cons in the doc)
July 19, 2022
- proposal to sync flake8 configs in the project
- https://github.com/pulp/plugin_template/pull/558
- no concerns heard at the meeting
- Remove david from pypi projects
- Change test assumption to assume all tests will be run on the same system pulp is installed on
- https://discourse.pulpproject.org/t/functional-tests-will-now-run-inside-the-container-running-pulp/498/5
- would include retiring pulp-smash
- Yank pulp-smash from pypi?
- would include retiring pulp-smash
- https://discourse.pulpproject.org/t/functional-tests-will-now-run-inside-the-container-running-pulp/498/5
July 12, 2022
- proposal: remove pep8speaks. It's redundant give flake8 is in the linter. currently enabled on pulpcore, pulp_file, plugin_template, and pulp_ansible
- Time to squash migrations?
- it is almost 100 now (pulpcore)
- about 40 in ansible, container, rpm
- https://docs.djangoproject.com/en/4.0/topics/migrations/#squashing-migrations
- Can this be done without A) special requirements about the upgrade process and B) requiring plugins to squash at the same time?
- Call for feedback: an experiment: marking 50% of tests as nightly only
- https://discourse.pulpproject.org/t/marking-majority-of-tests-to-only-run-nightly/491/4
- How are the tests chosen?
- What would be the revert policy?
July 5, 2022
- Discourse down?
- galaxy NG requested changes (if you missed this meeting, check the comments on the issues to see what was agreed)
- Discussion: Preventing the removal of remotes that are in-use
June 28, 2022
- need to reduce CI runtime, right now it's 40+ minutes per PR (assuming no failures!)
- proposing we mark 70% of tests to only run nightly in pulpcore and pulp_file
- https://discourse.pulpproject.org/t/marking-majority-of-tests-to-only-run-nightly/491
- call for interest on generic UI components for Pulp
- HA and health checks
- https://discourse.pulpproject.org/t/pulp-health-checks/514
- healthy worker?
- don't need to look at worker health since it's a totally async component
- healthy content app?
- use
HEAD /pulp/content/if it returns 200 it'sImage Not Showing Possible Reasons- The image file may be corrupted
- The server hosting the image is unavailable
- The image path is incorrect
- The image format is not supported
- use
- galaxy NG requested changes (if you missed this meeting, check the comments on the issues to see what was agreed)
- https://github.com/pulp/pulpcore/issues/2865
- https://github.com/pulp/pulpcore/issues/2825
- https://github.com/pulp/pulpcore/issues/2824
- did not discuss this one, need input from more people
June 21st
- Pulpcore 3.20 release
June 14th
- [lmjachky] What to do next with https://github.com/pulp/pulpcore/pull/2713?
- exporting/importing repositories, distributions, and other types
- converting to different types of repositories (https://github.com/pulp/pulp_container/pull/826#issuecomment-1153680951)
- telemetry update:
- dev systems will now report to the "dev analytics site" which gets summarized every 24 hours
- going to make new chart for x.y components instead of x.y.z
- going to configure analytics.pulpproject.org to the workers with a "coming soon" page
- ordering filter vs sort filter
June 7th
- telemetry pr in-review
- CI Update
- Functional tests to be run inside the container instead of the host
- Installation is a nightmere today
- More similar to dev-environment
- Functional tests to be run inside the container instead of the host
- Reducing CI runtime:
- moving most tests to nightly
- Collaboration with david N on using galaxy NG generic React + Patternfly components for a future Pulp UI
May 31st
- [mdellweg] Serving artifact urls from the pulp_api for external verification (e.g. Virus scanning)
- telemetry PoC request for review
- changing the tests run by default
- LTS continued?
- wait on Dennis' results
May 24th
- [ipanova] need to release 3.19.1 https://github.com/pulp/pulpcore/pull/2567
- continued LTS discussion
- What is will be the next LTS?
- How aggressively should we be backporting to the LTS?
- How many LTS branches are supported concurrently?
- telemetry PoC request for comment
May 17th
- Look through remaining maybe-finished issues
- Convert char to text field https://github.com/pulp/pulp_rpm/pull/2515
- guardian removal side effects https://hackmd.io/sPmy4YS6TQ-iGn6xyYZP7g?both
- Discussing LTS declaration for pulpcore
- https://discourse.pulpproject.org/t/need-to-reduce-the-number-of-release-branches-aka-we-need-an-lts-strategy/449/
- Is 3.18 an acceptable one
- we think so, but let's discuss it a little more (bmbouter wants to think it over some more)
- What is will be the next LTS?
- How aggressively should we be backporting to the LTS?
- How many LTS branches are supported concurrently?
May 10th
- Look through remaining maybe-finished issues
- z-releases
- 3.20 tentative release date? first go/no-go?
- status of django 2.2 in pulpcore 3.7 / 3.14
- updating core/3.7 to last 2.2LTS, will require a change in (at least) matching pulp_container version
- need to evaluate other plugins (
Required PR:?)- file/rpm/container/ansible
- https://github.com/pulp/pulpcore/pull/2651
- need a plan for who/when/testing
- bmbouter and dkliban are updating pulp_file tests to use pytest, merging them with pulpcore
May 3rd
- Removing the "test keys" from the repo, recently flagged by security scanners and bmbouter was notified
- 3.20 removal of MissingResource exception
- 3.20 tentative date (or not ready yet?)
- (dalley) not ready yet IMO
- Worth doing new Z-releases today?
April 26th
- Discussion around list APIs for Exporters and Remotes
- Downloader SIG: need to schedule kick-off meeting
- Release rotation - time to rotate?
- Brian will take care of 3.20 release
- what to do with the blogposts on the pulpproject.org? Probably we will drop those and have only discourse release announcements
- need to fix deps requirements( upper bound) on older releases and main branch before 3.20 release
- decision on compatibility release?
- if 3.25 then it wil be 9-10 months roughly until next breaking change. 3.25 it is
- https://discourse.pulpproject.org/t/is-the-pulp-logo-available-under-creative-commons-cc0-cc-by-or-cc-by-sa-licenses/432
- our logo is unlicensed
- we need to open a ticket with legals - Jeffrey Kaufman is the legal who can help
- Streaming support needs a little more work around content ranges and caching; test will probably a new ci scenario with sftp storage
- triage security alerts?
- yes, add them next to the PR review ( from the next meeting)
April 19th
- test improvements update
- unittest switch to pytest is done on master and all branches back to 3.14
- plugins switching in-progress (thanks @fao!)
- moving the using_plugin tests to pulp_file
- next step: make pulpcore a pytest plugin so its fixtures can be versioned with it
- involves moving all pulpcore pytest fixtures from pulp-smash to pulpcore
- unittest switch to pytest is done on master and all branches back to 3.14
- 3.20 plugin API breaking changes
April 12th
- 3.19 release day
- No objections? Will wait for task lock PR and import/export PR to merge
- functional test improvements
- accomplished
- got pulp_file tests running along with the pulpcore tests
- improved pulp_file tests to be independant
- the –pulp-no-leftover-objs feature is ready for pulp-smash, still off by default
- next steps
- merge or revise –pulp-no-leftover-objs
- move all pulpcore.tests.functional.api.using_plugin tests to pulp_file
- porting tests:
- run in parallel
- native pytest
- use local fixtures (not fixtures.pulpproj.org)
- adding list endpoints for remote and alternate content sources
- accomplished
- dkliban is switching unittest runner to pytest
- https://github.com/pulp/pulpcore/pull/2498/ <- main branch
- https://github.com/dkliban/pulpcore/pull/63/ <– 3.14 branch
- https://github.com/pulp/plugin_template/pull/612
- Goal: to not have unittests require pulp-smash to be configured
- Solution: need to move conftest.py into the functional folder "down a level"
- Will have to do similarly for any plugin that already adopted conftest.py
- Should we stop i8ln log messages?
- Mark issues as "good first issue"
- Would make it much easier to hand off issues to interns / people who want to get started with development (e.g. Taft Sanders)
April 5th
- How many releases do we want to keep supporting?
- Currently 5, in a few weeks, it will be 6
- https://github.com/pulp/pulpcore/blob/main/template_config.yml#L35
- switch to lower bound which is 3.14
- We can't drop 3.14
- Should we come up with some kind of LTS policy or continue with as-needed unofficial LTS?
- we should also audit and establish an upper bound for our requirements.txt deps
- Currently 5, in a few weeks, it will be 6
- FYI here's a proposal to auto-check that tests are always independant
- 3.19 - last call, are there any PRs which need to merge today
- let's push back a week
- FYI - do not use gettext in log statements https://github.com/pulp/pulpcore/issues/2477
- read and provide feedback
March 29
-
filed issue to remove the queues in 3.20
-
Closing old/outdated/finished issues: https://github.com/pulp/pulpcore/labels/Finished%3F
- Went through and marked issues I thought could be closed
- 86 issues!
- Options:
- Have a meeting to go through some of the issues and discuss if they should be closed
- Have everyone look through some of the issues on their own and close them if they feel confident
-
Unittest update
- Added ability to run additional tests in plugin_template
- Using that to run the pulp_file tests
- https://github.com/pulp/pulpcore/pull/2432/files
- Fail in CI due to pulp_file tests not being a clean environment after pulpcore tests
- Goals:
- Test independance, run single tests, or in any order
- Cleanup all objects except orphans, a test that needs its orphans removed needs to run orphan cleanup at the beginning
- move the pulpcore.tests.functional.api.using_plugin tests to pulp_file.tests.functional.api
- to run the "pulpcore" tests you have to run pulp_file.tests.functional and pulpcore.tests.functional to be sure
- We thought better this than a formal import dependency on pulp_file
- switching unittest runner to pytest is also continuing
-
Do we want to do this eventually https://github.com/pulp/pulpcore/issues/1984? -YES
- used in pulp-ansible https://github.com/pulp/pulp_ansible/blob/main/pulp_ansible/app/tasks/collections.py#L398
- and similarly in content app https://github.com/pulp/pulpcore/blob/main/pulpcore/content/handler.py#L774
March 22
- memory PoC continued!
- 27% reduction in memory by reducing queue count from 1000 to 1
- should we remove the queues entirely?
- concern: deadlocking due first-stage waiting on the resolution of a future, or some even the pipeline is supposed to produce
- more of a concern now that the pipeline is even less filled
- 27% reduction in memory by reducing queue count from 1000 to 1
- proposal: removing django admin UI views
- bindings docs
- should we publish for both: ruby and python?
- https://github.com/pulp/plugin_template/pull/599
- let's ask katello
- mkdocs switch?
- should we?
- fabricio will show his PoC
- weird "db connection already closed" CI failure https://github.com/pulp/pulpcore/pull/2357#issuecomment-1072333934
- maybe issue with a test, @dkliban will take a look
- absolutely weird RemoteArtifactSaver failure in pulp_container
- Goal: Deduplicating tests between pulpcore and pulp_file
March 15th
- memory PoC concluded
- adjusting batch size is effective at reducing memory
- see the results
- dynamically adjusting it doesn't keep it within a "target amount"
- it runs extremely slow 10x plus
- memory is never returned to the OS, the pages mostly just stay allocated and later
- new memory charting debugging feature added:
- proposal: to expose the batch size to users, the batch size is likely the single best adjustment to reduce memory
- we need each plugin to also look at their own first-stage code to see if it can be memory optimized
- proposal: https://github.com/pulp/pulpcore/issues/2069#issuecomment-1066853598
- adjusting batch size is effective at reducing memory
- maintainability policy
- how far should we keep ensuring old branches are working?
- single container currently does latest 3 y-stream
- pulp-cli tests with 5 y-streams
March 8th
- memory PoC in-progress
- one feature tries to log the relative memory usage of declarative versus everything else
- adds a new setting TASK_MEMORY_LIMIT
- Attempts to reduce batching and put() calls at first stage
- pending openapi schema changes to improve usability
- update on pytest unittest issue
- we will need to release 3.15.5 once QE test the patches
- [daniel] will take care
March 1st, 2022
- Mirrorlist / Metalink support in downloaders
- Old issue, still a good idea: https://github.com/pulp/pulpcore/issues/1815
- https://github.com/pulp/pulp_rpm/issues/2386
- Consider implementing it in the 3.20 / 3.21 timeframe along with downloader refactor?
- (mirrorlists) are a current source of a lot of hacks in the RPM plugin w/ lack of native support
- Can we prioritize https://github.com/pulp/pulpcore/issues/2101 ?
- yes - ggainey has a branch already
- dkliban to keep ggainey on track :)
- https://github.com/pulp/pulpcore/pull/2261 needs review, merge and backports
- z-stream Releases for 3.17 && 3.18 maybe 3.16 needed.
- bmbouter switching unit test runner to use pytest and the django-pytest plugin
- will involve rewriting or removing 7 or so unit tests on pulpcore
- going into plugin_template as a non-optional change. will advertise at open floor
- An effort is underway to have a validating openAPI spec
February 22, 2022
- [ggainey] What does the team think about implementing automerge?
- 2 approvals, tests green, no merge conflicts == immediate/automatic merge
- implies "DO NOT approve a PR unless/until you're willing to see it merge immediately"
- [update 2022-03-01] automerge isn't nearly as 'auto' as we want, not nearly as interesting as a result
February 15, 2022
- rerooting update
- pulp-cli and pulpcore is working well
- pulp_installer to get some updates to its reverse proxy configs
- plan is to test in one of the jobs
- review periodic tasks discussion (aka TaskSchedule)
- Request to post periodic metrics
- https://github.com/pulp/pulpcore/pull/2208
- plugin removal
- blocked by CI weirdness around service names in ci container
- We fix the ci base container to forget all workarounds for the old tasking system
February 8, 2022
- periodic tasks request from galaxy_ng for them to run a galaxy_ng task periodically
- https://github.com/pulp/pulpcore/issues/2204
- [bmbouter] my proposal is to move the PeriodicTasks work to its own PR.
- proposal: plugin writer only feature, not user-facing
- AI: all discuss on issue and review next week
- plugin removal test
- as a separate job which runs similar to the upgrade test but only nightly?
- current PR which struggles to make it work as typical test
- Touch() bug fix - when can we get it tested?
- 3.18 schedule discourse thread
February 1, 2022
- Discuss memory use proposal https://discourse.pulpproject.org/t/reducing-memory-usage-during-sync/277
- Github issues process adjustment suggestions
- Changelog cherry-picking problem
- No longer using backport issues. Changelog commit removes files from the CHANGES directory. These files need to be present on 'main' branch until the changes are released in the latest release.
- We need to figure out Something - because this is the last, larget, manual step blocking releases
- Maybe each previous-branch-changelog is a separate file?
- How?!?
January 25th, 2022
- 3.18 release - Feb 8 (tentative)
- we have 2 features merged
- any blockers/must-haves
- redirect content guard
- plan to schedule go/no-go on Feb 1
- scheduled Feb 2 based on availability
- Need to provide the ability to reroot
- High-priority bugs that need fixing ASAP:
- touch() deadlock
- import/export unique-constraint failure
- taskreservedresource migration
- Should / Can we provide (general) manage command to help admins to migrate existing permission sets to existing roles?
- Idea: manage command to report "unmigrated" permissions and let the admin assign the roles via api
- Create the command in pulp_container codebase
- Downloader Refactor
- started out obvious
- grew out of scope
- we will slow role and think about separation of concerns again
- nothing urgent; please don't hold any other work for it
January 18th, 2022
- pulpcore is on github issues now! (BZs updated)
- we need to rethink triage, planning, Epics, Milestones …
- ggainey has been looking at github-projects to see if we can use it
- github projects - not quite rdy for primetime
- write scripts to take advantage of GH API?
- label-driven queries
- only need "current sprint" label
- PRs-merged/issues-closed within date-range
- epics are an epic pain right now
- maybe use projects?
- or milestones? (maybe not)
- Example query for issue-triage
- Discuss memory use proposal https://discourse.pulpproject.org/t/reducing-memory-usage-during-sync/277
- Want to look at basic PoC soon
- Downloader logic is convoluted (and broken to some extend)
- can we schedule the custom-file-object for removal in 3.20?
- [x9c4] file a PR and discuss improvements there
- pulpcore deprecation warns for the use of the old orphan interface
- It's a valid test that keeps the old interface running and it's going nowhere in the percievable universe
- we could try to distiguish rest and plugin api deprecations
- we don't warn the user about the use of the old interface
- https://github.com/pulp/pulpcore/pull/2100
[x9c4] remove the deprecation logging and make sure the deprecation is in the api.docs- [bmbouter] keep track of user facing logging
- pulpcore is getting parallel test running
January 11th, 2022
- Discuss memory use proposal https://discourse.pulpproject.org/t/reducing-memory-usage-during-sync/277
- Hard to decide without seeing some code (PoC is needed)
- Even some way to measure where what amount of memory is bound would be helpful
- should we protect remotes from being deleted if they are used in other repos? https://github.com/pulp/pulpcore/blob/main/pulpcore/app/models/repository.py#L64 Should we audit all FKs that use on_delete=SET_NULL?
- Deleting with SET_NULL can break stuff, open security holes, or just be annoying
- https://pulp.plan.io/issues/8305 is a special variant of this
- [ipanova] write an issue
- upgrades to the new tasking system
- It seems to be a good idea to clean up the tasks table (maybe releated tables too)
- [ggainey] reach out to katello
January 4th, 2022
- Rotate meeting lead
- github migration
- There is an API interaction limit; need to run in batches
- Bugzilla interaction is confusing ATM
- [dkliban] Talk to downstream to (re-)define Bugzilla interaction
- patchback - https://discourse.pulpproject.org/t/automating-backports/153/4
- Let's change the Commit validation to accept closed issues on cherry-picks
- https://tutorial.octomachinery.dev/en/latest/hall-of-fame.html
- open cherry-picks from the newest branch to the oldest needed
- to request a backport, leave a comment on a PR, tagging @to-be-decided.
- update on improving tests in pulpcore
- new pytest fixtures in pulp-smash merged
- new pulp_file fixtures in pulpcore merged
- working on switching unit tests to pytest
- any 3.18 plans or deadlines?
- no known driver
