https://hackmd.io/@pradyunsg/pycon2024-pack-summit
Slides: bit.ly/pycon-packaging-metadata-2024
I'm at Astral and work with uv, and been thinking a lot about this.
Been writing up a doc on how to group the into ways people use packaging into two big buckets
Carol's framing of producers and consumers: https://pysplash.github.io/dog-paddle/
My personal feel about this is that the story of how to get Python is completely decupled from how to get pPython packages
I think really focusing on the actors in this story is the important thing we need to spend time on
Couple of anecdotes here, I have a 15 yo daughter who wanted to learn games in Python. So I said I'd help her get her packages set up so she could do it. Took me an hour to get a working system and be able to use it from VSCode, and I have many many years of experiance in this, had to try three or four different ways before it worked.
If others saw Leah's talk earlier, it emphasizes the concept of having too many options and too many optinos, and its just too hard for overwhealmed new users
Having been around with pip for a while and then thrown into Conda, I saw firsthand how it was such a problem for users, where people were mostly focused on technical standards rather than on users
See doc linked above, but wondering if the consumer side of packaging needs to be addressed outside of PyPA and the tooling.
You hinted at two dimensions, environment activation and ???
Feel like there are two setspof problems
In my team have people who've never worked on Python before, there's a right of passage where people install WSL2, break your OS, reinstall everything, then install and use the correct Python, etc.
Want to be a little compassionate about how we got to where we are today, seen it over and over again other places. Ended up here because there were problems that needed to be solved at the time in the past, but the truth is the universe always changes so we're in a different place now.
Regarding the two Pythons, what happened when Apple took Python out of Mac? Did it help anything?
Going to suggest somethng that requires little to no development
I just want to concur with what was just said, the idea of coming back to one authoritative source of information, and I understand if the PyPA isn't ready to take that on, but someone needs to take that on. Would be happy to help with that
Q: Think this is great, just if we're adding more paid people want to make sure we also acknowlge the volunteers who've been doing this for many years
Want to comment on process of getting money from funders
A: For the PSF as US nonprofit, need to fund things that are on the PSF roadmap already and align with our mission, and needs to be ratified by the community
To have successful governence, we need to have all the major voices reprisented, pip world, conda world and the rest. I would love to see some solid user research, as decisions made in the past were the best decisions at those times with those needs. If we're going to redo things now, important to first listen to what the community needs and understand what they care about, and assume things about the user that might not be the case.
One thing that I think needs to get done is the governance of how money gets spend, whether its the Python Packaging SC or soemthing else, as right now if two people object to something it doens't get done
First, we have to decide how we decide before we can decide
Think that the user experiance needs to be decoupled from packaging
What is the reasoning we don't have a Linux installer on Python.org? Lack of someone stepping up to deal with it, or something on the PSF side that is not providing support, or is there something else?
If its a matter of connecting with the Linux community, we can pay to fly someone out here, or go to their events
Other issue is on Linux, you don't have one operating system but 20, and what we discussed today would break the policies of at least 10 of them becuase they don't want it to be outdated and seperate from the system
Give our new community developed tutorial - create a pure python package using hatch a test run!
Talked about this last year, though about making a PEP but someone suggested just making a library, so have something right here, not quite ready for production
We build a package resolver and installer called uv
--user
Want to give a quick demo of Conda-Store
Two quick things
Want share pixi, have a similar package manager project focused on the conda ecosystem
CI/CD environment and maintainer
[jaraco] Looking to get in to the packaging challengs and apply the lessons I've learned working on monorepos at enterprises
Two things
Carol: different eco-systems that need to be addressed
PyPI – where the packages live
A better story in supporting GPUs
Packaging tools – lots of them
The brand new Pythonista experience
Some thoughts
New languages are really easy for people, so more pressure to make Python easy
Seeing more convergence instead of divergence in software generally
Deb: The Python website should be great for newcomers
Phebe: We've invested a lot into the tooling But almost nothing in the documentation We should be giving that work respect, responsibility and funding
We need to cultivate that respect
Tania: everything is tool-centric.
Mike: Personas, "This is for you, newcomer" Could the PSF hire a full-time technical writer?
Carol: "pathways" beginners, and data scientists, different pathways
Mike: will they find what they need at Conda? (Carol thinks no)
Conda is a whole ecosystem
Pradyun: we need to set up the lay of the land
Phebe: frustrated with the lack of a clear communications channel We need both the legacy and the "board"
Rust's "This week in Rust" is an example of a great resource
Deb: PSF's bubble should be getting "sticky" for newcomers and casual users
Barry: Steering council has been bad at transparency Should be better soon
Deb: PSF can handle managing writers/community comms/community relations people
Jannis: "cabal" Let's not get too into technical writer idea, and not into anyone making decisions
Carol: Wants to add "folks coming from other languages" to the PSF's bubble The scope is something we need to decide
Phebe: to build trust, we need to make a long-term commitment instead of a lot of six-month experiments, facilitation role
Mike: It's been challenging to "pick a winner" We should be opinionated about where you start and still have a lot of things Can we all acknowledge that no solution will accommodate 100% of use cases? When we fight back on all the edge cases and it makes us stop moving forward
Barry: past the time to put our money on a horse… let's not stifle innovation "standard" path plus support for other options The experience of using the toolset Personas; actual beginners, part-timers, tool builders that support those scientists, (people doing glue work?)
Andy: newcomers is where I want to focus too, have things work for 1000 GPUs Small shops are being cut out of the pathways
Carol: we'll never find the "one tool to rule them all" pip up, pie up, or conda up, but we could make the commands a little more standardized
Pradyun: the king is the maintainer and we're talking about ways to take away thier control and people will be mad, like taking a teddy bear from a kid
Phebe: the packaging council would need to have the respect of teddy bear holders and making that communication piece centralized
Pradyun: the missing piece is "how to establish the authority" has to be someone everyone would listen to
Peter: when decisions are made without the full dimensionality of the problem. (five dimensions) My proposal is if we converge on a view of the problem My dimensions interfaces are not the same as packages or run-times, We have to start further towards the beginning "what the heck is a package?" have to treat C dependencies as a first class concern The build system is very important for this concern Exploding problem of low-skilled open source contributors, coming from Excel, etc There is no standard Python application, and that is different again from distributions
Carol: we have a responsibility to mitigate the pain, now
Mike: we split out the build front end and back end
Shauna: a lot of the things reflect the people and perspective we'd need on a steering council
except the user experience perspective.
We can get people to give us their teddy bears, if we make them feel respected and part of the whole picture
Barry: it has gotten a lot better, honor all the work that has gone before We can evolve, and get to a better place. Make sure the teddy bear owners have a voice in the bear's future. Governance, the model for that.. we have a steering council that delegates work to other entities, eg Typing council if we made a "packaging council"
Carol: the focus is on tech at the expense of the user Rust doing a great job,
Barry: Rust has been really good in many ways They've put their weight beyond Cargo
Andy: The response is two fingers, new topic is one finger
Pradyun: the typing model will not translate they are very much not independent, packaging is all interoperability CPython says This is how the implementation will look Packaging is always about use cases We need functional decision making and I think we need a council to make that happen Let's just take the first step
Jannis: it took years of effort to make Cargo user-friendly, with designers and researchers We should look at user-expereince/success/happiness no one will lose their face
Pradyun: Cargo is at the same level as the language in importance at Rust
Barry: The PSF should fund that user experience research
Phebe: The packaging council could have sub-committees on "operating systems" and "Peps" or "tools" and "user experience"
Peter: Telemetrics are not a nice to have, they are 100% necessary Let's do it non-creepy
Jeremiah: Is the SDK in there? Is there a place for environment managers?
Pradyun: responsibility of the steering council
Phebe: opinonated, but not opinionated. We need to make decisions timebound so we can actually decide things
Carol: my proposal: let's have the PSF create a user-success workgroup that can hire a person
Barry: Packaging Council
Recurring ideas about work the PSF could fund: Communication Technical documentation User experience research
Outcomes: Create a user success working group, ask Board members to flesh out and instantiate Not necessarily limited to Packaging (UPDATE: a draft charter is already in the works!)
Broad agreement about a packaging council that will probably need to set up some sub-committees to focus on things like operating systems, or tools
Should we do this
Symlinks are useful for shipping versioned binary libraries (e.g. libcudnn.so.9) with specific version info (e.g. libcudnn.so.9.1.1) and potentially editable installs.
What shouldn't we do
What will we do
LINKS
file? Could be in the RECORD
file. Preferable to simply stuffing POSIX-y symlinks into the wheel; e.g., would potentially allow for NTFS support, and portable wheels (e.g. how do you deal with multiple platforms? New platforms?)Unsure
Distribution A packages can have links targeting B packages iff it depends on B? Do we need this, and is it possible to do this performantly.
conda-pypi
(eradicates the need to move PyPI packages to conda-forge)See above for full summary
See above for full notes
See above for full notes, hidden in foldout