Pradyun Gedam
    • Create new note
    • Create a note from template
      • Sharing URL Link copied
      • /edit
      • View mode
        • Edit mode
        • View mode
        • Book mode
        • Slide mode
        Edit mode View mode Book mode Slide mode
      • Customize slides
      • Note Permission
      • Read
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • Write
        • Only me
        • Signed-in users
        • Everyone
        Only me Signed-in users Everyone
      • Engagement control Commenting, Suggest edit, Emoji Reply
    • Invite by email
      Invitee

      This note has no invitees

    • Publish Note

      Share your work with the world Congratulations! 🎉 Your note is out in the world Publish Note No publishing access yet

      Your note will be visible on your profile and discoverable by anyone.
      Your note is now live.
      This note is visible on your profile and discoverable online.
      Everyone on the web can find and read all notes of this public team.

      Your account was recently created. Publishing will be available soon, allowing you to share notes on your public page and in search results.

      Your team account was recently created. Publishing will be available soon, allowing you to share notes on your public page and in search results.

      Explore these features while you wait
      Complete general settings
      Bookmark and like published notes
      Write a few more notes
      Complete general settings
      Write a few more notes
      See published notes
      Unpublish note
      Please check the box to agree to the Community Guidelines.
      View profile
    • Commenting
      Permission
      Disabled Forbidden Owners Signed-in users Everyone
    • Enable
    • Permission
      • Forbidden
      • Owners
      • Signed-in users
      • Everyone
    • Suggest edit
      Permission
      Disabled Forbidden Owners Signed-in users Everyone
    • Enable
    • Permission
      • Forbidden
      • Owners
      • Signed-in users
    • Emoji Reply
    • Enable
    • Versions and GitHub Sync
    • Note settings
    • Note Insights New
    • Engagement control
    • Make a copy
    • Transfer ownership
    • Delete this note
    • Save as template
    • Insert from template
    • Import from
      • Dropbox
      • Google Drive
      • Gist
      • Clipboard
    • Export to
      • Dropbox
      • Google Drive
      • Gist
    • Download
      • Markdown
      • HTML
      • Raw HTML
Menu Note settings Note Insights Versions and GitHub Sync Sharing URL Create Help
Create Create new note Create a note from template
Menu
Options
Engagement control Make a copy Transfer ownership Delete this note
Import from
Dropbox Google Drive Gist Clipboard
Export to
Dropbox Google Drive Gist
Download
Markdown HTML Raw HTML
Back
Sharing URL Link copied
/edit
View mode
  • Edit mode
  • View mode
  • Book mode
  • Slide mode
Edit mode View mode Book mode Slide mode
Customize slides
Note Permission
Read
Only me
  • Only me
  • Signed-in users
  • Everyone
Only me Signed-in users Everyone
Write
Only me
  • Only me
  • Signed-in users
  • Everyone
Only me Signed-in users Everyone
Engagement control Commenting, Suggest edit, Emoji Reply
  • Invite by email
    Invitee

    This note has no invitees

  • Publish Note

    Share your work with the world Congratulations! 🎉 Your note is out in the world Publish Note No publishing access yet

    Your note will be visible on your profile and discoverable by anyone.
    Your note is now live.
    This note is visible on your profile and discoverable online.
    Everyone on the web can find and read all notes of this public team.

    Your account was recently created. Publishing will be available soon, allowing you to share notes on your public page and in search results.

    Your team account was recently created. Publishing will be available soon, allowing you to share notes on your public page and in search results.

    Explore these features while you wait
    Complete general settings
    Bookmark and like published notes
    Write a few more notes
    Complete general settings
    Write a few more notes
    See published notes
    Unpublish note
    Please check the box to agree to the Community Guidelines.
    View profile
    Engagement control
    Commenting
    Permission
    Disabled Forbidden Owners Signed-in users Everyone
    Enable
    Permission
    • Forbidden
    • Owners
    • Signed-in users
    • Everyone
    Suggest edit
    Permission
    Disabled Forbidden Owners Signed-in users Everyone
    Enable
    Permission
    • Forbidden
    • Owners
    • Signed-in users
    Emoji Reply
    Enable
    Import from Dropbox Google Drive Gist Clipboard
       Owned this note    Owned this note      
    Published Linked with GitHub
    1
    • Any changes
      Be notified of any changes
    • Mention me
      Be notified of mention me
    • Unsubscribe
    # [Packaging Summit at PyCon US 2025](https://us.pycon.org/2025/events/packaging-summit/) https://hackmd.io/@pradyunsg/pycon2025-packaging-summit --- ## PEP 772 - Packaging Council governance (Barry Warsaw) [Slides](https://docs.google.com/presentation/d/1H1n05s1dw84DKgmNAUn2bdXxFmFyalF8wNJ0HdDRJN0/edit?usp=sharing) * [Packaging governance process: PEP 772](https://peps.python.org/pep-0772/) * See slides for more links * Motivation * Builds on [PEP 609](https://peps.python.org/pep-0609/) * Packaging community at a point where some amount of formalized governance is needed * Avoid bus factor of one * Advocate for critical importance of packaging * Bring in more diverse opinions * Work with PyPI admins and tool developers to identify challenges and opportunities * Structure * Split cohort elections to bring more continuity * Balance room for innovation with forging interoperability * Current standing delegations and packaging workgroup authority would be transferred to packaging council * Relationship with PyPA: TBD, would not be explictly specified at first in the PEP * Voting * Initial members: PyPA, packaging workgroup, CPython core devs, major organization reps * Orgs was the biggest gray area * Initially 7 members per org, now 3 for more balance * Want to make sure we capture a wide and reprisentative net of perspectives and stakeholders * How to review initial membership * SC? * Feedback welcome! * (Deb) PSF interest is in collecting consensus from packaging community * Goal is to get community's objectives done once consensus is built * Can't just ask one or a small number of people ### Discussion - Open questions - Communications - How else can we reach out to everyone? - How can we be transparent and communicate with the community? - Maybe share resources with the SC which has the same issue - Sync up with SC and other stakeholders - (Bernat) SC is a member of the community, only contentious question is how to select initial members - First three groups are not controversal, the fourth (community orgs) could be - For first three coulds could self-nominate, the fourth members could be nominated and the first three groups vote on them? - (Barry) So sort of a two-step bootstrap - Maybe could nominate individual people rather than just orgs - E.g. someone doing packaging tutorials - (Carol) Push back a little on last bullet point - One of the things lacking is input from packaging consumers - Where a lot of the frustration has happened - Many community members involved in making things better - E.g. PyOpenSci - By not having initial orgs be part of slate to vote in makes the first three insider parties the gatekeeper of the fourth, community orgs - (Bernat) Anyone can propose, rest of the members can vote - (Carol) Still not 100% on this since the fourth group also includes packaging tool authors like the first three - (Barry) Well most of those people will also be PyPA members? - (???) Poetry and uv are widely used but not PyPA members or part of other groups - (???) We have general trust in our leaders now, we should get started moving right away and then rely on next election for accountability - Lots more time to bring people in when we are underway - (Carol) Guess I'm not entirely clear on the seeding process. Can you highlight what happenes when? - (Barry) That's kinda why were here - How do we make sure we're bootstrapping right? - That process is something we've all been struggling with - Looking for a way to move forward - (Zanie) Perhaps my points are bit stale but feel the claim that it is not controversial to exclude community members is not true - Many projects decided not to put their projects under the PyPA - Think the SC is sufficient to make the decisions to approve who gets in - (Barry) I don't think its uncontroviersla that PyPA, etc. is sufficient, but rather who to include beyond that - I believe current draft of the PEP gives the SC authority to filter the orgs that fall into the wider community members - Then orgs nominate their own members - (Geoffrey) Packaging council doens't have to be members, more important than the voters - If we're confident that it is important to reprisent community on Packaging Council, more important than just having community members as voters - Sounds like people here sympathetic to that goal - Want to make sure ethat happens - (Deb) SC makes sense to me but want to push back - Don't think month is long enough to conduct this process - Not everyone is plugged in/on Discourse - Three month nomination period with commitmenet from PSF to do outreach to get more nominees - (Carol) Going back a decade that was when the pip vs conda split happened - So much "you're not part of this communiyt, you have different needs" - As someone who worked on Jupyter it was a nighmare to help people in stall things - Need to think about the users more - Has improved but wider community voices are just as important as the other three - We have been living with it - We were the voices not listened to enough earlier - (Phebe) Because ecosystem so fractured rn, if I understand proposal correctly there's PyPA, pseudo-PyPA (conda, brew, uv), adjcacent tools, projects within those ecosystems different sizes, Linux community/distros - When it comes to accurately reprisenting diaspora, need to be able to reach them, inform them that they are voting members - What is the plan to do that? - (Barry) Can speak to intent - Ensure that Conda community is well reprisented re:frustaration that Carol expressed - And all the downstream consumers for Linux distros, Windows, Brew, etc., all important voices to bring into this conversation of evolving packaging - (Phebe) And that was jsut all the open source communities, then have all the for-profit companies with internal stuff, artifactory, etc. - (Barry) Yeah that's what we're asking cause those are super important voices in how packaging looks 5-10 years from now - (Phebe) And we're thinking this can be reprisented by a 5-person, 2 term commitee? - Defintely a start but going to be an enourmous communication problem - (Pradyun) That's one of the open questions - (???) Things have gotten better but also way more fragmeneted - Leaving out all the people who are driving forward on innovation as well as community adoption - Should be inviting people from those groups - Talked to those groups, felt like it was very hard to engage - (Bernat) Had nomination process in PyPA for past 5 years, wasn't a single vote that didn't pass unanimously - (Carol) I wanted to join PyPA but I wasn't allowed in since I didn't maintain a packaging tool - (Bernat) Yeah but we are expanding the group here, I don't think if anyone nominates someone from poetry or uv that they would say no - (Carol) My point was more governance, have respect for first three groups but also fourth group have. - Don't want to dilute voices of first three groups but thinking maybe three people from fourth group would potentially skew membership - (Barry) Want to clarify, membership for voting in packaging council is individual, not organization - Not like "Hatch is reprisented", people going to come in and out of projects all the time - Like in CPython, people change jobs but does not relate to organization people are affliated with - Intend to move from PyPA model that is project based to just making that the bootstrapping and then making it individual from there - (Carol) So why would we want to initially exclude the initial group from the fourth group - (Jonathan H) I'm worried about the number of votes people initially have, e.g. PyPA could probably vote whoever they wanted - (Pradyun) As an author, there are groups we've engaged in this process that are much larger than the groups initially involved - Shifting from a model of entities to individuals - So if company A has a reprisentative and they fire them, then individual still has membership - Trusting the individual will stay engaged after leaving - (Travis Oliphant) Thanks to Barry for doing this - In centralized governance over a very wide bredth of people, need to have humiliyt and trnasparency - Need to make sure we're not pushing things - E.g. Everyone in the room agrees, well its decided - May not have everyone in the room - Overcommunicate is always better - E.g. I think I've already communicated things but only reached 10% of who I wanted to - Let's get the PSF involved and do a big outreach campaign - Over the years many conversations I wanted to jump in on but didn't know about - Think its true for a lot of people - Have to recognize there have been harm created in the past - I've done it too - Can celebrate activities but recognize we need to do better - (Barry) Thanks, communication works both ways - All of the folks in other slide want to make sure voices and feedback coming back into packaging community is also heard - Ears are open as well as our mouths - SPeaking for myself don't have clear picture of entire eocystem and what we need to do - Ultimately we are all proxiesis for our users and need to have compassion for them - They are the ones struggling and we want to distill that into solutions - (Pradyun) Out of time, just want to say we are being slow and deliverate about this and don't want to rush intentionally - Wnat to make sure we can leverage all comm channels - Summit is one of them - (Barry) We all appreciate your passion and dedication to making this beter - Thank yourselves, not just us - (Geoffrey) Where do you want feedback - In breakout sessions - New thread created on DPO - Put in the notes document [will create header for it] ### Community feedback on packaging governance - _(Name) Feedback_ - (Geoffrey) We need to phrase the roles and goals more clearly than the PEP for advertising to the broader community - AIUI, the membership has no role other than voting, so unless there's anyone you specifically want to vote for or against this doesn't communicate "you're (not) a member of the packaging community", and AIUI, the council has an oversight role (roughly BDFL-delegate?) and a PSF advisory role, not a direct design/architecture role. If this understanding is correct, I really think people should not be under the impression that having a vote makes you a "member of the packaging community" or not, or that you need representation on the packaging council to make things happen (or that representation is _sufficient_ to make stuff happen.) - What would go wrong if we let literally everyone who wanted to be a voting member be a voting member, no restrictions other than signing up (and maybe some amount of making sure you're a real person)? The only thing I can think of is a big company telling all their employees to signup and vote - but we already restrict too many actual council members from the same company. There is the ability to step in if something goes horribly wrong (e.g. Steering Council refuses to make the delegation), can we just open up voting? - (Geoffrey) Can we run banners on PyPI? Can we run text banners on build tools like `python -m build` or Twine? - (Leah//pyOpenSci) It seems to me that any new process here will be challenging and tricky but the goals and outcomes are important. Couldn't we spend a year testing the new process with extra attention placed on transparency around who / which community members / organizations are represented in the first round and open "documentation/ notes" around why someone might not be icluded. Then after a year, we could run a surveyto better understand how things went and how the community feels about this process. I think a huge win is increased transparency around process and also communication around what's new is the most important thing here. - (Andy Terrel) I would like to see representation from library writers. Many of the specifications that are being proposed will cause library maintainers more work. Not having people who have built and maintained popular Python libraries would be a miss. +1 ## Developing a Python packaging ecosystem for Android and iOS (Malcolm Smith) [Slides](https://docs.google.com/presentation/d/1lNaCNlgpVovEUR2L_8sLV6bJu0ibdWifvnwjBqUi7hQ/) * [Adding iOS as a supported platform PEP 730](https://peps.python.org/pep-0730/) * [Adding Android as a supported platform PEP 738](https://peps.python.org/pep-0738/) * Feel we need to enhance Python support on mobile platforms since they are such an important part of many people's lives * Status * First supported as sdists without official binaries * Now supporting on PyPI as well as official binaries, tier 2 support coming * cibuildwheel * Has become de-facto tool for building wheels for many different platforms * Hope to retire custom beware tools and built that into cibuildwheel * First challenge: Cross-compilation * Focusing on mobile but general issues with cross-compilation * Also relevant for wasm and emscriptem * Quick summary of cross compliation * Two kinds of platforms: build platform and host platform * More more details see [PEP 720](https://peps.python.org/720) * Main element is setting a bunch of configuration flags * In CPython monkeypatch APIs that depend on the platform * Update sysconfig data to reflect host platform * End result: Python actually running on build platform tricked into believing its running on the host platform * Works most of the time but always corner cases where you do want it to think it is on the buld platform * Want a single source of truth for data relevant to the host platform whether we are cross compiling or not * Second challenge: requirements * Pure Python packages its streightforward * Build tools like CMake need to run on the build platform * But if elements that need to access things on the host platform no way to do that currently * E.g. need access to NumPy headers for the host platform but if install different ones for the build platform will cause incompatibility * Non-Python libraries * Many packages will incorperate the build of the non-Python library into it * Others simiply assume they exist on the host which may not be the case * How do we host all of these non Phython packages? * I suggest Conda-Forge * Already designed for non-Python libraries * Would only include non-Python pacakges as Python ones would still be compiled into wheels * Wnat to see if people think it is feasible to do this on Conda forge * Second question: What syntax do we use to specify build dependencies? * [PEP 725] * Example from cryptography in slides * Goes beyond just cross-compiliation as it would help things with Conda and downstream Linux distros [PEP 725]: https://peps.python.org/pep-0725/ ### Discussion - (Jonathan H) Conda-Forge could add another platform, have added in the past - Usually what is needed is a reliable CI system that has the platform - Conda-Build has very rudimentary support for building wheels - Required to build Conda package for Python - Could we use Conda-Build instead of CIbuildwheel to output wheels as well as Conda packages - Not going to be PEP 517 build backend - (Jannis) Why not? - (Malcom) That was what we proposed two years ago but cibuildwheel has grown greatly since then so we focused on that - Re:reliable CI platform, prettty good situation as we've been able to run Android and iOS emulators on GitHub Actions - Slower than running natively but high level of relaibility - Packages building with cibuildwheel will get full test coverage - (Cheng) So now that we have the wheel, how do we get it to the users without running afoul of app store requirements - (Malcom) Wheel hosted on PyPI, developer at the end creating an app will use a seperate tool responsible for dealing with app store requirements, signing, etc. - (Hood) Think what is needed is not a build-numpy but a cross-numpy that runs on build system but gives information about the targe - Way I get my corss numpy is stiching notgether manylinuxnumpy and downloaded target numpy - Think its more complicated than just here are the host requirements and here are the build requirements - (Malcom) Difficulty with Numpy is you need to import the whole thing but just need to get the headers - (Hood) Maybe numpy can be improved for this, we also have patches for pyiodide that do a similar thing with sitching things together - (Malcom) What do you mean by sitching - (Hood) Copy files between target and build - (Russel) Piece of metadata here that needs to be captured, need to come up with a better way - Coming back to other questoin, there is a build step needed with Xcode difficult to do with wheel format but where wheel needs to be binary is not allowed to be, can be worked around by putting wheel where binary needs to be - (Emma) Is this information availbile statically or does user need to know every detail, permissions, etc. of the libraries the yare using? - (Malcom) What details you mean? - (???) E.g. on iOS need to know what permissions libraries need - (Malcom) Responsibility of the app developer, beeware tools and other tools have configurations to specify whether you need access to camera, geolocation, etc. - If your app includes such things need to enable it at the app level - (Emma) Regarding Numpy and crossbuild, want to talk about libraries during unconfrence and finding native library files - (Pradyun) Will have times after the talk to pitch topics - (Geoffrey) Did talk at wheelnext yesterday about picking up [PEP 725] and doing something about it - Also talked about need to figure out better stories for libraries on PyPI on desktop platforms too - What to do about libraries on mobile platforms, should pay attention to that too - How to describe cross dependencies, recommend looking at how Debian multi-arch does things - (Jonathan H) I was curious on non-python build dependencies, do those eventually get vendored into the wheel - (Malcom) Exactly - So the storage on PyPI or CF the storage is only temporary storage for those? - (Malcom) Yeah - So app developers will be pulling them down and including them in their app - (Malcom) Yeah - So you could just cache these? - (Malcom) Non-Python library doesn't need to be part of Python library - (Russel) So there's an overlap there, e.g. every project needs to be able to compile LibPNG, so instead of doing it everyone themselves would be great to have somewhere to get it from - (Henry) Want to make sure people don't get it from homebrew where it is not redistribtable - Debain package might be better example - (Henry) Yes but there's a special compiler environment, special symbols, older versions, etc. so something like this could be helpful for everyone - (Pradyun) How much of this is general rather than specific to mobile? - (Malcom) Mentioned corss-compilation, WASM platforms facing same issues - PEP 725 has gotten a lot of views outside cross compilatoin and even ordinary compilation - (Jonathan H) I will say vendoring into wheels using CF packages doesn't have the greatest history - E.g. Pyarrow used to do this, maintainrs stopped providing dynamic library and the entire build system broke - If you go down that route make sure what the guartees of Conda-Forge are - (Malcom) Good news is Android and iOS have a very defined list of what libraries are availible so more like macos/Windows and less like Linux - (Eric) Also relelvant when any new CPU chip comes out, e.g. new mac M1 needed cross compilation to boostrap - And also platforms that aren't new but GH Actions doesn't have runners for, e.g. done a lot of cross compilation in Python build standalone for this ## Wheel Variants - Finer grained operational environment compatibility (Jonathan Dekhtiar) [Slides](https://wheelnext.dev/assets/slidedecks/pycon/2025/PyCon_2025_-_Packaging_Summit_-_Wheel_Variants.pdf) * [Wheelnext](https://wheelnext.dev/) * Interest group/working group (Barry likes to call it accelerator) re:problems around packaging wheels * Especially in scientific and AI space * Reprisents people in a lot of different projects * Have contributed to what I'm presenting today * Why varients? * When we pip install package, very limited in what the computer installing knows about CPU, GPU, other hardware * E.g. ARM not enough, ARMv7, ARMv8a, etc. * Same happens with libraries, e.g. with Numpy * Was it compiled with BLAS, ATLAS, etc. * Same with open MP * We have had this problem over and over in this space * Need to come up with language with this * Best we have done today * Custom indicies and hacks * Wheel varients says let's take wheel tags and go a step beyond that * Example * Speed up Numpy for my specific CPU and GPU and drivers * Wnat to use specific MPI optimized for my CPU * What is wheel varients? * Almost semantic PEP * Building a language * Building a description ability to filter and sort * Namespace, descriptor, value * Examples in slide for arch and for Nvidia GPUs * Describing a protocol * Using typing protocol * Want to be fast * uv has shown this important * Steps * Filter varients * Then sort to find the best for a specific machine * Demo * Let's go deeper into UX * First, add metadata in pyproject.toml * Reason we have varients is because we have tags * Already have many * If we don't want to grow this expoentially need to create a system of arbitrary markers * Don't want to put burden on maintainers of packaging tools * Just need them to be externally defined * Thus the plugin UI we are working on * How does it look in wheel * Just changing a few things and injecting a few things in WHEEL metadata * Varient name is key into hash table and also unique id to ensure wheel has a different name * Can find the markers of the different things the wheel was compiled for * Hashed together to give an id for the wheel * As of today you can test this * Many tools listed in slides can already do this * Nothing is set in stone but trying to build something people can test hands on * Building packages * Varient with flit * flit wheel * Pass a bunch of flags * Boom, a varient * Meson more complicated as building actually code * Rspec links varient information to compilation flags to pass to tooling * Bridge that link with meson-python * Make varient tool * Take a normal wheel * Edit and make it a varient * Can already build, test, deploy varient wheels * Install side * Don't want to change anything * E.g. trying to install numpy on arm8.3 * Download aarch plugin which would analyze my platform * Best varient availible is 8.3 * Pick it up and install it * Same exact thing with x86064 * Download x86-64 plugin * Scan varients availble * Find the best * Install * Pytorch * Download nvidia plugin * Scan varients * Select best one * Install * Laptop without GPU * Download nividia plugin * Scan varients * Determine none matches * Fall back to CPU * Don't care about varients * pip --no-varient * Certain varient has a bug * Exclude or select varient by hash * Please come to sprint on Monday and Tuesday to try it out! ### Discussion - (Russell) Wondering about xcompilation story - Plugins running on installer platforms - Need cross platform environment to install platforms for the environment you're targetting - Need to pass int pip --platform to pass in appropriate detaisl - (Jonathan) That's an open question., e.g. in nvidia plugin have a way to force a specific varient - (???) Varient info just in the wheel metadata? - (Jonathan) It was but now summarized in JSON file - Agreagates all varient information - Allows speeding up resolution as it can download a single file from the index per package - (???) Notcied plugins where installed in current environment, is that nessesary? - (Jonathan) No, working with pypa build to do that in an isolated environemnt, was a few bugs which we're fixing - (Pradyun) Can we have this info in the index living on a per-file level - (Jonathan) Probably, need to have a way to get that in htere - (Pradyun) Have a JSON reprisentation so should be able to put the list of strings in there - (Tim) Following up on plugins, plugins are just normal Python packages, any requirements? - (Jonathan) yes, no requriement on anything - (Tim) You're created an ACE system, any requirement that would restrict people from providing a more specific plugin that would be malicious - (Jonathan) We already have ACE already - (Everyone) No we don't, not with wheels - Today to hack numpy you have to hack either numpy or setuptools. These are creating high value targets that would allow hacking everyone all at once - (Jonathan) Problem is to analayze the paltfomr you have to do this to analyze the paltform - (Tim) Person controlling nvidia plugin could release compromised version that could compromise all users of Numpy and everyone else - (???) Would recommend pinning - Not enough - (Emma) Cannot trust PyPI at the end of the day - Correct thing to do is use signatures and verify them instead of just assuming safety - (Jonathan H) Feel like these plugins need to have the same level of trust as build backends - Need a blessed list? - We don't bless build backends - We do bless mypy types - Yeah but not build packends - Yeah but they don't run at install types - (Tim) Executing arbitrary code at solve time is a regression from what we have right now - I'm more worried about what this does for reproducible instgalling - If we have lockfile in uv don't need to perform any additional code execution to install from that lockfile - With this proposal any installation will require executing a bunch of Python plugins - (Jonathan) As a mitigation, we used to have it requiring the plugins to be installed manually - Objection was that no one would pick up the plugins manually - Creates a chicken or the egg scenario - Does it have to be Python? Just inside a wheel, could use sandboxed Java or something like that - (Barry) Maybe if we analyze the platform in one step prior to resolution and install time rather than per package? - Basically what plugins are doing is figuring out exact uarch and gpu - Why can't we do that once and capture the information - (Emma) Going back to security and resolution, wheels are not always secure - Pip doesn't guarentee that installing a wheel won't cause code execution - Bug report that package can break into one of pip's dependency folders and cause code execution - Fixed but Paul Moore was adament that it was not aguarentee - Do expect these things that the user has to trust like the build system - If you don't want the potential for plugins can disable them - Can pin to download a specific varient - Cna make that not require plugins - (Geoffrey) Plugin system more of a mitigation of we don't know what varients people wnat - If we had a process of definiging varients could build a secure system - Overwhealming feedback was they didn't want pip modified - If security is a top line requiement, lets add it and then resolve how we're going to secure it - (Pradyun) Wnat to push back on multiple fronts - Don't think you can write arbitrary wheels is equivalent to running arbitrary code when installing things - You can mitigate the former and can check wheels for it - Unlike ACE - This is a new thing and need to cater to its risks - Think that having a frozen list of varients and feeding that to the resolver resolves that concerns - Think that the security story is willy nilly is not the case - If you have a secure build pipeline then there is no ACE - You are running build backends, once you know what you're building there is a defined list of them - We area dding another layer here that is a change in the security model - The concern isn't we want to change pip, rather we don't want to change pip every single time there's a varient that we need - We want a generic systme, not one that is too locked down - (Travis) Want to step back, should we be encouraging people to put more binaries on PyPI - PyPI has grown to accomadate what multiple organizations and ecosystems are doing - Displaces the ability for those to help their users - Just because we can do this, should we be doing this on PyPI - (Jonathan) Doesn't have to be on PyPI, can be any index - (Travis) A lot of people who ship Python a lot of differnet ways - We cotemplated varients years ago, you said varients could be used for OpenBLAS but that's a dependency problem not a varient problem - (Geoffrey) I think there isn't any packaging system that isn't pip that consumes wheels - This is in the vien of manylinux tags, if you're shipping for Debian you aren't using manylinux wheels - Only realized recently that there's a whole bunch of parts here that don't need to be used together - This is a way to define wheels and how to get them, not where to get them - Ultimate motivation is that you want pip install whatever to work and not start doing a 24 hour C compile - Can use language of varients identpently from how to resolve them and how to discover them - For example if rust based package manager decide they can get new versions in people's hands and choose to implement plugins on our own as they show up, that is compatible with this spec - (Joanthan) That's important point, we want to establish mechanism and language not dicate usage - Barry was joking that one day it will be useful for computing, we just don't know yet --- ## Quick plugs and callouts ### Packaging survey Link: [Python Packaging Survey](https://anaconda.surveymonkey.com/r/py-package-2025) - (Ee) PSF and a group from various packaging orgs supported by Anaconda are promoting this packaging survey, fill it out yourself and don't stop there, make sure as many peole as possible fill it out - Would like to recieve emails in 3 weeks with feedback on the survey itself for next year - Want to do it annually ## Unconference ### (Partially) dynamic metadata - Henry [Slides](https://docs.google.com/presentation/d/1h_tv54xmkBNAgBaW-lMtLjD7LPCr3jML0BE_4Me_1do/edit?usp=sharing) * Partially dynamic metadata * PEP being prepared * Project, dependencies, want build backend to build wheels and link against an exact version of torch * When its built want to pin version of Torch in the wheel * But sdist supports older version of Torch * Right now can't have dynamic dependencies and static dependencies * Either fully static or fully dynamic * This PEP would relax that description so you'd have additive metadata * If something is staticly and dynamically listed it would be the static + dynamic * Special case for license metadata * Use cases: Licenses, SBOMs, generated entrypoints and classifiers * Draft PEP not published yet * Dynamic metadata package * Mentioned 2 years ago packaging summit * Brett suggested it be a package * So now it is! * Based on implementation in scikit-build-core * Want to provide it as something other backends could use * Want to talk to other backend authors * Two proposed APIs * Original design: everything's a table/dictionary, specify one plugin per value * E.g. version provided by version plugin, optional deps by optional deps plugin * Designed to become a PEP: project.metadata, become a table * Other alternative, doing it as a list * Would allow specifying the smae field multiple times * Does affect backend API * Need to pick one now * Feedback * At a table here * Also a GitHub repo scikit-build/dynamic-metadata * PyPI: dynamic-metadata * Comments on pre-PEP stuff comment on repo or Python Discourse ### Beyond [PEP 639](https://peps.python.org/pep-0639/) - C.A.M. Gerlach [Slides](https://docs.google.com/presentation/d/1KHPTJW68juimKSxsKAQ0wgiirBUlCO3pZA1x0JzIZAk/edit?usp=sharing) * PEP 639 license metadata * Implemented and supported in most backends/frontends * Issues knowning if license is for project vs distribution * Clarify current spec * License is for distribution * Followup PEP * specify project, sdist, and wheel license in table * Allow dynamic additions * License information shown on PyPI * Open Questions * Feedback at tables ### Improving native library packaging - Emma Smith [Website](https://wheelnext.dev) * Wanted to discuss further bits about wheelnext proposals * On Wheelnext site lists the various ongoing proposals * Jonathan with Wheelnext proposals * Default extras - PEP 771 * Build isolation passthrough * Extrnal wheel hosting - deferered * Want to revist how people use third party indicies * Two connected propsals for discussions * How to reinvent the wheel and Symlinks * Motivated by how to ship native libraries in wheels * Maybe that shouldn't be waht we're doing * But if it is how we can make that better * How to find native libraries * Currently have to import packages * Maybe we can put this in package metadata * Upload API 2.0 for Python package indicies * Resurrented an old idea for staged releases * Index priority * If hosting a wheel in different places, how to make sure I get the right wheel * PEP for that * Discuss on DPO and talk to us about it * (Barry) Wheelnext is an incubator for improving the wheel experiance * Probably other things you're thinking about that could go under this umbrella, so come and participate * Website, Python Discourse, Discord, etc. * (Jonathan) Will leave stickers for those wanting to participate in wheel next ### PEP 783 Enscriptem Packaging - Hood [Slides]() - [PEP 783](https://peps.python.org/783) - Enscriptem: Python for the web - Any platform that has Chrome [vendor lock in much? -Ed] can run it - eg: French students - Only known way to do sandboxed Python execution - CPython Tier 3 - Build ~260 Python Packages built, distributed via jsdelivr - Want to upload them to PyPI instead of jsdeliver - Various CPython features that don't exist in wasm implementations - First do old hacks - Eventually web-assembly spec involves to formalize it - Someone needs to define ABI for the target - CPython not currently distributing binaries - But positive feedback wanting to do that in the future - In a few years move to Python distributed binary and make that the source of truth for what the platform ABI is - Therefore want this PEP to go forward and have the platform defintion for PyPI be in the pyiodide documentation - Pyiodide says you will have an ABI per Python version - Unless someone goes and build Python 3.13 with the ABI for Python 3.12 - Can validate the wheel has the right ABI by just loaindg it since loader is sandboxed - Can also do introspection like what auditwheel does - Load node on the machine and load the wheel - Customization - Use build front-ends instead of back-ends - Since there are way more backends than frontends - Sets up the build and the cross-build, works with all backends we know about, so we can build a lot of stuff - Most packages if they do compute we can build them unless they have undefined behavior on common platforms - Installers - Pyidode tooling can use pip as a cross installers - Not sure if worth upstreaming this patches to pip - But uv we have a branch that can install for Pyiodide that is quite easy - Since uv is setup to be a cross installers - Whole thing is in rust, queries target platform once and gets the platform defintion - Modest set of changes, no clear blcokers, support from Astral team - PR is up: https://github.com/astral-sh/uv/pull/12731 - Testing - Can make a pyiodide venv - Install requirements for pip - Install package - It just works - Can do it in GitHub actions - Repalce python -m build with pyiodide build - Mostly works okay - How to move forward with PEP - Put it up - Back and forth with Paul [Moore] - No clear blockers but not sure what to do - Please share opinions or concerns ### PyOpenSci Update - Leah Wasser [Website](https://pyopensci.org) - Coming to the summit for the third time, pretty awesome to see what's going on with the packaging council, huge improvmenet, super exciting - Work with a lot of maintainers, scienctiest, people to find and contribute to better software and resuable code - All about beginner friendly in a technical landscape - Beginner friendly packaging guide - Translating it - jp done - es underway - Sprint Monday and Tuesday morning to furthter this - Open Space Sunday at 2 pm - Want to hear from you what we should includ to make things more accessable - Set of beginning to end Python package tutorials - Also created copier template ### Shipping own glibc as part of Python-Build-Standalone - ??? - Want feedback on this - Python-Build-Standalone - Version of Python from pipx, uv, hatch, poetry - Self-contained python in your home directory - Why ship glibc - People with old glibc - People on alpine who don't have it - A lot of questions on how glibc handles native library loading - Auditwheel is a workaround for this - Feel like you should tell program where to look for wheels but can't - Since anything you get in glibc tkaes 5 years to get to users, no feedback cycle, bad for users - Want to make this possible to fix things in glibc and get them in the hands of users - Want to hear from users - E.g. $BigCorp is scketched out by this more than they are sketched out already by us shipping our own OpenSSL ### Splitbrain: isolated depencency trees - ??? * Use subinterpreter to load seperate depencency trees in a subprocess * Used uv to do this * Two problems here * Subinterepters not very convienet yet * Want to mention the namespacing issue in the packaging working group * If we don't have a structure for how packages load native modules can't go further with this idea * When we have varient wheels, should expect that size of wheel packages will increase due to many combinations of wheel varients * Individual wheel size will decrease but overall size will increase due to shared parts in files * Think we can introduce a concept of layered filesystems using containers and others * Rough idea but wanted to spark discussion --- ## Breakout discussion notes ### Governance - Splitting the groups, similar to the Rust foundation - For-profit companies - It's going to be difficult to define the groups. - Memberships are also fluid. - Idea: why can't everyone cast a vote for the packaging council - Might want to limit for profit companies in this process so they don't send 5000 people to vote... - psf membership - allows someone to vote on the board. There is a payment for membership but also a free membership for non profit / open source - now - we're talking about bootstrapping. once we get the system going, anyone can participate - How about we couple voting with PSF membership? - Some folks like with this idea. - Some folks don't like this idea. - PSF (board like voting) -- this could potentially cause PSF resources that aren't available as the voting annually would consume resources - Could we align steering council, xxx and packaging councile voting - maybe it would be easier for Ee. - Tania: it does take considerable staff time to run the vote - would need to check with deb about resources. - there are annual psf board elections - pradyun: does packaging really need such an extensive voting process? - Such a vote process could enhance psf membership? - Carol: aren't we talking about seeding the council - pradyun: long term model - seeding from interest groups to allowing anyone to "run" - Barry: part of the responsibiliyt of the council is to evaluate - are we hearing from the right stakeholders? the council could then work to engage groups whose voices are missing from the space - A lot of interest in this area - has been driven by pytorch... <something about a gap in python users who don't associate / engage with PSF> - one challenge: it's really hard to wade through pages of discourse threads and such - there are so many niches groups that may not see themselves a part of the psf ... - when we commnhicate with the potential voters we need to think about why might be interested / why they should get involved / care about this. - Tania: PSF membership is broad. It's very easy for elections to become popularity contest rather than the people who are best fit for the positions - shauna - messaging might help with this - Could avoid the popularity item byt allowing anyone to be nominated but maybe there is a process that would allow some core group to decide if the person is a good fit / qualified or not. The challenge here is who will make that judgement around who is qualified. - Tania: last year maybe 24 people ran for 3 seats -- (aligned with that popularity contest) - Jannis: there is some bias in the PSF membership - is that a problem. - Deb's take on this: talk to Ee about rolling things in with the steerinc council. the board electsion - run by deb and marie - would need 3 months to talk to everyone to let them know this is happening - one thing that could happen -- you will get a different set of voters - could be good to get new members for psf and more funding associated with the membership $$$ - For this year we'd need the pep to be passed in 3 weeks - shauna: would it be possible to do an initial vote without the psf running it - barry: could structure initial voting to not be tited to the psf board in year 1 - but then maybe align the following year. - pradyun: this can't happen in 2 weeks - Jannis: is worried about psf membership not being diverse enough which leads to bias in the votes (psf members is not globally represented). is the membership body too biased -- is that really better then an open vote? (white north american bias??) - to have credibility in packaging world, you probably need 5-10 years of communitey engagement and that subset bby itself it will be less diverse. just because of that - deb: we cant fix overrepresentation of white males in tech. but we can potentially account for ... <missed this>m - deb: the voting body in membership might not translate to the voting community that we want for the packaging council. The more your work supports travel, etc - the easier it will be for you to get votes, etc (so deb is suggesting that the bias jannis is talking about is the right fit ) - barry: but anyone can be nomiated for the board - carol: we are talking about 2 things. initial sseeding of the concil and then ongoing vote process. - And we are discussing the 4th group in the list (which was non profits, companies, players int he space) - Barry: suggests keeping the first 3 bullets which is pypa, pacakging .... and then make the 4th bullet - psf members. - Deb: if we didn't align the election with board election - we could do it 3-6 months after (to avoid confusion) - Pheobe: will be important to get messaging and outreach right. If we have the ability to do outreach to multiple communities... - Shauna: i don't kwno if i should vote or not as an example. - geoffrey: important to consider community messaging - how do we describe the roles? How do we describe who can vote and who's voices we want. - deb: maybe we want to talk about it in terms of a stakeholder. messaging is a lot of work - leah: leaving a note -- i thijnk that 4th bullet should be used as strategy around which groups we wnat to be represented in the elections - jannis: what are the values and the goals that we want to reach. - barry: if we adopt the block star voting, we don't have software to readily use. - jannis we may not have the knowledge to solve this... - could we reach a consensus on there is a packaging council, what the dates are, the basic items -- and communicate about that now but take more time to decide the process. ### Cross-builds and Emscripten - Emscripten PEP 783 Delegation process - Packaging delegation is weird - Delegation is a standing order - Donald Stufft has authority for packaging tags - Packaging council is intended to avoid this problem - Cross builds - Corporate governance will have issues with unofficial distributions - BeeWare/Chaquopy/Emscripten aims to stop doing this anyway - Although Emscripten's approach is slightly different as it involves *some* injestion of builds from upstream sources - May be necessary to retain for the short term for build-time dependencies - [PEP 720](https://peps.python.org/pep-0720/) provides information on approaches for cross compiling - uv provides methods for cross compiling using a `--target` - Proposal to try doing this for Android with uv during the sprints as a step forward - BeeWare currently has a [script](https://github.com/beeware/Python-Apple-support/blob/main/patch/Python/_cross_venv.py) to create a cross build a venv - For Android, Choquapy is creating an environment, then it is calling the `build` API in to seperate steps to install the dependencies and then finally running the build without isolation - For iOS, we are monkeypatching build - Both Pyodide and Android are also wrapping commands sent to the compiler - One approach might be to abstract a `cross-build` tool from what we are doing for Pyodide, Android, and iOS - Discussed complexities of building NumPy and SciPy, there are small changes that could be made to these projects to make them much easier to cross compile against - There are 4 packages that have exceptions in Pyodide with numpy, scipy, cffi, and pycparser - Could we create a build frontend that covers Android, BeeWare, and Pyodide - with exceptions for the hard ones that we then work on? ### Dynamic metadata and licensing - Do we need statically specified metadata fields at all? - Maybe better to have this specified by tools - Also what we were thinking - From PSF perspective want to make sure that the proposal will not remove the license files currently added in PEP 639 - Nope, that will stay for sure - Boost instagram has files in the sdist not in the repository - Submodules get added to the sdist - What I would do is say `MIT AND Boost` - If it was split then it would go into `distribution` - Hypothetical: schemastore - Pulls in schemas from all over the internet with different licenses - Haven't worked out how to do a build pipeline that checks all this dynamically daily but only puts out releases when something changes - Repo would be MIT but downloads would be datapackages - Don't have a lot of data packages in Python - How this ties to dynamic metadata - Already have dynamic in sdist to tell that wheel have something different - Dynamic metadata PEP/package makes it easier for users to get this access to this without every backend implementing this - Could have a single plugin handle this - Could get rid of sdist, wheels and wheeltag as the plugin would handle this - Current issue is that with dynamic in pyproject.toml is cannot statically specify any licenses if any are to be added dynamically or very between backends - Need to strictly specific what is allowed to be added - Current thinking in the PEP is you can append but not modify - Maybe need to tighten this up more - Say we have static dependencies A and B, say they are partially dynamic - What if backend returns B and C as the dynamic metadata? - What would the current PEP allow - Current PEP allows ABBC or ABC - Maybe should tighten it up to just require ABBC - Case I'm worried about is if you have A & B<1.0 and dynamic you had B<0.5 & C. There's a logical merge here - Under current PEP 414 you can't logically merge these specifiers - Because of handling the prerelease specification, less than is non-transitive specific - Hope to write a PEP someday to clarify this - With example of literal identical specs you could simplify that, but maybe we don't want to - Most common case A and then A<something - Can following the spec, but `packaging` implements different logical filtering system that makes them incompatible with one another - PR open to fix this but no response [PR#897](https://github.com/pypa/packaging/pull/897) - specifierset.filter has two codebaths for specified vs unspecified and the logic is completely different - logic is to make everything consistant - What is the use case for license fields that are partially dynamic - Could specify in pyproject.toml the licenses of NumPy itself and the core componets - Then when Meson is running it could extend the license field and add licenses of things it bundles in

    Import from clipboard

    Paste your markdown or webpage here...

    Advanced permission required

    Your current role can only read. Ask the system administrator to acquire write and comment permission.

    This team is disabled

    Sorry, this team is disabled. You can't edit this note.

    This note is locked

    Sorry, only owner can edit this note.

    Reach the limit

    Sorry, you've reached the max length this note can be.
    Please reduce the content or divide it to more notes, thank you!

    Import from Gist

    Import from Snippet

    or

    Export to Snippet

    Are you sure?

    Do you really want to delete this note?
    All users will lose their connection.

    Create a note from template

    Create a note from template

    Oops...
    This template has been removed or transferred.
    Upgrade
    All
    • All
    • Team
    No template.

    Create a template

    Upgrade

    Delete template

    Do you really want to delete this template?
    Turn this template into a regular note and keep its content, versions, and comments.

    This page need refresh

    You have an incompatible client version.
    Refresh to update.
    New version available!
    See releases notes here
    Refresh to enjoy new features.
    Your user state has changed.
    Refresh to load new user state.

    Sign in

    Forgot password
    or
    Sign in via Facebook Sign in via X(Twitter) Sign in via GitHub Sign in via Dropbox Sign in with Wallet
    Wallet ( )
    Connect another wallet

    New to HackMD? Sign up

    By signing in, you agree to our terms of service.

    Help

    • English
    • 中文
    • Français
    • Deutsch
    • 日本語
    • Español
    • Català
    • Ελληνικά
    • Português
    • italiano
    • Türkçe
    • Русский
    • Nederlands
    • hrvatski jezik
    • język polski
    • Українська
    • हिन्दी
    • svenska
    • Esperanto
    • dansk

    Documents

    Help & Tutorial

    How to use Book mode

    Slide Example

    API Docs

    Edit in VSCode

    Install browser extension

    Contacts

    Feedback

    Discord

    Send us email

    Resources

    Releases

    Pricing

    Blog

    Policy

    Terms

    Privacy

    Cheatsheet

    Syntax Example Reference
    # Header Header 基本排版
    - Unordered List
    • Unordered List
    1. Ordered List
    1. Ordered List
    - [ ] Todo List
    • Todo List
    > Blockquote
    Blockquote
    **Bold font** Bold font
    *Italics font* Italics font
    ~~Strikethrough~~ Strikethrough
    19^th^ 19th
    H~2~O H2O
    ++Inserted text++ Inserted text
    ==Marked text== Marked text
    [link text](https:// "title") Link
    ![image alt](https:// "title") Image
    `Code` Code 在筆記中貼入程式碼
    ```javascript
    var i = 0;
    ```
    var i = 0;
    :smile: :smile: Emoji list
    {%youtube youtube_id %} Externals
    $L^aT_eX$ LaTeX
    :::info
    This is a alert area.
    :::

    This is a alert area.

    Versions and GitHub Sync
    Get Full History Access

    • Edit version name
    • Delete

    revision author avatar     named on  

    More Less

    Note content is identical to the latest version.
    Compare
      Choose a version
      No search result
      Version not found
    Sign in to link this note to GitHub
    Learn more
    This note is not linked with GitHub
     

    Feedback

    Submission failed, please try again

    Thanks for your support.

    On a scale of 0-10, how likely is it that you would recommend HackMD to your friends, family or business associates?

    Please give us some advice and help us improve HackMD.

     

    Thanks for your feedback

    Remove version name

    Do you want to remove this version name and description?

    Transfer ownership

    Transfer to
      Warning: is a public team. If you transfer note to this team, everyone on the web can find and read this note.

        Link with GitHub

        Please authorize HackMD on GitHub
        • Please sign in to GitHub and install the HackMD app on your GitHub repo.
        • HackMD links with GitHub through a GitHub App. You can choose which repo to install our App.
        Learn more  Sign in to GitHub

        Push the note to GitHub Push to GitHub Pull a file from GitHub

          Authorize again
         

        Choose which file to push to

        Select repo
        Refresh Authorize more repos
        Select branch
        Select file
        Select branch
        Choose version(s) to push
        • Save a new version and push
        • Choose from existing versions
        Include title and tags
        Available push count

        Pull from GitHub

         
        File from GitHub
        File from HackMD

        GitHub Link Settings

        File linked

        Linked by
        File path
        Last synced branch
        Available push count

        Danger Zone

        Unlink
        You will no longer receive notification when GitHub file changes after unlink.

        Syncing

        Push failed

        Push successfully