# Python Packaging Summit - PyCon US 2022

This will contain a written summary of the topics discussed at the summit. Present at the summit 3 organizers and 15 attendee.

## Let’s fix extras in Core Metadata 3.0 - Tzu-ping Chung

[Link to slides](https://speakerdeck.com/uranusjr/lets-fix-extras-in-core-metadata-3-dot-0)

Current metadata version is `2.4`. Main problem is extras, we don't know how to normalize and compare it (part solved by Brett's PEP-685 - extra must be normalized but if a tool doesn't do it we're in trouble). We need to make sense of have extras work in markers. What does `==` means in case of extra? `pip install package[docs,test]` but we evaluate with any.

Proposal: introduces extras instead of extra, and the only operator valid is `in` (allow `not` for negation).

Other problem is default extra groups. Pradyun showed interest. Problem: what happens when `pip install package[another]` should it pull in the default? What about when the user specifies `pip install package[]`.

We can keep version `2.x` and keep compatible if we still allow extra. Alternative is to bump to `3.0` and then cleanup the spec of `Author`. `Maintainer`, `Home`-page; plus any other we can find.

Concerns:

- Backwards compatibility - new metadata support with old pip?
- have pip-requires to fail with requiring newer pip?
- Dustin raised that when he looked into it default extra wasn't easily and cleanly possible
- Allow libraries to choose what metadata version they want to support? E.g. library maintainer can opt-in into it eventually, but not day 1; similar to how they do it for Python. No support for user to specify the package metadata version though.
- Can we add metadata part of the wheel name? It's difficult to filter upfront without it, currently we have to download the wheel to extract the metadata version.

## Standardizing External Data - Steven Silvester

[Link to slides](https://drive.google.com/file/d/1LXkNtxvWRATl6rSiCa4PdfYQ6g7vgIRW/view?usp=sharing)

Jupyter ecosystem uses data_files, but is deprecated. There's no standard in puting files at given location. Data files can be put anywhere. Flit added support for external data in 3.7.

- It affects wheels? Should we put it into the wheels project instead?
- Dustin raised concerned people abusing to store more data and can have PyPI impact.

## Building C Extensions without Setuptools - Henry Schreiner III and Ofek Lev

[Link to slides](https://docs.google.com/presentation/d/1XDC3A3DqB9O0JIyqefLDN-NIAwD0uLqTgYDM8ASRiog)

Shared libraries that contains compiled code. We achieved PEP-517 for pure python packages. Extension builders still are mostly locked in into setuptools (mesonpy and maturin are the two exceptions).

API:

Init: root + config
Required methods: build + get_expected_outputs

Idea to write a plugin standard for extensions. Build backend would be expected to go over the extensions arreay and call each builder and manage the returned files (multiple to allow build rust + c build at the same time). Add a default flag to build by default or not. Add requires flag for each extension? Would make it a lot more complex. This way less isolated build environments needed.

Question:
- Would PyPA support it? Would be nice, but wondering feasibality
- What currently happens (high level) to convert options to final subprocess command?
- Discussed the API and when the build backend should run the extensions - up to the build backend when during the build runs it (must be during `build_wheel`)

## Python Packaging Vision and Strategy - Ee Durbin

[Link to slides](https://drive.google.com/file/d/1hMGHsLfIfF3bMCUL7db2GUPyZjgFAJgy)

On behalf of Shamika, PSF Packaging project manager. Motivation, packaging is:

- breadth and variety
- silos (sections focus on one special problem)
- lack of cohesion
- lack of direction - where we heading, we generally address needs as comes up.

Objectives:

How do we become the best packaging environment?

- identify factors
- determin and catalog how we make it more sustainable
- identify funding.

Plan posted by Shamika. Lack of authority? How do we ensure it manifests and progresses. The goal of PSF is not to become authority in the packaging domain. Primary goal is to facilitate the communication within the community. Secondarily is to collect all the goals of the community they should be flexible and far. This would allow to make decision with more of a goal, a direction. Once we have the goals we can have a better presentation for potential investors for projects to fund.

A big concern at the moment lack of funding for projects for complex topics and projects.

## New Contributors in the Python Packaging Community - Ee Durbin

[Link to slides](https://drive.google.com/file/d/1hMGHsLfIfF3bMCUL7db2GUPyZjgFAJgy)

[Detailed proposal](https://tinyurl.com/2wccjawr):

- High expectations
- Latency
- Sustainability issue
- Diversity

One way is to offer Python Packaging Fellowship. Students -> New Contributors -> Long time contributors. Students is for talent pool and interest/benefit in doing OOS. How?

Phase 0. Fellowship Wokring Group to decide numbers, logistics and success criteria and obtain funding.
Phase 1. 3 months duration, initially contributors aimed to resolve issues within the project, maintainers compenstated financially.
Phase 2. 6 to 9 months, work on bigger projects. Fellows are expected to contribute more broader. Mentors can provide 1:1 mentoring and compenstate both fellows and mentors.

Discussion:
- What makes them stick around later? We don't expect all to stick around but good experience but some of the many contractors on PyPI do stick around.
- Who will drive this? PSF, could be Shamika or another group within PSF or antoher contracter.
- How does student selection work? No concrete yet, but probably some CV/application.
- If you're targeting under graduates are we too low on Python knowledge to perform meaningful work. Graduates are better at this instead? We target everything after seconday school, mostly students after high school.
- Cash prizes - as in hacker compentition model can spark people just trying stuff. This might bring more people into the space? EE says they considered this for PyPI, the problem is that the maintainers can be overloaded. There's no global lock of contributors for bounties.
- Worry that students can try to chew way more than they can do in 6 months. We can easily scare people in this.
- Flask had similar program (major league hacking - company doing interview/scholarship/hiring). It was a variable success for Flask. Some session that closed tons of issues, some where they were 2-3. No long term commitment, most stayed around <9 months. In the first session when they had tons of issues seemed more successful. Later on more complicated problems it felt more like pair programming which wasn't the best use of the maintainers time.
- Long term feedback in the Princeton university has been good for similar programs, but students don't necessarily end up working on the same project as the ones they started. EE says they know what a trial of this might look like (funded by PSF, very small size initially - probably 1 project 2/3 intern and 1 fellow). PSF plans to take on board what they learn.
- PSF plans to make the program global (not US based). Excluding sanctioned countries PSF can pay people in any other countries.
- Should we limit this to graduate? Some people raised concern that many high school people could contribute if they are outstanding in programming.
- How will projects and students be selected? What role maintainers have in project and student selection? EE has no idea yet. A logistical thing that needs to be decided later.

## Robyn project - async based python web framework with rust runtime

Created by Sanskar Jethi in 2021. Created as an async Flask. [Project url](https://github.com/sansyrox/robyn). Using PyO3 via maturin (initially setuptools-rust). [Documentation](https://sansyrox.github.io/robyn/#/). Architecture presented. Uses rust native threads to avoid issues of the GIL. reddit advertised generally loved. Presented as a use case for binary extensions. Packaging challenges? Had to use maturin.

## Standardizing external dependencies - Fillipe Lains

We already have `Requires-External` field but has not semantics attached to it. Proposal to attach semnatic. How?

- Create a DB to map dependency format to system provided dependencies. Can be iehter community maintained (Debian/Arch) or vendor maintained (Conda Forge/Anaconda). We should also provide an excape hatch: allow users to specify explicitly what to use.

E.g. defined as `libblas.so.3-abi` that translate for debian to `libblas == 3.*` and on arch to `blas == 3.*`. Escape hatch would look like in the `Requires-External` as `debian/libblas = 3.*` or `archlinux/blas == 3.*`. Questions:

- In the DB who decides what is the key? Community would decide. We'd have some guidance.
- What happens with a lib not in
- Installer checks or build backend? Fillipe would like wheels not to vendor it, so it would require user to install it globally and the installer would be the one doing the check. We could allow some way to install the system package without root (conda does support some of this). A user of this might be CUDA.

## [Scikit - HEP](https://scikit-hep.org ) - Henry Schreiner III

[Link to slides](https://www.slideshare.net/HenrySchreiner/pycon-2022-scikithep-developer-pages-guidelines-for-modern-packaging)

https://scikit-hep.org built to help scikit people how to create a good package. The only thing specific is package requirements. Everything is generic to any python project. More freedom to change things than https://packaging.python.org. Makes some arbitrary decisions (e.g. `src` layout). On style it uses pre-commit configuration recommended. Information on how to setup on Github Actions. Page on how to use nox. And has a repo review tool. Need for negative extras, so could install by default just the package and on the web specify to not install some extras. Uses python in web browser.