Distutils

Links

• CPython code: https://github.com/python/cpython/blob/master/Lib/distutils/command/install.py
• Fedora patch: https://github.com/fedora-python/cpython/commit/acba1a24b6ad5b894b4d6ef3223c2d174e225d21
• Debian patch: https://salsa.debian.org/cpython-team/python3-stdlib/blob/master/debian/patches/3.8/distutils-install-layout.diff
• PyPy patch: https://github.com/pypa/distutils/commit/2f1fae1bfceb88cf812a45c44cd18ce604b69c69#diff-6742915ffa9375e1830aa1e69016d3cf
  • PR: https://github.com/pypa/distutils/pull/1
• Initial discussion: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059
• PEP 632 discussion: https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/97
• PEP 632: https://www.python.org/dev/peps/pep-0632/
• Distutils issue: https://github.com/pypa/distutils/issues/12
  • Debian-specific: https://github.com/pypa/distutils/issues/2
  • Patch: https://github.com/pypa/distutils/pull/4/files
• Setuptools history: https://setuptools.readthedocs.io/en/latest/history.html
• Setuptools must support Python 3.5+ (maybe 3.6+)
• bpo-43976: Allow Python distributors to add custom site install schemes

Meeting notes from 2020-09-24

meet.jit.si/distutils

Status

• [Miro] Proposed text completed (today). \o/ Nice work!
• [jaraco] Setuptools docs revamp merged (with minor issues). Ready to merge distutils docs (imminent).

Discussion

[Petr] Use-case not covered: older distros that aren't updated.

[jaraco] Mentioned concern about Python version in egg-info. Defer until Mathias has more information.

Proposed text for PEP 632 discussion

https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/104

Meeting notes from 2020-09-17

[mathias] Two issues mainly (Debian):

• Headers support
• Location support
• Python version in metadata (egg-info)

Fedora (and most likely Debian):

• root user installs Python packages to /usr/local/lib64/pythonX.Y by default
• option to install Python packages to /usr/lib64/pythonX.Y (used when building RPM/DEB packages)
• (Fedora has) /usr/local/lib64/pythonX.Y/site-packages in sys.path first, but not with -s

• (recent – pip >=10.0) pip refuses to remove PKG-INFO-less installations
  • but that makes sudo pip install -U ... fail anyway
  • metadata missing in that case?

Issue: sudo pip install -U setuptools loses patches

pip: Long plan to move to sysconfig from distutils.sysconfig.

[Miro] Proposal: platforms can patch sysconfig and distutils can rely on that to load install schemes. That means setuptools can only switch to its distutils for Python 3.10+ (assuming we do this in 3.10).

[Jason] Problem: Old Python versions. Setuptools would like to be able to change distutils.

[Jason] Define schemes in setuptools…distutils that "work good enough" for both Debian and Fedora.

[Mathias] Sysconfig is a mess; exposes too much. Need vision first.

[Pradyun] distutils would need a stopgap solution until 3.9 goes EOL.

[Jason] Short term, let's have a "hook" the distributors can put to the standard library… For long term, define better defaults.

[Miro] Defaults that satisfy everybody may not be possible.

Design (install schemes):

1. pypa/distutils keeps INSTALL_SCHEMES defined for compatibility with Python < 3.10
2. distutils in Python 3.10+ moves INSTALL_SCHEMES to sysconfig and imports them to distutils for backwards compatibility
   TODO: Miro to look into the implications of "moving"; how would that work? since we already have a definition in sysconfig.
3. distributors would backport (2) to older Pythons
4. pypa/distutils uses INSTALL_SCHEMES from sysconfig if found, fallbacks to its own definition if not
5. pypa/distutils fallbacks will honor distributors intentions to the greatest extent possible

Problem: INSTALL_SCHEMES don't contain logic of picking the appropriate scheme.

Possible solution: See sysconfig._get_default_scheme() https://github.com/python/cpython/blob/master/Lib/sysconfig.py#L179 - distros would patch (or patch-add) this.

Debian has --install-layout for distutils, we can keep that in pypa/distutils and make it override sysconfig._get_default_scheme() value.

Next steps:

1. Publish draft design (over email)
2. Put that up on discourse or github/pypa/distutils issue
3. Share with Steve
4. Follow up next week to include Steve Dower (zooba)
5. Fun and profit!?
   Image Not Showing Possible Reasons

   • The image file may be corrupted
   • The server hosting the image is unavailable
   • The image path is incorrect
   • The image format is not supported

   Learn More →

Well, that didn't make anyone's day worse, I think. :)

• Pradyun

2021-03-12 Notes

FFY00 and jaraco brainstorming

dist install Python to /usr
as user build from source to /usr/local
How to avoid: both installations pull libraries from /usr/local/lib/python/site-packages?

Proposal: We should disallow /usr loading any packages from /usr/local.

Issue - still need multiple package directories for each Python installation. Why? To avoid conflicting pip-installed packages with package manager-installed packages. Package managers can elect an install scheme for this need.

Goal: Uniform behavior (including bugfixes and improvements in pypa/distutils) on all Pythons supported by Setuptools (3.6+).

How to use new distutils with old CPythons?

• Users can fallback to stdlib distutils patches with SETUPTOOLS_USE_DISTUTILS=stdlib. (discouraged)
• Distributions could patch sysconfig.INSTALL_SCHEMES on older Pythons.
• distutils will detect no INSTALL_SCHEMES (or other patch) to sysconfig and fallback to default behavior (breaks when expecting custom behavior).
• Use older Setuptools.

Why not just use INSTALL_SCHEMES from sysconfig? That's a good plan to aim for.

Proposal: Solicit flag from sysconfig to signal that it was patched to be compatible with new scheme.
Currently, sysconfig.INSTALL_SCHEMES does not exist. Could that serve as the flag?
When it exists on older Pythons (any), use it. If not, detect patch distutils (Debian, Fedora), warn that behavior may not be compatible with Distribution-patched distutils.
How important is this use-case?

Proposal: Remove 'unix-local' scheme.
It's just a footgun.
Let's explore the possibility.

2021-04-03 Notes

• Allowing packagers to customize/patch sysconfig.
• Distutils will use sysconfig.
• What about integration with site module. Making aliases for site modules.
• Schemes in sysconfig, but will also be imported in site module.
• Prefer for distributions to supply a file instead of patching.
  • How viable is that?
  • Single file, underscore-prefixed.
  • sysconfig and site would load this module
  • Just a dictionary with the schemes.
  • Special module for configuration.
  • Extend the schemes. Add/override.
  • Maybe delete if needed, but probably not needed. Will assume not needed until we identify a need.
• Opportunity to make things correct.

How does site use install schemes:

• Looks up default scheme.
• Uses that to set up default paths.

Is a PEP needed?

• For now, no.
• Piggyback on existing distutils PEP.

Next steps:

• FFY00 working on configuration design and opening pull request in CPython.
• Command line argument support in distutils to allow installer to specify install scheme.