Distutils - HackMD

# Distutils ## Links * CPython code: https://github.com/python/cpython/blob/master/Lib/distutils/command/install.py * Fedora patch: https://github.com/fedora-python/cpython/commit/acba1a24b6ad5b894b4d6ef3223c2d174e225d21 * Debian patch: https://salsa.debian.org/cpython-team/python3-stdlib/blob/master/debian/patches/3.8/distutils-install-layout.diff * PyPy patch: https://github.com/pypa/distutils/commit/2f1fae1bfceb88cf812a45c44cd18ce604b69c69#diff-6742915ffa9375e1830aa1e69016d3cf * PR: https://github.com/pypa/distutils/pull/1 * Initial discussion: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059 * PEP 632 discussion: https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/97 * PEP 632: https://www.python.org/dev/peps/pep-0632/ * Distutils issue: https://github.com/pypa/distutils/issues/12 * Debian-specific: https://github.com/pypa/distutils/issues/2 * Patch: https://github.com/pypa/distutils/pull/4/files * Setuptools history: https://setuptools.readthedocs.io/en/latest/history.html * Setuptools must support Python 3.5+ (maybe 3.6+) * [bpo-43976: Allow Python distributors to add custom site install schemes](https://bugs.python.org/issue43976) ------------- ## Meeting notes from 2020-09-24 meet.jit.si/distutils ### Status * [Miro] Proposed text completed (today). \o/ Nice work! * [jaraco] Setuptools docs revamp merged (with minor issues). Ready to merge distutils docs (imminent). ### Discussion [Petr] Use-case not covered: older distros that aren't updated. [jaraco] Mentioned concern about Python version in egg-info. Defer until Mathias has more information. ------------- ## Proposed text for PEP 632 discussion https://discuss.python.org/t/pep-632-deprecate-distutils-module/5134/104 ------------------ ## Meeting notes from 2020-09-17 [mathias] Two issues mainly (Debian): * Headers support * Location support * Python version in metadata (egg-info) ----- Fedora (and most likely Debian): * root user installs Python packages to `/usr/local/lib64/pythonX.Y` by default * option to install Python packages to `/usr/lib64/pythonX.Y` (used when building RPM/DEB packages) * (Fedora has) `/usr/local/lib64/pythonX.Y/site-packages` in `sys.path` first, but not with `-s` ----- * (recent -- pip >=10.0) pip refuses to remove `PKG-INFO`-less installations * but that makes `sudo pip install -U ...` fail anyway * metadata missing in that case? ---- Issue: `sudo pip install -U setuptools` loses patches pip: Long plan to move to [sysconfig](https://docs.python.org/3/library/sysconfig.html) from distutils.sysconfig. [Miro] Proposal: platforms can patch sysconfig and distutils can rely on that to load install schemes. That means setuptools can only switch to its distutils for Python 3.10+ (assuming we do this in 3.10). [Jason] Problem: Old Python versions. Setuptools would like to be able to change distutils. [Jason] Define schemes in setuptools...distutils that "work good enough" for both Debian and Fedora. [Mathias] Sysconfig is a mess; exposes too much. Need vision first. [Pradyun] distutils would need a stopgap solution until 3.9 goes EOL. [Jason] Short term, let's have a "hook" the distributors can put to the standard library... For long term, define better defaults. [Miro] Defaults that satisfy everybody may not be possible. ---- Design (install schemes): 1. pypa/distutils keeps `INSTALL_SCHEMES` defined for compatibility with Python `< 3.10` 2. distutils in Python 3.10+ moves `INSTALL_SCHEMES` to sysconfig and imports them to distutils for backwards compatibility TODO: Miro to look into the implications of "moving"; how would that work? since we already have a definition in sysconfig. 3. distributors would backport (2) to older Pythons 4. pypa/distutils uses `INSTALL_SCHEMES` from sysconfig if found, fallbacks to its own definition if not 5. pypa/distutils fallbacks will honor distributors intentions to the greatest extent possible Problem: `INSTALL_SCHEMES` don't contain logic of picking the appropriate scheme. Possible solution: See `sysconfig._get_default_scheme()` https://github.com/python/cpython/blob/master/Lib/sysconfig.py#L179 - distros would patch (or patch-add) this. Debian has `--install-layout` for distutils, we can keep that in pypa/distutils and make it override `sysconfig._get_default_scheme()` value. Next steps: 1. Publish draft design (over email) 2. Put that up on discourse or github/pypa/distutils issue 3. Share with Steve 4. Follow up next week to include Steve Dower (zooba) 5. Fun and profit!? :moneybag: Well, that didn't make anyone's day worse, I think. :) - Pradyun # 2021-03-12 Notes ## FFY00 and jaraco brainstorming dist install Python to `/usr` as user build from source to `/usr/local` How to avoid: both installations pull libraries from `/usr/local/lib/python/site-packages`? Proposal: We should disallow `/usr` loading any packages from `/usr/local`. Issue - still need multiple package directories for each Python installation. Why? To avoid conflicting pip-installed packages with package manager-installed packages. Package managers can elect an install scheme for this need. Goal: Uniform behavior (including bugfixes and improvements in pypa/distutils) on all Pythons supported by Setuptools (3.6+). How to use new distutils with old CPythons? - Users can fallback to stdlib distutils patches with `SETUPTOOLS_USE_DISTUTILS=stdlib`. (discouraged) - Distributions could patch sysconfig.INSTALL_SCHEMES on older Pythons. - distutils will detect no INSTALL_SCHEMES (or other patch) to `sysconfig` and fallback to default behavior (breaks when expecting custom behavior). - Use older Setuptools. Why not just use INSTALL_SCHEMES from sysconfig? That's a good plan to aim for. Proposal: Solicit flag from sysconfig to signal that it was patched to be compatible with new scheme. Currently, `sysconfig.INSTALL_SCHEMES` does not exist. Could that serve as the flag? When it exists on older Pythons (any), use it. If not, detect patch distutils (Debian, Fedora), warn that behavior may not be compatible with Distribution-patched distutils. How important is this use-case? Proposal: Remove 'unix-local' scheme. It's just a footgun. Let's explore the possibility. # 2021-04-03 Notes - Allowing packagers to customize/patch sysconfig. - Distutils will use sysconfig. - What about integration with site module. Making aliases for site modules. - Schemes in sysconfig, but will also be imported in site module. - Prefer for distributions to supply a file instead of patching. - How viable is that? - Single file, underscore-prefixed. - sysconfig and site would load this module - Just a dictionary with the schemes. - Special module for configuration. - Extend the schemes. Add/override. - Maybe delete if needed, but probably not needed. Will assume not needed until we identify a need. - Opportunity to make things correct. How does `site` use install schemes: - Looks up default scheme. - Uses that to set up default paths. Is a PEP needed? - For now, no. - Piggyback on existing distutils PEP. Next steps: - FFY00 working on configuration design and opening pull request in CPython. - Command line argument support in distutils to allow installer to specify install scheme.