owned this note changed 4 years ago
Published Linked with GitHub

Changes in Python packaging guidelines

These are changes between the current (“201x-era”) Python packaging guidelines and the proposal as of 2021-03-03.

All MUST and SHOULD items from both versions are covered. (For the old guidelines I added more, where I feel they should have been marked SHOULD.) If you have comments or reservations, please read the whole section of the new guidelines. Direct your comments to the Python SIG mailing list. (Comments on HackMD are hard to manage.)

Minor changes are hidden in *details* tags like this. To open one, click the summary.

(Now you can see these details.)

I tried to list the most important changes first.

Rationales and explanations

The new guidelines give rationales and explanations where applicable. In many cases these changed. This list of changes generally doesn't list these “soft” changes.

PyPI Parity

The most profound change is synchronization of Python project names with the Python Package Index, allowing Python-level metadata across the ecosystem.

NEW: Every Python package in Fedora SHOULD also be available on the Python Package Index (PyPI).

The command pip install PROJECTNAME MUST install the same package (possibly in a different version), install nothing, or fail with a reasonable error message.

If this is not the case, the packager SHOULD contact upstream about this. The goal is to get the project name registered or blocked on PyPI, or to otherwise ensure the rule is followed.

As a reminder: if you want to comment, please read the relevant section for details.

Upstream metadata

The second major paradigm shift is that package metadata is primarily taken from upstream, since upstream “dist-info” metadata is now standardized and shareable with other distributors (be it Linux distros or others). Packagers are expected to treat metadata bugs as any other bugs, (ideally, patch them and present the patches upstream).

NEW: Packagers SHOULD be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem.

Beta period (before the guidelines are approved)

The new guidelines are “beta” period where they can be used instead of the old ones, but ask for increased cooperation from packagers who opt in to use them.

NEW: These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this. Older guidelines are still being kept up to date, and existing packages MAY use them instead of this document

NEW: While these guidelines are in Beta, each Python package MUST have BuildRequires: pyproject-rpm-macros to access the beta macros.

Python 3

Python 2 has been deprecated for a while now; using it requires a FESCo exception. Text related to the py2/py3 split is generally removed or simplified.

NEW: Fedora packages MUST NOT depend on other versions of the CPython interpreter than the current python3. [] Packages such as pip or tox, which enable setting up isolated environments and installing third-party packages into them, MAY, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter.

OLD: If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged into Fedora without FESCo exception.

The following redundant note was put in the guidelines to address pushing "remove of python2 package" changes to stable branches. Since not many python2 packages are left, the note is not necessary any more:

OLD: Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.

Dependency Generator

The dep generator is now mandatory (i.e. upstream metadata must be used/patched):

NEW: Packages MUST use the automatic Python run-time dependency generator. Packages SHOULD use the opt-in build-dependency generator if possible. he packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.” Any necessary changes MUST be done by patches or modifying the source (e.g. with sed), rather than disabling the generator. The resulting change SHOULD be offered to upstream. As an exception, filtering MAY be used for temporary workarounds and bootstrapping. Dependencies covered by the generators SHOULD NOT be repeated in the .spec file. (For example, if the generator finds a requests dependency, then Requires: python3-requests is redundant.)

OLD: This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: %{?python_disable_dependency_generator}

OLD: [if using dependency generator] The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.

Applicability

New clarification:

NEW: Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities[]. However, if [you need] a more complex Python library, the library SHOULD be packaged as an importable library under these guidelines.

Naming

Clarified wording on General naming:

NEW: A built (i.e. non-SRPM) package for a Python library MUST be named with the prefix python3-.

A source package containing primarily a Python library MUST be named with the prefix python-.

OLD: The source package for a Python library MUST be named with the python- prefix.

OLD (in Naming guidelines): Python2 binary packages MUST be named using a python2- prefix. Python3 binary packages MUST be named with a prefix of python3-.

Clarified wording on `python3-` provides for importable modules:

NEW: For any module intended to be used in Python 3 with import MODNAME, the package that includes it SHOULD provide python3-MODNAME, with underscores (_) replaced by dashes (-).

OLD: For any module foo intended to be used in Python 3 with import foo, the package that includes it should provide python3-foo

Clarified wording on `python-`/`python3-`/`python3.X-` provides:

NEW: For any FOO, a package that provides python3-FOO SHOULD use %py_provides or an automatic generator to also provide python-FOO and python3.X-FOO, where X is the minor version of the interpreter. The provide SHOULD NOT be added manually: if a generator or macro is not used, do not add the python-FOO / python3.X-FOO provides at all.

OLD: All packages that provide python3-... (for any ...) SHOULD also provide python-... and python3.X-....

OLD: Any manually added virtual provides of python3-... SHOULD be done via the %py_provides macro.

Removed ban on python- Requires, since the python- prefix now refers to Python 3:

OLD: Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python- prefix. Dependencies on Python packages instead MUST use names beginning with python3-.

Best practices (SHOULDs):

NEW: If the importable module name and the project name do not match, users frequently end up confused. In this case, packagers SHOULD ensure that upstream is aware of the problem and (especially for new packages where renaming is feasible) strive to get the package renamed. The Python SIG is available for assistance.

NEW: The Fedora package's name SHOULD contain the canonical project name. If possible, the project name SHOULD be the same as the name of the main importable module, with underscores (_) replaced by dashes (-).

NEW: Packages that primarily provide applications, services or any kind of executables SHOULD be named according to the general Fedora naming guidelines (e.g. ansible).

OLD (in Naming guidelines): The package name SHOULD reflect the upstream name of the Python module, and SHOULD generally take into account the name of the module used when importing it in Python scripts. This name will be prefixed depending on the type of the package.

Removed rules on removing %py_provides (which is documented as being for cases where the generator doesn't work) and %python_provide (which is listed in Deprecated macros):

OLD: Packagers SHOULD try to remove explicit %py_provides calls for package names, but MAY preserve them if they aim for compatibility with older releases or packages without files.

OLD (note): Historically, there was %python_provide macro with similar but different semantics. It still works for compatibility reasons but it is deprecated and SHOULD NOT be used and packagers SHOULD replace is with appropriate %py_provides call.

Naming Extras

Slight addition on including devel extras (diff emphasized):

NEW: Python packages SHOULD have Provides for all extras the upstream project specifies, except: those that are not useful for other packages (for example build/development requirements, commonly named dev, doc or test), and those that have requirements that are not packaged in Fedora.

OLD: Python packages SHOULD have Provides for all extras the upstream project specifies, except those that are not useful for other packages (for example build/development requirements, commonly named dev, doc or test).

New best practice re. removing extras:

NEW: If an existing extra is removed from an upstream project, the Fedora maintainer SHOULD try to convince upstream to re-introduce it. If that fails, the extra SHOULD be Obsoleted from either the main package or another extras subpackage.

Reworded text on +; same meaning

NEW: The character + in names of built packages (i.e. non-SRPM) that include .dist-info or .egg-info directories is reserved for extras and MUST NOT be used for any other purpose. As an exception, + characters MAY appear at the end of such names.

OLD: The character + in names of built packages (i.e. non-SRPM) that include .dist-info or .egg-info directories is reserved for Python Extras and MUST NOT be used for any other purpose. As an exception, + characters are permitted at the end of the name.

NEW: A package that provides a Python extra MUST provide python3dist(DISTNAME[EXTRA]) and python3.Xdist(DISTNAME[EXTRA]), where X is the minor version of the interpreter, DISTNAME is the canonical project name, and EXTRA is the name of a single extra. For example, python3.9dist(requests[security]). These requirements SHOULD be generated using the automatic dependency generator.

OLD: A package that provides a Python extra MUST provide python3dist(…[…]) and python3.Xdist(…[…]), for example, python3.9dist(requests[security]). These requirements SHOULD be generated using the automatic dependency generator.

Unchanged guidelines

NEW: A package that provides a Python extra MUST require the extra's main package with exact NEVR.

OLD: A package that provides a Python extra MUST require the extra’s main package with exact NEVR.

NEW: A subpackage that primarily provides one Python extra SHOULD be named by appending "+" and the extra name to the main package name. For example, python3-requests+security.

OLD: A subpackage that primarily provides one Python extra SHOULD be named by appending + and the extra name to the main package name. For example, python3-requests+security.

Files to include

More detailed rules on what to include (and where):

NEW: Packages MUST include the source file (*.py) AND the bytecode cache (*.pyc) for each pure-Python importable module. The source files MUST be included in the same package as the bytecode cache.

NEW: Each Python package MUST include Package Distribution Metadata conforming to PyPA specifications (specifically, Recording installed distributons). The metadata SHOULD be included in the same subpackage as the main importable module, if there is one. [] As an exception, the Python standard library MAY ship without this metadata.

OLD: When packaging Python modules, several types of files are included: *.py source files[,] *.pyc byte compiled files[,] *.egg-info or *.dist-info files or directories. [technically, there is no MUST here]

The source files MUST be included in the same package as the byte compiled versions.

New best practice: not byte-compiling scripts:

NEW: Scripts that are not importable (typically ones in %{_bindir} or %{_libexecdir}) SHOULD NOT be byte-compiled.

Consolidated rules on content in shared directories

NEW: Packages MUST NOT own shared directories owned by Python itself, such as the top-level __pycache__ directories (%{python3_sitelib}/__pycache__, %{python3_sitearch}/__pycache__).

NEW: Packagers SHOULD NOT simply glob everything under a shared directory. In particular, the following SHOULD NOT be used: %{python3_sitelib}/*, %{python3_sitearch}/*, %{python_sitelib}/*, %{python_sitearch}/*, %{_bindir}/*, %pyproject_save_files '*', %pyproject_save_files +auto.

OLD: Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used: %{python3_sitelib}/*, %{python3_sitearch}/*, %{python_sitelib}/*, %{python_sitearch}/*. And packages MUST NOT include the top-level __pycache__ directory (see below).

OLD: You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.

Mandatory Requires / Provides

NEW: Every package that uses Python (at runtime and/or build time), and/or installs Python modules MUST explicitly include BuildRequires: python3-devel in its .spec file, even if Python is not actually invoked during build time. If the package uses an alternate Python interpreter instead of python3 (e.g. pypy, jython, python2.7), it MAY instead require the corresponding *-devel package.

NEW: As mentioned above, each Python package MUST explicitly BuildRequire python3-devel. Packages MUST NOT have dependencies (either build-time or runtime) with the unversioned prefix python- if the corresponding python3- dependency can be used instead. Packages SHOULD NOT have explicit dependencies (either build-time or runtime) with a minor-version prefix such as python3.8- or python3.8dist(. Such dependencies SHOULD instead be automatically generated or a macro should be used to get the version. Packages SHOULD NOT have an explicit runtime dependency on python3.

NEW: Every Python package MUST provide python3dist(DISTNAME) and python3.Xdist(DISTNAME), where X is the minor version of the interpreter and DISTNAME is the canonical project name corresponding to the dist-info metadata. For example, python3-django would provide python3dist(django) and python3.9dist(django). This is generated automatically from the dist-info metadata. The provide SHOULD NOT be added manually: if the generator fails to add it, the metadata MUST be fixed.

Tests

Running upstream tests is mandatory; linters are discouraged

NEW: If a test suite exists upstream, it MUST be run in the %check section and/or in Fedora CI. You MAY exclude specific failing tests. You MUST NOT disable the entire testsuite or ignore the result to solve a build failure. As an exception, you MAY disable tests with an appropriate %if conditional (e.g. bcond) when bootstrapping.

XXX The above was changed to SHOULD after this document was created

NEW: In %check, packages SHOULD NOT run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality.

Shebangs & Mandatory macros

The rule regarding /usr/bin/python has been generalized to mandatory use of some macros, and rules on shebangs were split off and made stricter.

NEW: The following macros MUST be used where applicable. %{python3}, %{python3_version}, %{python3_version_nodots}, %{python3_sitelib}, %{python3_sitearch}.

NEW: Shebang lines to invoke Python MUST use %{python3} as the interpreter.

NEW: Shebang lines to invoke Python SHOULD be #!%{python3} -%{py3_shebang_flags} and it MAY include extra flags. If the default flags from %{py3_shebang_flags} are not desirable, packages SHOULD explicitly redefine the macro to remove them.

OLD: Packages in Fedora MUST NOT use /usr/bin/python. Instead packages for Python 3 MUST use /usr/bin/python3 (even if upstream supports both Python 2 and 3). As a result of that /usr/bin/python (as well as /usr/bin/env python and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed (for example by using the %py3_shebang_fix macro in the spec file).

PyPI sources

PyPI archives that omit tests/docs are discouraged:

NEW: Packages MAY use sources from PyPI. However, packages SHOULD NOT use an archive that omits test suites, licences and/or documentation present in other source archives.

Specific macros

Redefining %python3_pkgversion is banned:

NEW: Packages in Fedora MAY use [%{python3_pkgversion}] (e.g. in package names: python%{python3_pkgversion}-requests), but MUST NOT redefine it.

Note on `%pyproject_save_files` features unsuitable for Fedora

NEW: Note that README and licence files are not included. Also, while the macro allows including executable and other files (using the +auto flag), this feature MUST NOT be used in Fedora.

Cython

Clarification; same meaning

NEW: Tightening the general Fedora policy, packages MUST NOT use files pre-generated by Cython. These MUST be deleted in %prep and regenerated during the build. As an exception, these sources MAY be used temporarily to prevent build time circular dependencies by following the bootstrapping guidelines.

OLD: Tightening the general Fedora policy, packages MUST NOT use pre-generated Cython sources. They MUST be deleted in %prep and regenerated during the build. Any exception to this rule should be considered a bootstrapping.

Best practices

New note on when to prefer python3 -m tool:

NEW: Every executable TOOL for which the current version of Python matters SHOULD also be invokable by python3 -m TOOL. If the software doesn't provide this functionality, packagers SHOULD ask the upstream to add it.

Select a repo