# Changes in Python packaging guidelines

These are changes between the [current (“201x-era”) Python packaging guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/) and the [proposal as of 2021-03-03](https://fedoraproject.org/wiki/Changes/PythonPackagingGuidelines202x).

All **MUST** and **SHOULD** items from both versions are covered. (For the old guidelines I added more, where I feel they should have been marked SHOULD.)
If you have comments or reservations, please *read the whole section* of the new guidelines.
Direct your comments to the [Python SIG mailing list]([e-mail]).
(Comments on HackMD are hard to manage.)

<details>
<summary>Minor changes are hidden in *details* tags like this. To open one, click the summary.</summary>

(Now you can see these details.)

</details>

I tried to list the most important changes first.

## Rationales and explanations

The new guidelines give rationales and explanations where applicable.
In many cases these changed.
This list of changes generally doesn't list these “soft” changes.

## PyPI Parity

The most profound change is synchronization of Python project names with the Python Package Index, allowing Python-level metadata across the ecosystem.

> **NEW:** Every Python package in Fedora **SHOULD** also be available
> on [the Python Package Index](https://pypi.org) (PyPI).
>
> The command `pip install PROJECTNAME` **MUST** install the same package
> (possibly in a different version),
install nothing,
> or fail with a reasonable error message.
>
> If this is not the case, the packager **SHOULD** contact upstream about this.
> The goal is to get the project name registered or blocked on PyPI,
> or to otherwise ensure the rule is followed.

As a reminder: if you want to comment, please read the [relevant section](https://hackmd.io/XzJe-sHUQvWK7cSrEH_aKg?both#PyPI-parity) for details.

## Upstream metadata

The second major paradigm shift is that package metadata is primarily taken from upstream,
since upstream “dist-info” metadata is now standardized and shareable with other distributors (be it Linux distros or others).
Packagers are expected to treat metadata bugs as any other bugs, (ideally, patch them and present the patches upstream).

> NEW: Packagers **SHOULD** be prepared to get involved with upstream projects to establish best practices as outlined here. We wish to improve both Fedora and the wider Python ecosystem.

## Beta period (before the guidelines are approved)

The new guidelines are “beta” period where they can be used instead of the old ones, but ask for increased cooperation from packagers who opt in to use them.

> NEW: These Guidelines represent a major rewrite and paradigm shift, and not all packages are updated to reflect this.
Older guidelines are still being kept up to date, and existing packages **MAY** use them instead of this document

> NEW: While these guidelines are in Beta, each Python package **MUST** have `BuildRequires: pyproject-rpm-macros` to access the beta macros.

## Python 3

Python 2 has been deprecated for a while now; using it requires a FESCo exception.
Text related to the py2/py3 split is generally removed or simplified.

> NEW: Fedora packages **MUST NOT** depend on other versions of the CPython interpreter than the current `python3`.
> [...]
> Packages such as `pip` or `tox`, which enable setting up isolated environments and installing third-party packages into them, **MAY**, as an exception to the rule above, use these interpreters as long as this is coordinated with the maintainers of the relevant Python interpreter.

> OLD: If a piece of software supports `python3`, it MUST be packaged for `python3`. Software using `python2` MUST NOT be newly packaged into Fedora without FESCo exception.

The following redundant note was put in the guidelines to address pushing "remove of python2 package" changes to stable branches. Since not many python2 packages are left, the note is not necessary any more:

> OLD: Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.

## Dependency Generator

The dep generator is now mandatory (i.e. upstream metadata must be used/patched):

> NEW: Packages **MUST** use the automatic Python run-time dependency generator.
Packages **SHOULD** use the opt-in build-dependency generator if possible.
he packager **MUST** inspect the generated requires for correctness. All dependencies **MUST** be resolvable within the targeted Fedora version.”
Any necessary changes **MUST** be done by patches or modifying the source (e.g. with `sed`), rather than disabling the generator. The resulting change **SHOULD** be offered to upstream. As an exception, [filtering](https://docs.fedoraproject.org/en-US/packaging-guidelines/AutoProvidesAndRequiresFiltering/) **MAY** be used for temporary workarounds and bootstrapping.
Dependencies covered by the generators **SHOULD NOT** be repeated in the `.spec` file. (For example, if the generator finds a `requests` dependency, then `Requires: python3-requests` is redundant.)

> OLD: This generator is enabled by default in Fedora. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager SHOULD opt out explicitly by adding: `%{?python_disable_dependency_generator}`

> OLD: [if using dependency generator] The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.

## Applicability

New clarification:

> NEW: Except for the two "Distro-wide guidelines", these Guidelines do not apply to simple one-file scripts or utilities[...].
> However, if [you need] a more complex Python library, the library **SHOULD** be packaged as an importable library under these guidelines.

## Naming

<details>
<summary>Clarified wording on General naming:</summary>

> NEW: A built (i.e. non-SRPM) package for a *Python library* **MUST** be named with the prefix `python3-`.
>
> A source package containing primarily a *Python library* **MUST** be named with the prefix `python-`.

> OLD: The source package for a Python library MUST be named with the `python-` prefix.

> OLD (in [Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules)): Python2 binary packages MUST be named using a `python2-` prefix.
> Python3 binary packages MUST be named with a prefix of `python3-`.

</details>

<details>
<summary>Clarified wording on `python3-` provides for importable modules:</summary>

> NEW: For any module intended to be used in Python 3 with `import MODNAME`,
the package that includes it **SHOULD** provide `python3-MODNAME`,
with underscores (`_`) replaced by dashes (`-`).

> OLD: For any module foo intended to be used in Python 3 with `import foo`, the package that includes it should provide `python3-foo`

</details>

<details>
<summary>Clarified wording on `python-`/`python3-`/`python3.X-` provides:</summary>

> NEW: For any `FOO`, a package that provides `python3-FOO`
**SHOULD** use `%py_provides` or an automatic generator
to also provide `python-FOO` and `python3.X-FOO`, where `X` is the minor version of the interpreter.
The provide **SHOULD NOT** be added manually:
if a generator or macro is not used,
do not add the `python-FOO` / `python3.X-FOO` provides at all.

> OLD: All packages that provide `python3-...` (for any `...`) SHOULD also provide `python-...` and `python3.X-...`.

> OLD: Any manually added virtual provides of `python3-...` SHOULD be done via the `%py_provides macro`.

</details>

Removed ban on `python-` Requires, since the `python-` prefix now refers to Python 3:

> OLD: Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned `python-` prefix. Dependencies on Python packages instead MUST use names beginning with `python3-`.

Best practices (SHOULDs):

> NEW: If the importable module name and the project name do not match,
users frequently end up confused.
In this case, packagers **SHOULD** ensure that upstream is aware of the problem
and (especially for new packages where renaming is feasible)
strive to get the package renamed.
The Python SIG is available for assistance.

> NEW: The Fedora package's name **SHOULD** contain the *canonical project name*.
If possible, the project name **SHOULD** be the same
as the name of the main importable module,
with underscores (`_`) replaced by dashes (`-`).

> NEW: Packages that primarily provide applications, services or any kind of executables
**SHOULD** be named according to the general [Fedora naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/)
(e.g. `ansible`).

> OLD (in [Naming guidelines](https://docs.fedoraproject.org/en-US/packaging-guidelines/Naming/#_python_modules)): The package name SHOULD reflect the upstream name of the Python module, and SHOULD generally take into account the name of the module used when importing it in Python scripts. This name will be prefixed depending on the type of the package.

Removed rules on removing `%py_provides` (which is documented as being for cases where the generator doesn't work) and `%python_provide` (which is listed in Deprecated macros):

> OLD: Packagers SHOULD try to remove explicit `%py_provides` calls for package names, but MAY preserve them if they aim for compatibility with older releases or packages without files.

> OLD (note): Historically, there was `%python_provide` macro with similar but different semantics. It still works for compatibility reasons but it is deprecated and SHOULD NOT be used and packagers SHOULD replace is with appropriate %py_provides call.

### Naming Extras

Slight addition on including devel extras (diff emphasized):

> NEW: Python packages **SHOULD** have Provides for all extras the upstream project specifies, except: those that are not useful for other packages (for example build/development requirements, commonly named `dev`, `doc` or `test`), **and those that have requirements that are not packaged in Fedora.**

> OLD: Python packages SHOULD have `Provides` for all extras the upstream project specifies, except those that are not useful for other packages (for example build/development requirements, commonly named `dev`, `doc` or `test`).

New best practice re. removing extras:

> NEW: If an existing extra is removed from an upstream project, the Fedora maintainer
**SHOULD** try to convince upstream to re-introduce it.
If that fails, the extra **SHOULD** be Obsoleted from either the main package
or another extras subpackage.

<details>
<summary>Reworded text on <code>+</code>; same meaning</summary>

> NEW: The character `+` in names of built packages (i.e. non-SRPM)
> that include `.dist-info` or `.egg-info` directories
> is reserved for *extras*
> and **MUST NOT** be used for any other purpose.
> As an exception, `+` characters **MAY** appear at the *end* of such names.

> OLD: The character `+` in names of built packages (i.e. non-SRPM) that include `.dist-info` or `.egg-info` directories is reserved for Python Extras and MUST NOT be used for any other purpose.
> As an exception, + characters are permitted at the end of the name.

> NEW: A package that provides a Python extra **MUST** provide `python3dist(DISTNAME[EXTRA])` **and** `python3.Xdist(DISTNAME[EXTRA])`, where `X` is the minor version of the interpreter, `DISTNAME` is the [canonical project name](#Canonical-project-name), and `EXTRA` is the name of a single extra. For example, `python3.9dist(requests[security])`.
> These requirements **SHOULD** be generated using the automatic dependency
generator.

> OLD: A package that provides a Python extra MUST provide `python3dist(…[…])` and `python3.Xdist(…[…])`, for example, `python3.9dist(requests[security])`. These requirements SHOULD be generated using the automatic dependency generator.

</details>

<details>
<summary>Unchanged guidelines</summary>

> NEW: A package that provides a Python extra **MUST** require the extra's main
package with exact NEVR.

> OLD: A package that provides a Python extra MUST require the extra’s main package with exact NEVR.

> NEW: A subpackage that primarily provides one Python extra **SHOULD** be named by appending "+" and the extra name to the main package name. For example, `python3-requests+security`.

> OLD: A subpackage that primarily provides one Python extra SHOULD be named by appending `+` and the extra name to the main package name. For example, `python3-requests+security`.

</details>

## Files to include

More detailed rules on what to include (and where):

> NEW: Packages **MUST** include the source file (`*.py`)
> **AND** the bytecode cache (`*.pyc`)
> for each pure-Python importable module.
> The source files **MUST** be included in the same package as the bytecode cache.

> NEW: Each Python package **MUST** include *Package Distribution Metadata*
conforming to [PyPA specifications](https://packaging.python.org/specifications/)
(specifically, [Recording installed distributons](https://packaging.python.org/specifications/recording-installed-packages/)).
The metadata **SHOULD** be included in the same subpackage
as the main importable module,
if there is one.
[...] As an exception,
the Python standard library **MAY** ship without this metadata.

> OLD: When packaging Python modules, several types of files are included: `*.py source files`[,] `*.pyc byte compiled files`[,] `*.egg-info` or `*.dist-info` files or directories. [technically, there is no MUST here]
>
> The source files MUST be included in the same package as the byte compiled versions.

New best practice: not byte-compiling scripts:

> NEW: Scripts that are not importable
(typically ones in `%{_bindir}` or `%{_libexecdir}`)
**SHOULD NOT** be byte-compiled.

<details>
<summary>Consolidated rules on content in shared directories</summary>

> NEW: Packages **MUST NOT** own shared directories owned by Python itself, such as the top-level `__pycache__` directories (`%{python3_sitelib}/__pycache__`, `%{python3_sitearch}/__pycache__`).

> NEW: Packagers **SHOULD NOT** simply glob everything under a shared directory.
> In particular, the following **SHOULD NOT** be used: `%{python3_sitelib}/*`, `%{python3_sitearch}/*`, `%{python_sitelib}/*`, `%{python_sitearch}/*`, `%{_bindir}/*`, `%pyproject_save_files '*'`, `%pyproject_save_files +auto`.

> OLD: Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used: `%{python3_sitelib}/*`, `%{python3_sitearch}/*`, `%{python_sitelib}/*`, `%{python_sitearch}/*`. And packages MUST NOT include the top-level `__pycache__` directory (see below).
>
> OLD: You MUST NOT include the directories `%{python3_sitearch}/__pycache__` or `%{python3_sitelib}/__pycache__` because they are already owned by the `python3-libs` package.

</details>

## Mandatory Requires / Provides

> NEW: **Every** package that uses Python (at runtime and/or build time), and/or installs Python modules **MUST** explicitly include `BuildRequires: python3-devel` in its `.spec` file, even if Python is not actually invoked during build time.
If the package uses an alternate Python interpreter instead of `python3` (e.g. `pypy`, `jython`, `python2.7`), it **MAY** instead require the corresponding `*-devel` package.

> NEW: As mentioned above, each Python package **MUST** explicitly BuildRequire `python3-devel`.
Packages **MUST NOT** have dependencies (either build-time or runtime) with the unversioned prefix `python-` if the corresponding `python3-` dependency can be used instead.
Packages **SHOULD NOT** have explicit dependencies (either build-time or runtime) with a minor-version prefix such as `python3.8-` or `python3.8dist(`. Such dependencies **SHOULD** instead be automatically generated or a macro should be used to get the version.
Packages **SHOULD NOT** have an explicit runtime dependency on `python3`.

> NEW: Every Python package **MUST** provide `python3dist(DISTNAME)`
**and** `python3.Xdist(DISTNAME)`,
where `X` is the minor version of the interpreter
and `DISTNAME` is the *canonical project name*
corresponding to the *dist-info metadata*.
For example, `python3-django` would provide `python3dist(django)`
and `python3.9dist(django)`.
This is generated automatically from the dist-info metadata.
The provide **SHOULD NOT** be added manually:
if the generator fails to add it, the metadata **MUST** be fixed.

## Tests

Running upstream tests is mandatory; linters are discouraged

> NEW: If a test suite exists upstream, it **MUST** be run in the `%check` section and/or in Fedora CI.
You **MAY** exclude specific failing tests.
You **MUST NOT** disable the entire testsuite or ignore the result to solve a build failure.
As an exception, you **MAY** disable tests with an appropriate `%if` conditional (e.g. bcond) when [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping).

XXX The above was changed to SHOULD after this document was created

> NEW: In `%check`, packages **SHOULD NOT** run “linters”: code style checkers, test coverage checkers and other tools that check code quality rather than functionality.

## Shebangs & Mandatory macros

The rule regarding `/usr/bin/python` has been generalized to mandatory use of some macros, and rules on shebangs were split off and made stricter.

> NEW: The following macros **MUST** be used where applicable. `%{python3}`, `%{python3_version}`, `%{python3_version_nodots}`, `%{python3_sitelib}`, `%{python3_sitearch}`.

> NEW: Shebang lines to invoke Python **MUST** use `%{python3}` as the interpreter.

> NEW: Shebang lines to invoke Python **SHOULD** be `#!%{python3} -%{py3_shebang_flags}` and it **MAY** include extra flags.
If the default flags from `%{py3_shebang_flags}` are not desirable, packages **SHOULD** explicitly redefine the macro to remove them.

> OLD: Packages in Fedora MUST NOT use `/usr/bin/python`. Instead packages for Python 3 MUST use `/usr/bin/python3` (even if upstream supports both Python 2 and 3). As a result of that `/usr/bin/python` (as well as `/usr/bin/env python` and similar) MUST NOT be used in shebang lines or as a dependency of a package. All uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed (for example by using the `%py3_shebang_fix` macro in the spec file).

## PyPI sources

PyPI archives that omit tests/docs are discouraged:

> NEW: Packages **MAY** use sources from PyPI.
However, packages **SHOULD NOT** use an archive that omits test suites, licences and/or documentation present in other source archives.

## Specific macros

Redefining `%python3_pkgversion` is banned:

> NEW: Packages in Fedora **MAY** use [`%{python3_pkgversion}`] (e.g. in package names: `python%{python3_pkgversion}-requests`), but **MUST NOT** redefine it.

<details>
<summary>Note on `%pyproject_save_files` features unsuitable for Fedora</summary>

> NEW: Note that README and licence files are not included.
Also, while the macro allows including executable and other files (using the `+auto` flag), this feature **MUST NOT** be used in Fedora.

</details>

## Cython

<details>
<summary>Clarification; same meaning</summary>

> NEW: Tightening the [general Fedora policy](https://docs.fedoraproject.org/en-US/packaging-guidelines/what-can-be-packaged/#_pregenerated_code), packages **MUST NOT** use files pre-generated by Cython. These **MUST** be deleted in `%prep` and regenerated during the build.
As an exception, these sources **MAY** be used temporarily to prevent build time circular dependencies by following the [bootstrapping](https://docs.fedoraproject.org/en-US/packaging-guidelines/#bootstrapping) guidelines.

> OLD: Tightening the general Fedora policy, packages MUST NOT use pre-generated Cython sources. They MUST be deleted in %prep and regenerated during the build. Any exception to this rule should be considered a bootstrapping.

</details>

## Best practices

New note on when to prefer `python3 -m tool`:

> NEW: Every executable `TOOL` for which the current version of Python matters **SHOULD** also be invokable by `python3 -m TOOL`.
If the software doesn't provide this functionality, packagers **SHOULD** ask the upstream to add it.