owned this note
owned this note
Published
Linked with GitHub
# Fedora Python interpreters maintenance guide
:::warning
:atom_symbol: This process is a nuclear power plant. We don’t expect casual contributors to follow it.
By *we*, we mean Red Hat's python-maint team, which currently handles Python interpreter maintenance in Fedora.
If you're not in the team, you're certainly welcome to look how we do things and follow/improve this guide (or parts of it) if you want to, but your time might be better spent learning other things :)
It's perfectly OK to just file bugs in Bugzilla, submit Pull Requests in Pagure, and talk on the mailing list.
:::
:::info
:construction: This document is work in progress. Python-maint members should have write access—if it doesn't work, ask @torsava. Feel free to edit to fix typos or reword sentences. Keep discussion in e-mail or in person, please avoid comments here, they are hard to follow. Feel free to add XXX notes.
:::
## What is covered by this guide?
Follow the guide when you need to:
- Update (rebase) Python X.Y in Fedora to a newer version.
- Add/remove/modify a Python X.Y patch in Fedora.
- Do adjustments in spec file of Python X.Y in Fedora.
- Do adjustments in other dist-git (src.fedoraproject.org) files of Python X.Y in Fedora.
Things not yet covered:
- Introducing a new Python X.Y version into Fedora.
- Removing an old Python X.Y version from Fedora.
## Which dist-git component?
| | Fedora 37 🔧 | Fedora 36 | Fedora 35 | EL note 🎩 |
| ------ | ------------------ | -------------- | ----------------- | ----------------- |
| 3.11 🔧 | **python3.11** | python3.10 | python3.10 | |
| 3.10 | python3.10 | **python3.10** | **python3.10** | |
| 3.9 🔒 | python3.9 | python3.9 |python3.9 | RHEL9, 8 module |
| 3.8 🔒 | python3.8 | python3.8 | python3.8 | 8 module, SCL |
| 3.7 🔒 | python3.7 | python3.7 | python3.7 | |
| 3.6 💀 | python3.6 | python3.6 | python3.6 | RHEL7, 8, SCL |
| 3.5 💀 | ☠ | ☠ | ☠ | |
| 3.4 💀 | ☠ | ☠ | ☠ | EPEL7 |
| 2.7 💀 | python2.7 | python2.7 | python2.7 |7, 8 module, SCL |
| 2.6 💀 | ☠ | ☠ | ☠ |RHEL6 |
- **bold** means "main Python" (see later)
- ☠ retired
- 🕒 planned
- 💀 not supported at all upstream
- 🔒 security only support upstream
- 🔧 development release
- 🎩 RHEL/CentOS/EPEL/SCL components would make the table overly complicated, the EL column is informative only
The above Python releases are about CPython. However, there are also other Python implementations in Fedora: :snake: PyPy2.7 (component `pypy`) and :snake: PyPy3.X (components `pypy3.X`), which are not covered by this guide.
## Where should I fix a problem?
When applicable, always fix issue in upstream first. Then if possible, wait for the next CPython release that will contain the fix. If you need an upstream fix in an older Python release, help backporting it in upstream if not already done. Note that some Python versions are security only or even EOL, this means backporting in upstream and waiting for the next version is not always an option.
If waiting for the next CPython release is not an option, backport the merged changes to Fedora. Only in extreme/time-critical circumstances can you fix the problem in a downstream patch first and then work to upstream it.
:::info
:snake: See https://devguide.python.org/#branchstatus
:::
### The "main" Python
Each Fedora release has only one "main" Python version. This is the version you'll get when you `dnf install python3` or use `/usr/bin/python3` in a given Fedora release. When the version is upgraded, this is coordinated through the Fedora change process. See for example: https://fedoraproject.org/wiki/Changes/Python3.9
The established procedure is that we only ship stable or release candidate versions of Python in released Fedoras. Therefore we look at the planned release schedule of the Python version and compare with the Fedora release schedule. We check if the first release candidate of that Python version is planned to be shipped well before the Final Freeze of the Fedora release (in case there are slip ups). If so, we upgrade the "main" Python to the latest development release of the Python version (usually a beta or rc).
:::info
:arrow_right: See https://fedoraproject.org/wiki/SIGs/Python/UpgradingPython for the process of upgrading the "main" Python to a newer version.
:::
Since Fedora 33, you can see the definition of the "main" Python version in the `python-rpm-macros` component (`%__default_python3_version`). In Fedora 32 and earlier, the `python3` component is the "main" Python.
General issues should be fixed in the "main" Python versions for each Fedora, starting from the highest affected Fedora version. Changes are only backported to older Fedoras when backwards compatible.
Changes in the "main" Python can affect critical Fedora components, such as dnf, anaconda, fedpkg stack etc. When creating bodhi updates for stable Fedora releases, set high karma limits (countless +1s without context are very common).
### The "next main" Python
The "next main" Python version is a package containing Python 3.N+1 (where 3.N is the "main" Python in rawhide). Such package might not exist yet, it's created during the alpha phase of the next Python release upstream.
- If you're fixing a packaging problem, you should fix the "next main" Python in addition to fixing all the "main" Pythons. (Otherwise you risk losing the fix when the "next main" Python becomes the "main" Python.)
- If you're fixing an upstream issue, in most cases you can wait for the next upstream release that will contain the fix.
### Other Python versions
We don't generally actively fix non-packaging problems in the "non-main" Python packages. For upstream supported Python versions, the fix will get to Fedora with the next rebase. For EOL versions upstream, we don't offer additional support, we only provide them as-is.
As an exception, we generally backport fixes for problems that makes the packages fail to build. When not trivial, we skip some tests with a rationale.
For Python 2.7, we might backport security fixes from RHEL (this has not happened yet).
> XXX Replace with this?
> For older Pythons, we might backport security fixes from RHEL/EPEL (or put those in Fedora first).
Packaging problems are usually fixed in rawhide, backported to older Fedora releases only if they affect their users (Python developers using multiple Python versions to test their software). We want to fix packaging problems even in older Python versions that are EOL upstream, so that developers can use them for testing.
## Where do we update (rebase) to new releases
XXX When to update and where to find out the dates? See also https://github.com/fedora-python/python-release-schedule-ical
### Stable versions of Python
#### Bugfix and security releases
When a new stable version of Python bugfix or security release (e.g. 3.8.1) is released, we update Python X.Y to that version as soon as possible everywhere, starting with rawhide.
#### Release candidates of bugfix and security releases
When a new release candidate version of Python bugfix or security release (e.g. 3.8.1rc1) is released, we update Python X.Y to that rc version as soon as possible in rawhide. For branched Fedora, we treat it like rawhide until the Beta freeze is approaching. Consider if the final version will be released before the Fedora Beta freeze with time to spare. We don't generally update to bugfix/security release candidates in stable Fedoras.
Updating to release candidate versions allows to discover problems early, but we don't want to ship them to users of stable Fedoras in case we actually discover issues.
:::info
:bulb: This is especially useful for Python versions that are "main" in any supported Fedora version or shipped in RHEL. For example Python 3.5 or 3.7 are neither and hence a release candidate update might get skipped for capacity reasons.
:::
### Development versions of Python
For Python versions that have not yet reached 3.XX.0 final, we update to a new alpha, beta, rc version as soon as possible. We start with rawhide and proceed with older Fedora releases.
:::warning
:warning: When the development version is also the "main" version on rawhide/branched, extra caution needs to be taken in there (wrt ABI compatibility, bytecode magic numbers, etc. -- sometimes a mass rebuild of some Python packages is required). The [upgrade guide](https://fedoraproject.org/wiki/SIGs/Python/UpgradingPython) covers this (or at least should).
:::
## Where do we do enhancements
### Python enhancements
We do them in upstream. Possibly, we use Fedora to pioneer an upstream change, see for example:
- core change: https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
- stdlib enhancement: https://pypi.org/project/compileall2/
### Packaging enhancements
Generally, we only do packaging enhancements in rawhide (possibly also in branched before the Beta Freeze).
Backwards compatible enhancements can optionally be backported to older Fedora, usually when deemed useful or when easier to backport together with a bugfix or a rebase.
Backwards incompatible enhancements (unusual but possible) are not backported.
Generally we start with the highest relevant Python version and optionally we proceed to lower versions later. It's also possible to start with the "main" Python (if not yet the highest), however, it's not advised as one may forget to "backport" the enhancement to the highest Python version.
## Patches
Maintenance cost of every downstream-only patch is high as we have to keep rebasing and maitaining it with every new Python version. We try to avoid such patches if at all possible and fix the issue instead in upstream. However, we use patches to:
- backport not yet released fixes
- backport fixes for versions that are EOL upstream
- pioneer future upstream changes in Fedora (XXX link to the section above)
- temporarily workaround/revert breaking upstream changes before a proper fix exists
We do *not* use patches for Fedora/EL-only changes if there is no plan to eventually bring those to upstream. Every change should either be temporary or a (long term) plan must exist to include it upstream.
XXX (Security) fixes that are only relevant to an EOL version of Python are always Fedora/EL-only -- an exception?
### Tracking patches
We keep track of our patches in https://fedoraproject.org/wiki/SIGs/Python/PythonPatches. The patch numbers are shared between Python versions packaged in Fedora, RHEL, EPEL, SCLs, modules etc. A patch with a common number doesn't need to be bit-by-bit identical between different Python components/branches, but it must have the same purpose.
We keep our Fedora patches in https://github.com/fedora-python/cpython -- the main purpose is to make creating and rebasing patches easier.
In the repo there are branches named `fedora-X.Y`, where X.Y is the Python version (e.g. `fedora-3.8`). Such branch is forked from the latest tag of that Python minor version (e.g. `v3.8.3`) from CPython upstream git repo: https://github.com/python/cpython. On top of that tag we put the current patches for Fedora. We put the patch number at the start of the commit message (e.g. `00666: The patch of the beast`).
When a new Python version is released, the patches are rebased via git on top of the new latest tag for that upstream release, tagged with a new `fedora-` tag and force pushed to the `fedora-X.YY` branch for later use. The `fedora-X.YY.ZZ-R` tags serve as history references, so we can see the content otherwise erased by the force push.
```console
[cpython (...)]$ git remote -v
fedora-python git@github.com:fedora-python/cpython.git (fetch)
fedora-python git@github.com:fedora-python/cpython.git (push)
origin git://github.com/python/cpython.git (fetch)
origin git://github.com/python/cpython.git (push)
<myname> git@github.com:<myname>/cpython.git (fetch)
<myname> git@github.com:<myname>/cpython.git (push)
[cpython (...)]$ git switch fedora-3.8
[cpython (fedora-3.8)]$ git fetch fedora-python
[cpython (fedora-3.8)]$ git reset --hard fedora-python/fedora-3.8
[cpython (fedora-3.8)]$ git fetch origin # the python/cpython repo
[cpython (fedora-3.8)]$ git rebase -i v3.8.3 # the new Python release
... handle rebase ...
[cpython (fedora-3.8)]$ git tag fedora-3.8.3-1
[cpython (fedora-3.8)]$ git push fedora-python fedora-3.8.3-1
[cpython (fedora-3.8)]$ git push --force -u fedora-python fedora-3.8
```
Patches can change during one upstream release, therefore the git tag includes the Fedora release number (without the dist tag), e.g. `fedora-3.8.3-1` for python3.8-3.8.3-1.fc33.
In theory, patches can differ between Fedora releases. We generally assume the git repo represents rawhide. If needed, new branches or tags will be created, named according to the Fedora version they are for (there is no scheme for this, it has never been needed so far).
See existing tags for inspiration, e.g.:
- [fedora-3.8.3-1](https://github.com/fedora-python/cpython/commits/fedora-3.8.3-1)
- [fedora-3.9.0a3-1](https://github.com/fedora-python/cpython/commits/fedora-3.9.0a3-1)
:::danger
:no_good: We don't expect casual contributors to follow the process. We'll take their contributions and process them for them if needed. It is **not OK** to ask them to register patch numbers or to use our GitHub repo. OTOH It is OK to mention the process and offer guidance. This documentation is now targeted to the Python Maint team members. When it reaches general public quality, we *might* reconsider this rule.
:::
### Converting git commits to dist git patch files
To import patches from a GitHub branch to dist-git (and the spec file), we use a script called [importpatches](https://github.com/fedora-python/importpatches).
See its README for installation and usage.
The script automates a previously manual process. Always check its results, and if necessary, amend them or (better) improve the script.
Assuming you have `importpatches` in `PATH` and have it configured, then the process is roughly:
* Create the `fedora-X.Y-R` tag in your local CPython clone (see above, or *Creating new patches* below).
* In your local dist-git clone:
```console
$ git pull
$ rpmdev-bumpspec *.spec -c '... your comment ...'
$ importpatches
$ git status; git diff; ... # examine the result
$ git add .; git commit -m '... your comment ...'
```
then push to your fork & create a PR.
<details>
<summary>Notes on the manual process (you should not need to know this)</summary>
We use `git format-patch` to create patch files:
```console
[cpython (fedora-3.8)]$ rm -f *.patch
[cpython (fedora-3.8)]$ git format-patch --no-numbered v3.8.3
0001-00001-Fixup-distutils-unixccompiler.py-to-remove-sta.patch
0002-00102-Change-the-various-install-paths-to-use-usr-li.patch
0003-00111-Don-t-try-to-build-a-libpythonMAJOR.MINOR.a.patch
0004-00189-Instead-of-bundled-wheels-use-our-RPM-packaged.patch
0005-00251-Change-user-install-location.patch
0006-00274-Upstream-uses-Debian-style-architecture-naming.patch
0007-00328-Restore-pyc-to-TIMESTAMP-invalidation-mode-as-.patch
```
(The option `--no-numbered` relates to the subject lines inside the patches. There's currently no known way to generate unnumbered filenames.)
Unfortunately, the filenames in dist git do not correspond to the filenames generated by `git format-patch`. When moving the patches to dist git, one needs to get creative (this is the best place for improvements of the process), for example (from the dist-git folder):
```console
[python3.8 (rawhide)]$ CPYTHON=.../cpython # path to cloned forked repo with the git formated patches
[python3.8 (rawhide)]$ for p in $(ls ${CPYTHON}/*.patch | cut -d- -f2); do cp ${CPYTHON}/*-${p}-*.patch ${p}-*.patch; done
```
Don't forget to manually check for obsoleted patches, `git rm` them from dist-git and to copy and `git add` any new patches. You will also need to add and/or remove patches into/from the spec file.
:::warning
:spiral_note_pad: When the rebase is nontrivial, it's a good idea to note the changes in the dist git commit message. Some examples:
Patch 102 was dropped becasue Python now fixes the problem differently, see bpo-1294959.
Patch 123 was reworked beacuse upstream code was refactored wrt PEP 456.
Patches 348 and 351 were merged upstream.
:::
:::danger
:inbox_tray: **Patch 189** includes bundled setuptools/pip version information. The `bundled()` provides in spec need to be updated together with the patch when the versions are bumped by upstream.
:::
:::info
:zombie: **Python 2**
The Python 2 patches include the name of the patch as the commit message title, and the comment that goes into the spec file as the body. They can be made into patches with:
```bash
[cpython (fedora-2.7)]$ rm -f *.patch
[cpython (fedora-2.7)]$ git format-patch --no-numbered v2.7.18
```
```bash
[python3.8 (rawhide)]$ CPYTHON=.../cpython # path to cloned forked repo with the git formated patches (store this in your .bashrc file to boost productivity)
[python3.8 (rawhide)]$ for p in $(ls ${CPYTHON}/*.patch); do cp "$p" $(grep Subject: "$p" | sed -e's/^.*\[PATCH\] \(.*\)/\1/'); done
```
(This works for most of them. If you're touching the problematic one, fix it.)
:::
</details>
### Creating new patches
1. Reserve a new patch number (or reuse a number reserved for this problem) on the [wiki patch list](https://fedoraproject.org/wiki/SIGs/Python/PythonPatches).
2. Fork `fedora-X.YY`, add the commit either by one of the following:
* `git cherry-pick`/`git revert` an upstream commit
* `git cherry-pick` a downstream commit from a different `fedora-X.YY` branch
* create a brand new commit from scratch
3. Ensure the commit message starts with the patch number (e.g. by using `git commit --amend` on the cherry-picked upstream commit).
4. *(Optional)* Send a pull request with this to `fedora-X.YY` branch.
If the patch number is lower than the highest existing patch number currently in `fedora-X.YY`, keep your commit on top for easier Pull Request review. It will be reordered once merged.
You can use your open GitHub pull request to create patch files for new a dist-git pull request on [src.fedoraproject.org](https://src.fedoraproject.org/), or you can first wait for the GitHub pull request to be reviewed and merged.
After the pull request is merged, the branch must be rebased to properly reorder patches if necessary, and new tags must be created according to the related dist git change. Coordinate with the reviewer on who does this.
Alternatively, push directly to `fedora-X.YY` but be prepared to rebase if there is feedback.
### Modifying existing patches
:::info
:point_up: Rebasing Python to a newer version is not considered modifying patches and was already covered in previous sections.
::::
Follow the creating patches section. Send pull request with a fixup commit on top of the `fedora-X.YY` branch. When a new commit message is required, include it in the pull request description and/or discussion and make sure it is updated after the PR is merged, and the branch rebased and fixup commit squashed. Example:
```console
[cpython (fedora-3.8)]$ git fetch fedora-python
[cpython (fedora-3.8)]$ git reset --hard fedora-python/fedora-3.8
[cpython (fedora-3.8)]$ git log --oneline
aedd897c63 (HEAD -> fedora-3.8, fedora-python/fedora-3.8) 00328: Restore pyc to TIMESTAMP invalidation mode as default in rpmbuild
3172104314 00274: Upstream uses Debian-style architecture naming, change to match Fedora
197b8de27e 00251: Change user install location
36f1f2b462 00189: Instead of bundled wheels, use our RPM packaged wheels
50236468e8 00111: Don't try to build a libpythonMAJOR.MINOR.a
be6b980310 00102: Change the various install paths to use /usr/lib64/ instead or /usr/lib/
08c67bfedd 00001: Fixup distutils/unixccompiler.py to remove standard library path from rpath Was Patch0 in ivazquez' python3000 specfile
6f8c8320e9 (tag: v3.8.3) Python 3.8.3
...
[cpython (fedora-3.8)]$ git switch -c myfix
... edit the files relevant for your change and stage them in git...
[cpython (myfix)]$ git commit --fixup=3172104314 # use the hash of the original commit you want to modify
[cpython (myfix)]$ git log --oneline
e9557758a3 (HEAD -> myfix) fixup! 00274: Upstream uses Debian-style architecture naming, change to match Fedora
...
3172104314 00274: Upstream uses Debian-style architecture naming, change to match Fedora
...
```
Rationale: When you modify an existing commit, the git history of every commit afterwards is changed. That is very unpleasant to work with when reviewing a Pull Request, for example because it's not easy to tell if the commits after the changed one were changed as well or just rebased.
:::info
:bulb: A rebased branch is useful to generate patch files for dist-git, hence it is not a bad idea to also have a rebased branch ready together with the Pull Request and mention it in the PR description. This also makes it easier for the reviewer to see the modified commit as a single unit in addition to your fixup.
```console
[cpython (myfix)]$ git switch -c myfix_rebased
Switched to a new branch 'myfix_rebased'
[cpython (myfix_rebased)]$ git rebase --autosquash v3.8.3 # use the latest upstream tag
Successfully rebased and updated refs/heads/myfix_rebased.
```
:::
### Removing no longer needed patches
:::info
:point_up: Rebasing Python to a newer version that already includes a patch is not considered removing patches in this context and was already covered in previous sections.
::::
Follow the creating patches section. Send pull request with a revert commit. It will be rebased after the PR is merged.
For a dist-git pull request, feel free to just remove the patch without regenerating the others (by `git format-patch`). They will be regenerated with the next change. However, don't forget to also do the change on GitHub, so the commit/patch is not reintroduced later.
## How to backport changes
If you want to backport a change from one Python/Fedora version to another, it's tedious to manually have to edit the files over and over. But you can use git to make it easier and to preserve commit authorship, messages etc.
::: warning
:alien: When backporting stuff **consider future backports** as well. If possible strive to keep the files in older Fedoras **as identical as possible** to the new ones. This includes the release number and changelog entries. **Don't skip commits just because they are not needed.** As an example, if the rawhide commit you are about to backport bumps the release number from 2 to 3 and you backport the commit to an older release, where the release number is 1, see if the commit that bumped it to 2 is backwards compatible and backport it as well if it is. If not, skip it, but make the release number 3, so the next backport won't conflict.
:::
### Within the same component: Prefer FF merge, cherry-pick otherwise
In this section, we'll use `rawhide` as the reference to the branch you take the backport from and `fcXX` as a reference to the branch you want the backport to land into. However, this guide also applies to other kinds of backports (for example from `fc34` to `fc33` as well).
When you want to backport a commit within the same component, always consider fast-forward (**FF**) merging if possible. Ask yourself the following questions:
1. Is the fast-forward merge **technically possible**? I.e. is the HEAD of the`fcXX` branch an ancestor of the HEAD of the `rawhide` branch?
2. Are all other commits that will be FF merged together with my backport (if any) **backwards compatible**?
If the answer to all of these questions is yes, use a fast-forward merge.
If the answer to any of these questions is no, use a cherry-pick instead.
Note there is no harm in backporting moot commits (such as backporting "Fedora 35 mass rebuild" to Fedora 34) -- trying to skip such commits only makes things harder for both you and the person who will backport the next thing.
> **Why not 3-way merges?**
> We have a linear changelog that should match what's built in Koji. 3-way merges make the history hard to follow.
> 3-way merges are also much harder to manage in RHEL.
>
> There are some legitimate uses for 3-way merges, but they're not appropriate for day-to-day maintenance.
:::info
:mage: When backporting to multiple branches, consider fast-forward merge between them if possible. For example if you cannot merge `rawhide` into `f32` and you need to use cherry-pick, you don't necessarily need to cherry-pick to `f31` separately, but you can FF merge `f32` into `f31` after the merge. Often, you can even **open multiple pull requests from a single feature branch** to more Fedora branches -- when possible, this is the preferred approach because we can see multiple CI results before we decide whether to merge as is.
:::
### How to cherry pick commits from different dist-git components
If you need to cherry-pick a commit from one component to another, a trivial git merge or cherry-pick does not work. Different components have different git repos and also different spec filenames, which complicates cherry-picking.
The tested way of cherry-picking commits from one component to another is:
1. Use `git format-patch` to create patch file(s) of the commits you want to backport.
2. Find and replace the spec filename (and possibly other derived filenames) inside the patch file(s).
3. Use `git am` (possibly with `--reject`) to apply the commits into the target dist-git repo.
Changelog entries and release bumps are often causing conflicts that need to be solved manually.
When cherry-picking commits across components that represent the same Python version (e.g. from `python3.6` to `python36`), the changes in patch files will likely apply cleanly. However when backporting changes from one Python version to another (e.g. from `python3.9` to `python3.8`), it may be easier to avoid manually resolve cherry-pick conflicts in the dist-git Python patch files (which can lead to serious brain damage, because it's applying patches onto patches). Use the GitHub fork to re-generate them instead. (XXX link to the above guide).
:::warning
:package: When cherry-picking commits that update Python to a newer version from one component to another, it is necessary to re-upload new source tarballs (`fedpkg new-sources`), because the *lookaside cache* is namespaced by component name.
:::
#### Ferrypick
There is a alpha-quality tool that can help you do steps 1, 2 & 3 from the previous section to cherry-pick commits from a different dist-git component. You can use ferrypick to cherry-pick changes from the same component as well if you prefer.
https://github.com/fedora-python/ferrypick
Use it as this: XXX example from readme.
:::danger
:bug: Ferrypick was created at 2:00 AM within 15 minutes. It will likely fail, so please report issues when it does.
:::
