Fedora Python interpreters maintenance guide

What is covered by this guide?

Follow the guide when you need to:

• Update (rebase) Python X.Y in Fedora to a newer version.
• Add/remove/modify a Python X.Y patch in Fedora.
• Do adjustments in spec file of Python X.Y in Fedora.
• Do adjustments in other dist-git (src.fedoraproject.org) files of Python X.Y in Fedora.

Things not yet covered:

• Introducing a new Python X.Y version into Fedora. (See: New Python bootstrap in Fedora)
• Removing an old Python X.Y version from Fedora.

Which dist-git component?

• bold means "main Python" (see later)
• ☠ retired
• 🕒 planned
• 💀 not supported at all upstream
• 🔒 security only support upstream
• 🔧 development release
• 🎩 RHEL/CentOS/EPEL/SCL components would make the table overly complicated, the EL column is informative only

The above Python releases are about CPython. However, there are also other Python implementations in Fedora:

Where should I fix a problem?

When applicable, always fix issue in upstream first. Then if possible, wait for the next CPython release that will contain the fix. If you need an upstream fix in an older Python release, help backporting it in upstream if not already done. Note that some Python versions are security only or even EOL, this means backporting in upstream and waiting for the next version is not always an option.

If waiting for the next CPython release is not an option, backport the merged changes to Fedora. Only in extreme/time-critical circumstances can you fix the problem in a downstream patch first and then work to upstream it.

The "main" Python

Each Fedora release has only one "main" Python version. This is the version you'll get when you dnf install python3 or use /usr/bin/python3 in a given Fedora release. When the version is upgraded, this is coordinated through the Fedora change process. See for example: https://fedoraproject.org/wiki/Changes/Python3.9

The established procedure is that we only ship stable or release candidate versions of Python in released Fedoras. Therefore we look at the planned release schedule of the Python version and compare with the Fedora release schedule. We check if the first release candidate of that Python version is planned to be shipped well before the Final Freeze of the Fedora release (in case there are slip ups). If so, we upgrade the "main" Python to the latest development release of the Python version (usually a beta or rc).

Since Fedora 33, you can see the definition of the "main" Python version in the python-rpm-macros component (%__default_python3_version). In Fedora 32 and earlier, the python3 component was the "main" Python.

General issues should be fixed in the "main" Python versions for each Fedora, starting from the highest affected Fedora version. Changes are only backported to older Fedoras when backwards compatible.

Changes in the "main" Python can affect critical Fedora components, such as dnf, anaconda, fedpkg stack etc. When creating bodhi updates for stable Fedora releases, set high karma limits (countless +1s without context are very common).

The "next main" Python

The "next main" Python version is a package containing Python 3.N+1 (where 3.N is the "main" Python in rawhide). Such package might not exist yet, it's created during the alpha phase of the next Python release upstream.

• If you're fixing a packaging problem, you should fix the "next main" Python in addition to fixing all the "main" Pythons. (Otherwise you risk losing the fix when the "next main" Python becomes the "main" Python.)
• If you're fixing an upstream issue, in most cases you can wait for the next upstream release that will contain the fix.

Other Python versions

We don't generally actively fix non-packaging problems in the "non-main" Python packages. For upstream supported Python versions, the fix will get to Fedora with the next rebase. For EOL versions upstream, we don't offer additional support, we only provide them as-is.

As an exception, we generally backport fixes for problems that makes the packages fail to build. When not trivial, we skip some tests with a rationale.

For Python 2.7, we might backport security fixes from RHEL (this has not happened yet).

XXX Replace with this?
For older Pythons, we might backport security fixes from RHEL/EPEL (or put those in Fedora first).

Packaging problems are usually fixed in rawhide, backported to older Fedora releases only if they affect their users (Python developers using multiple Python versions to test their software). We want to fix packaging problems even in older Python versions that are EOL upstream, so that developers can use them for testing.

Where do we update (rebase) to new releases

XXX When to update and where to find out the dates? See also https://github.com/fedora-python/python-release-schedule-ical

Stable versions of Python

Bugfix and security releases

When a new stable version of Python bugfix or security release (e.g. 3.8.1) is released, we update Python X.Y to that version as soon as possible everywhere, starting with rawhide.

Release candidates of bugfix and security releases

When a new release candidate version of Python bugfix or security release (e.g. 3.8.1rc1) is released, we update Python X.Y to that rc version as soon as possible in rawhide. For branched Fedora, we treat it like rawhide until the Beta freeze is approaching. Consider if the final version will be released before the Fedora Beta freeze with time to spare. We don't generally update to bugfix/security release candidates in stable Fedoras.

Updating to release candidate versions allows to discover problems early, but we don't want to ship them to users of stable Fedoras in case we actually discover issues.

Development versions of Python

For Python versions that have not yet reached 3.XX.0 final, we update to a new alpha, beta, rc version as soon as possible. We start with rawhide and proceed with older Fedora releases.

Where do we do enhancements

Python enhancements

We do them in upstream. Possibly, we use Fedora to pioneer an upstream change, see for example:

• core change: https://fedoraproject.org/wiki/Changes/python3_c.utf-8_locale
• stdlib enhancement: https://pypi.org/project/compileall2/

Packaging enhancements

Generally, we only do packaging enhancements in rawhide (possibly also in branched before the Beta Freeze).

Backwards compatible enhancements can optionally be backported to older Fedora, usually when deemed useful or when easier to backport together with a bugfix or a rebase.

Backwards incompatible enhancements (unusual but possible) are not backported.

Generally we start with the highest relevant Python version and optionally we proceed to lower versions later. It's also possible to start with the "main" Python (if not yet the highest), however, it's not advised as one may forget to "backport" the enhancement to the highest Python version.

Patches

Maintenance cost of every downstream-only patch is high as we have to keep rebasing and maitaining it with every new Python version. We try to avoid such patches if at all possible and fix the issue instead in upstream. However, we use patches to:

• backport not yet released fixes
• backport fixes for versions that are EOL upstream
• pioneer future upstream changes in Fedora (XXX link to the section above)
• temporarily workaround/revert breaking upstream changes before a proper fix exists

We do not use patches for Fedora/EL-only changes if there is no plan to eventually bring those to upstream. Every change should either be temporary or a (long term) plan must exist to include it upstream.

XXX (Security) fixes that are only relevant to an EOL version of Python are always Fedora/EL-only – an exception?

Tracking patches

We keep track of our patches in https://fedoraproject.org/wiki/SIGs/Python/PythonPatches. The patch numbers are shared between Python versions packaged in Fedora, RHEL, EPEL, SCLs, modules etc. A patch with a common number doesn't need to be bit-by-bit identical between different Python components/branches, but it must have the same purpose.

We keep our Fedora patches in https://github.com/fedora-python/cpython – the main purpose is to make creating and rebasing patches easier.

In the repo there are branches named fedora-X.Y, where X.Y is the Python version (e.g. fedora-3.8). Such branch is forked from the latest tag of that Python minor version (e.g. v3.8.3) from CPython upstream git repo: https://github.com/python/cpython. On top of that tag we put the current patches for Fedora. We put the patch number at the start of the commit message (e.g. 00666: The patch of the beast).

When a new Python version is released, the patches are rebased via git on top of the new latest tag for that upstream release, tagged with a new fedora- tag and force pushed to the fedora-X.YY branch for later use. The fedora-X.YY.ZZ-R tags serve as history references, so we can see the content otherwise erased by the force push.

[cpython (...)]$ git remote -v
fedora-python	git@github.com:fedora-python/cpython.git (fetch)
fedora-python	git@github.com:fedora-python/cpython.git (push)
origin		git://github.com/python/cpython.git (fetch)
origin		git://github.com/python/cpython.git (push)
<myname>	git@github.com:<myname>/cpython.git (fetch)
<myname>	git@github.com:<myname>/cpython.git (push)

[cpython (...)]$ git switch fedora-3.8
[cpython (fedora-3.8)]$ git fetch fedora-python
[cpython (fedora-3.8)]$ git reset --hard fedora-python/fedora-3.8

[cpython (fedora-3.8)]$ git fetch origin  # the python/cpython repo
[cpython (fedora-3.8)]$ git rebase -i v3.8.3  # the new Python release
...  handle rebase ...

[cpython (fedora-3.8)]$ git tag fedora-3.8.3-1
[cpython (fedora-3.8)]$ git push fedora-python fedora-3.8.3-1
[cpython (fedora-3.8)]$ git push --force -u fedora-python fedora-3.8

Patches can change during one upstream release, therefore the git tag includes the Fedora release number (without the dist tag), e.g. fedora-3.8.3-1 for python3.8-3.8.3-1.fc33.

In theory, patches can differ between Fedora releases. We generally assume the git repo represents rawhide. If needed, new branches or tags will be created, named according to the Fedora version they are for (there is no scheme for this, it has never been needed so far).

See existing tags for inspiration, e.g.:

• fedora-3.8.3-1
• fedora-3.9.0a3-1

Converting git commits to dist git patch files

To import patches from a GitHub branch to dist-git (and the spec file), we use a script called importpatches.
See its README for installation and usage.

The script automates a previously manual process. Always check its results, and if necessary, amend them or (better) improve the script.

Assuming you have importpatches in PATH and have it configured, then the process is roughly:

• Create the fedora-X.Y-R tag in your local CPython clone (see above, or Creating new patches below).
• In your local dist-git clone:

  ​​​$ git pull
  ​​​$ rpmdev-bumpspec *.spec -c '... your comment ...'
  ​​​$ importpatches
  ​​​$ git status; git diff; ... # examine the result
  ​​​$ git add .; git commit -m '... your comment ...'
  

  then push to your fork & create a PR.

[cpython (fedora-3.8)]$ rm -f *.patch
[cpython (fedora-3.8)]$ git format-patch --no-numbered v3.8.3
0001-00001-Fixup-distutils-unixccompiler.py-to-remove-sta.patch
0002-00102-Change-the-various-install-paths-to-use-usr-li.patch
0003-00111-Don-t-try-to-build-a-libpythonMAJOR.MINOR.a.patch
0004-00189-Instead-of-bundled-wheels-use-our-RPM-packaged.patch
0005-00251-Change-user-install-location.patch
0006-00274-Upstream-uses-Debian-style-architecture-naming.patch
0007-00328-Restore-pyc-to-TIMESTAMP-invalidation-mode-as-.patch

[python3.8 (rawhide)]$ CPYTHON=.../cpython  # path to cloned forked repo with the git formated patches
[python3.8 (rawhide)]$ for p in $(ls ${CPYTHON}/*.patch | cut -d- -f2); do cp ${CPYTHON}/*-${p}-*.patch ${p}-*.patch; done

​​​​Patch 102 was dropped becasue Python now fixes the problem differently, see bpo-1294959.
​​​​Patch 123 was reworked beacuse upstream code was refactored wrt PEP 456.
​​​​Patches 348 and 351 were merged upstream.

[cpython (fedora-2.7)]$ rm -f *.patch
[cpython (fedora-2.7)]$ git format-patch --no-numbered v2.7.18

[python3.8 (rawhide)]$ CPYTHON=.../cpython  # path to cloned forked repo with the git formated patches (store this in your .bashrc file to boost productivity)
[python3.8 (rawhide)]$ for p in $(ls ${CPYTHON}/*.patch); do cp "$p" $(grep Subject: "$p" | sed -e's/^.*\[PATCH\] \(.*\)/\1/'); done

Creating new patches

1. Reserve a new patch number (or reuse a number reserved for this problem) on the wiki patch list.
2. Fork fedora-X.YY, add the commit either by one of the following:

• git cherry-pick/git revert an upstream commit
• git cherry-pick a downstream commit from a different fedora-X.YY branch
• create a brand new commit from scratch

3. Ensure the commit message starts with the patch number (e.g. by using git commit --amend on the cherry-picked upstream commit).
4. (Optional) Send a pull request with this to fedora-X.YY branch.

If the patch number is lower than the highest existing patch number currently in fedora-X.YY, keep your commit on top for easier Pull Request review. It will be reordered once merged.

You can use your open GitHub pull request to create patch files for new a dist-git pull request on src.fedoraproject.org, or you can first wait for the GitHub pull request to be reviewed and merged.

After the pull request is merged, the branch must be rebased to properly reorder patches if necessary, and new tags must be created according to the related dist git change. Coordinate with the reviewer on who does this.

Alternatively, push directly to fedora-X.YY but be prepared to rebase if there is feedback.

Modifying existing patches

Follow the creating patches section. Send pull request with a fixup commit on top of the fedora-X.YY branch. When a new commit message is required, include it in the pull request description and/or discussion and make sure it is updated after the PR is merged, and the branch rebased and fixup commit squashed. Example:

[cpython (fedora-3.8)]$ git fetch fedora-python 
[cpython (fedora-3.8)]$ git reset --hard fedora-python/fedora-3.8
[cpython (fedora-3.8)]$ git log --oneline
aedd897c63 (HEAD -> fedora-3.8, fedora-python/fedora-3.8) 00328: Restore pyc to TIMESTAMP invalidation mode as default in rpmbuild
3172104314 00274: Upstream uses Debian-style architecture naming, change to match Fedora
197b8de27e 00251: Change user install location
36f1f2b462 00189: Instead of bundled wheels, use our RPM packaged wheels
50236468e8 00111: Don't try to build a libpythonMAJOR.MINOR.a
be6b980310 00102: Change the various install paths to use /usr/lib64/ instead or /usr/lib/
08c67bfedd 00001: Fixup distutils/unixccompiler.py to remove standard library path from rpath Was Patch0 in ivazquez' python3000 specfile
6f8c8320e9 (tag: v3.8.3) Python 3.8.3
...

[cpython (fedora-3.8)]$ git switch -c myfix
... edit the files relevant for your change and stage them in git...

[cpython (myfix)]$ git commit --fixup=3172104314  # use the hash of the original commit you want to modify

[cpython (myfix)]$ git log --oneline 
e9557758a3 (HEAD -> myfix) fixup! 00274: Upstream uses Debian-style architecture naming, change to match Fedora
...
3172104314 00274: Upstream uses Debian-style architecture naming, change to match Fedora
...

Rationale: When you modify an existing commit, the git history of every commit afterwards is changed. That is very unpleasant to work with when reviewing a Pull Request, for example because it's not easy to tell if the commits after the changed one were changed as well or just rebased.

[cpython (myfix)]$ git switch -c myfix_rebased
Switched to a new branch 'myfix_rebased'
[cpython (myfix_rebased)]$ git rebase --autosquash v3.8.3  # use the latest upstream tag
Successfully rebased and updated refs/heads/myfix_rebased.

Removing no longer needed patches

Follow the creating patches section. Send pull request with a revert commit. It will be rebased after the PR is merged.

For a dist-git pull request, feel free to just remove the patch without regenerating the others (by git format-patch). They will be regenerated with the next change. However, don't forget to also do the change on GitHub, so the commit/patch is not reintroduced later.

How to backport changes

If you want to backport a change from one Python/Fedora version to another, it's tedious to manually have to edit the files over and over. But you can use git to make it easier and to preserve commit authorship, messages etc.

Within the same component: Prefer FF merge, cherry-pick otherwise

In this section, we'll use rawhide as the reference to the branch you take the backport from and fcXX as a reference to the branch you want the backport to land into. However, this guide also applies to other kinds of backports (for example from fc34 to fc33 as well).

When you want to backport a commit within the same component, always consider fast-forward (FF) merging if possible. Ask yourself the following questions:

1. Is the fast-forward merge technically possible? I.e. is the HEAD of thefcXX branch an ancestor of the HEAD of the rawhide branch?
2. Are all other commits that will be FF merged together with my backport (if any) backwards compatible?

If the answer to all of these questions is yes, use a fast-forward merge.
If the answer to any of these questions is no, use a cherry-pick instead.

Note there is no harm in backporting moot commits (such as backporting "Fedora 35 mass rebuild" to Fedora 34) – trying to skip such commits only makes things harder for both you and the person who will backport the next thing.

Why not 3-way merges?
We have a linear changelog that should match what's built in Koji. 3-way merges make the history hard to follow.
3-way merges are also much harder to manage in RHEL.

There are some legitimate uses for 3-way merges, but they're not appropriate for day-to-day maintenance.

How to cherry pick commits from different dist-git components

If you need to cherry-pick a commit from one component to another, a trivial git merge or cherry-pick does not work. Different components have different git repos and also different spec filenames, which complicates cherry-picking.

The tested way of cherry-picking commits from one component to another is:

1. Use git format-patch to create patch file(s) of the commits you want to backport.
2. Find and replace the spec filename (and possibly other derived filenames) inside the patch file(s).
3. Use git am (possibly with --reject) to apply the commits into the target dist-git repo.

Changelog entries and release bumps are often causing conflicts that need to be solved manually.

When cherry-picking commits across components that represent the same Python version (e.g. from python3.6 to python36), the changes in patch files will likely apply cleanly. However when backporting changes from one Python version to another (e.g. from python3.9 to python3.8), it may be easier to avoid manually resolve cherry-pick conflicts in the dist-git Python patch files (which can lead to serious brain damage, because it's applying patches onto patches). Use the GitHub fork to re-generate them instead. (XXX link to the above guide).

Ferrypick

There is a alpha-quality tool that can help you do steps 1, 2 & 3 from the previous section to cherry-pick commits from a different dist-git component. You can use ferrypick to cherry-pick changes from the same component as well if you prefer.

https://github.com/fedora-python/ferrypick

Use it as this: XXX example from readme.