New Python bootstrap in Fedora

What is covered by this guide?

• creating a new Fedora Change for the new Python X.Y version
• creating a new Python X.Y component in Fedora distgit
• creating a new Copr repository to build packages with the new Python X.Y as a main one
• testing the critical path components
• performing the mass rebuild in Koji
• obsoleting Fedora packages that require Python X.Y-1 version

Initial bootstrap

Don't panic.

Process gist: Start with Fedora Change. Prepare the Fedora copy of Python and create the first Python builds. Use Koji to run Python scratch builds (the builds last long and Koji has got all the architectures). Once having a working build, start with Copr and move the Fedora dist-git component in parallel. There may be different issues happening only in Copr or Koji, that's alright.

Fedora Change

0. Find the last Fedora Change for Python (e.g. Python3.14)
1. Create a new Fedora Change for the newest Python (create a new page from the empty template, copy the applicable contents to each section, change dates & links, add new sections if template changed in the meantime)
2. Open releng ticket at https://pagure.io/releng/issues (example)
3. Link the releng ticket to Fedora Change and set ChangeReadyForWrangler only after the previous step is done

Update Fedora-Python git repository with new Python

We maintain our git copy of Python, including the downstream-only patches which have to be rebased on top of the upstream code.

Repeat for every new Python release. See Fedora Python interpreter maintenance guide for the details of Python interpreter maintenance.

1. Go to your local copy of https://github.com/python/cpython (clone if you don't have it)
2. git pull origin
3. Make sure you've got the desired tag: git log vXXX (e.g. v3.13.0a1)
4. (only the first time) Add the fedora-python repository as a remote (https://github.com/fedora-python/cpython); it's called fedora in my local copy – referred as such later on
5. Fetch from the fedora repository: git fetch fedora
6. (only the first time) Checkout to the latest released version branch from fedora: git checkout fedora-X.Y (e.g. fedora-3.12)
7. Create and checkout a new branch for new Python: git checkout -b fedora-X.Y (e.g. fedora-3.13)
8. Rebase on top of the latest origin tag: git rebase --interactive vXXX (e.g. v3.13.0a1) – you'll see a lot of commits relevant to the after-branch phase of the previous release, remove them and leave only the Fedora's patches
9. Solve merge conflicts…
10. Push new branch: git push fedora fedora-X.Y (e.g. fedora-3.13)
11. Tag and push the new fedora tag:

• git tag fedora-3.13.0a1-1
• git push fedora fedora-3.13.0a1-1

In case something went wrong, don't worry. Force pushing is allowed and in fact desired.

Add the new Python component to the Fedora dist-git

We ship the new Python as a non-main Python in all Fedoras immediately. Even the one that'll be EOL soon. It's handy. Folks can use it to test their projects, e.g. using tox.

Process gist: We import the whole latest state of the Python n-1 to the distgit repository. Then we disable the possibility for direct pushes to the repository. The initial review for the new Python component takes place in Pull Requests. Only after merge the component is actually the new Python version.

1. Request the new repo (adding alternate versions has got review exception, so no Bugzilla review ticket): fedpkg request-repo pythonX.Y --exception --no-initial-commit
2. Add details to the request: skip README and initial commit! (example) - it may or may not work, this step will take a while - proceed with the next step in the meantime
3. Use the previous Python distgit repository as a starting point, but don't push into it
4. Start updating Python:

• rename specfile, strip changelog, leave only one changelog entry
• change hardcoded old Python version to new
• change it also in tests/*!
• download new sources (but don't upload them with fedpkg new-sources on the old component)
• enable the bootstrap bcond
• do any other necessary changes the specfile requires
• locally: run build until prep section (it's quicker), if succeeds: run a mock build with tweaks: --without tests --without optimizations
• fix missing/overflowing files
• build the full package in Koji rather than locally - it takes time

5. When the repository is created (in step 3), we need to import the Python history into it.

• stash all your started changes
• git pull (to make sure you've got the latest changes)
• add the new repository as a remote of the current one
• git push <remote_name> rawhide

6. Clone the new repository, request all active Fedora branches and sync them with rawhide:

• fedpkg request-branch fxx for each branch,
• git merge rawhide,
• git push --no-verify (--no-verify because git will complain about the sources)

7. Change repository settings:

• close the repo for direct pushes,
• disable creating new branches by pushing,
• add admins (see the older Python list),
• change default bugzilla assignee to python-maint,
• create the main->rawhide alias
• add repo to Zuul CI (needs to be done in fedora-project-config)

8. fedpkg new-sources the new sources + bring all your stashed changes to the new repository
9. Open Pull Requests for all the active Fedoras with the initial changes to the package to make it the new Python
10. Wait for CIs & ask for a review

Once shipped, you should open follow-up PRs to disable the bootstrap bcond.

New Python Copr

In parallel, the new Python Copr is created. It mocks an environment where the new Python is a main Python, so the packages actually build using it. We use Fedora Rawhide as the destination system, even though the actual switch to the new main Python will happen in Fedora Rawhide+1.

0. Clone whatdoibuild, locate the branch which was used the last time, typically python3.N-1, and look at the top comment in config.toml - it contains the bootstrap packages with their bconds to switch and build manually
1. Make sure you're part of the @python group in Copr (@python-packagers-sig in FAS).
2. Create a new Copr repository (use the description of the previous copr repo, adjust (e.g. Python 3.13)
3. Start with python-rpm-macros, option custom build with script (replace the N in the script with the actual major version number)

fedpkg clone -a python-rpm-macros
cd python-rpm-macros
sed -E -i 's/%__default_python3_version\s+.*/%__default_python3_version 3.N/' macros.python-srpm 
fedpkg srpm

• add fedpkg as dependency
• set the result directory to python-rpm-macros

This will ensure that generators are built with the new Python (3.N) as the main version - the basics for the rest of the builds

4. Build Python n-1, Python n and verify:

• Python n-1 MUST contain versioned artifacts: e.g. python3.12-devel, it MUST NOT contain python3-devel
• Python n MUST contain the main artifacts: python3-devel etc.

5. Build gdb without Python
6. Once Pythons are built (wait for both of them), go for the rest of the main dependencies from config.toml
7. Prepare for the repeated builds: Change configuration of the repos in config.toml: you want to get metadata from Koji Rawhide and build in Copr, this will change again after the switch of the main Python in Koji (see older commit for inspiration)

Rinse and repeat: build all the Python packages

Once the main dependencies are built successfully, it's time to start running builds in Copr.

1. whatdoibuild contains two scripts important for now:

• jobs.py:

  • run with python -u jobs.py > progress.pkgs
  • run immediately after the previous part has been built
  • it creates a dnf sack and tries to resolve all packages, it'll dump the packages that are ready to rebuild to progress.pkgs file

• bconds.py

  • run python bconds.py
  • run repeatedly every few days
  • it runs scratchbuilds in Koji for each package found in the config.toml with bconds, it'll switch them on/off, build the variants and get the metadata about the requirements. It takes a long time. Often we can build a package without tests/docs way before the full version can be built.

2. working with the output of jobs.py - progress.pkgs file

• it contains a sorted list of components that the automation figured out as ready for rebuild (= all of their dependencies are already available)

• we want to build them in bulk in Copr, it looks differenty for the bconded components and the full ones

• once a component appears in the list, it means it's ready for rebuild.

  • when it disappears after another jobs.py run - it was rebuilt successfully, that's the happy path scenario
  • when it stays in progress.pkgs, it means it failed to build. We need to investigate. Once in a while we submit builds for the whole progress.pkgs contents, it sometimes solves transient issues

• (only first time) Set COPR=@python/python3.N and send it to Copr with:

# "full" components don't contain ":" in the name
$ parallel copr add-package-distgit $COPR --webhook-rebuild on --commit rawhide --name -- $(cat progress.pkgs | grep -v :)
$ parallel copr build-package --nowait $COPR --background --name  -- $(cat progress.pkgs | grep -v :)
# bconded components are easily distinguishable because of ":"
$ for pkg in $(cat progress.pkgs | grep :); do; copr build --nowait $COPR --background _fedpkg_cache_dir/$pkg/*.src.rpm; done

• commit progress.pkgs to git each time after submitting packages to Copr to be able to see the updated diff after each jobs.py run

• (regular updates): run jobs.py and explore if there's anything new to build with git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//'; if yes, then:

# "full" components don't contain ":" in the name
$ parallel copr add-package-distgit $COPR --webhook-rebuild on --commit rawhide --name -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//'| grep -v :)
$ parallel copr build-package --nowait $COPR --background --name  -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep -v :)
# bconded components are easily distinguishable because of ":"
$ for pkg in $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep :); do; copr build --nowait $COPR --background _fedpkg_cache_dir/$pkg/*.src.rpm; done

• repeat until there are no changes to the diff - it's time to start unblocking

Next steps - recurring during the whole bootstrap year

Unblocking the possibility to build new packages

jobs.py outputs:

• The 50 most commonly needed components
• The 20 most commonly last-blocking components
• The 20 most commonly last-blocking small combinations of components
• Detected dependency loops

1. Components

last-blocking means that when this component builds, it will directly unblock the given amount of packages. To determine what the package needs (and has not been built yet) use jobs.py with the package name as the argument:

python jobs.py python-cffi

It can take more arguments, as you go down the dependency tree. Find the packages that were not built successfully, even though their dependencies are ready - these need to be repaired.

2. Dependency loops

They need to be broken, typically by bconds. Examine package in dist-git. If there's an applicable bcond, add it to config.toml. If there isn't, investigate the ways to break the loop (typically by conditionalizing the test run or docs build), commit and submit a PR to the package with an explanation why you do it. Once merged, add it to config.toml. Every time config.toml is updated, run python bconds.py to run the new builds and update the cached data.

Note that packages listed in config.toml are excluded from dependency loop detection (the detection assumes you broke the loop). But packages evolve over time and the bconds listed in configuration might no longer actually help. The list needs to be maintained.

When done, new packages from jobs.py may emerge.

Reporting the failures

1. Bugzillas

Start as soon as possible to get community involved in reporting and fixing upstream/downstream. It's impossible for us to fix everything, so that's crucial. Some reports will be moot, wrong component, issue not related to the new Python - it's normal and expected.

There's a script for that, monitor_check.py

• update the constants in the script: the new Copr name, bugzilla tracker, rawhide tracker and so on
• update the dictionary of failure reasons found in the logs - for the found matches, the description will be less generic and more to the point

In the beginning there will likely be hundreds of failures with repeating patterns. Take one and add it to the reasons dict in the script.

Then run the script: python monitor_check.py --open-bug-reports --with-reason

This will only open the bugs for packages with matches in the reasons dict. When you build the starting few reasons and get rid of the repeating patterns, it's time to report the rest. Drop the --with-reason option to open bugz for all the failures.

2. Upstream

Important components that block a lot of packages will use our heavy lifting. Report upstream, ask the team experts on CPython (@vstinner) for help, submit PRs that are in your capacity.

Knowledge base on issues we encounter

cmake

Heaven knows why we have to add new Python to cmake every year: example. Many projects bundle cmake scripts, so expect to patch ~10 of others just to make them see Python 3.N as a valid interpreter.

The latest commits in distgit cause package to FTBFS

The background: In Fedora distgit there are commits to a component that cause FTBFS. The last build in Koji failed (or the changes were never attempted to build). Our script gets the data from Koji, i.e. from the latest successful build. Then, once it figures out the dependencies, we build the state that's present in Fedora distgit - which is broken. Koschei will not notify us about such situations, because it rebuilds from the latest SRPMs rather than from distgit.

The solution: None known. When inspecting weird build failures, always take a look at the list of the latest commits in distgit of a component. Look at the build information associated with each commit (it's displayed as a number next to the commit hash in Pagure): there will be none or it will be written in red-coloured font. Explore the logs, if available, to check if the failure is similar. Open a generic FTBFS bugzilla.

Dirty hacks

The background: The package FTBFS in Copr. We decide to do some modifications to the spec (e.g. disable tests, remove some dependencies) and fire a build in Copr with such local change. The change is too hacky to add to the real component or we deliberately keep it only local for a moment. Then the actual mass rebuild happens and everything falls apart, because we forgot

The solution: We keep a document containing such changes: https://hackmd.io/@befeleme/shorime-za-to-v-pekle and regularly review the issues there.

Critpaths testing

When mass-rebuild is approaching and your Copr is in a quite good shape, it's beneficial to test the critical paths components and the python-classroom one. We don't want to start a mass rebuild and render an uninstallable Fedora by accident. A typical issue arises with anaconda, which doesn't run any tests during the build and hence is often uninstallable.

Critical path components are stored in fedora-comps repository, the list is maintained in a relevant file, e.g. comps-f43.xml.in. Clone the repository to work with it locally. Grab all the paths from the file: grep -F '>critical-path' comps-fNN.xml.in

Initialize empty mock with your copr config, ensure the Python version is the new one and try to install a critical path one after another, e.g.:

$ mock -r fedora-rawhide-python314 init && mock -r fedora-rawhide-python314 install 'python3 >= 3.14' @critical-path-server

Review the transaction errors to unblock the components on time.

Mass rebuild in Koji

TODO:

• exceptions (kernel etc.)
• config https://github.com/hroncok/whatdoibuild/commit/78c670e0299397c04b12c6e0ec0ccd71f8f5c396
• https://github.com/hroncok/whatdoibuild/commit/d9821a877b51695e079707edc00beb3d091dbb88
• coordinate with possible other mass rebuilds in the distribution
• openqa testing
• critpaths testing
• copr handling (add all remaining pkgs at the side-tag merge)

Side-tag requests

• side-tag creation request See e.g. https://pagure.io/releng/issue/12130 for inspiration Make sure to ask for signing the packages, it'll speed up the side-tag merge.
• side tag merge request See e.g. https://pagure.io/releng/issue/12155

E-mail announcements

When you're about to start the mass-rebuild, make sure you communicate with the Fedora community. Send the comms to devel-announce@lists.fedoraproject.org, devel@lists.fedoraproject.org, python-devel@lists.fedoraproject.org

It'll likely be a few e-mails in total:

• heads up - that you plan to start (see 2024 thread)
• the actual start of the rebuild – ask folks not to build anything in the regular Rawhide now (see 2024 message)
• periodical statuses, especially if your rebuild lasts longer than just a few days (see 2024 message). Use fedora-package-maintainers script to generate the lists of maintainers to append to the e-mails
• the merge of side tag - include instructions how to build packages now and how to solve issues (see 2024 message)

Preparing mass-rebuild

When the side tag is ready to build in it, we need to bootstrap Python and the basic components again there, very similar way we did it in Copr.

• distgit changes for the real rebuild

  • python-rpm-macros (https://src.fedoraproject.org/rpms/python-rpm-macros/c/a389cc467425766a0122be2c143a9a4fdb76191e?branch=rawhide)
  • bootstrap Python again (https://src.fedoraproject.org/rpms/python3.13/pull-request/70)
  • bootstrap the first packages
    • clone them to _fedpkg_cache_dir
    • add the applicable %global _with{out}_<bcond_name> to the first line of the specfile
    • git diff > ../../patches_dir/<component_name>.patch
    • build package in side-tag
    • git commit the content of patches_dir

• Changes to config.toml https://github.com/hroncok/whatdoibuild/commit/78c670e0299397c04b12c6e0ec0ccd71f8f5c396,

• try it in a dry mode first

• clean progress.pkgs

• when ready, comment out the exception raising (https://github.com/hroncok/whatdoibuild/commit/e24e03418c58a211bde50ea090f6f7556a72751a)

Mass-rebuild

Use jobs.py as with copr to generate the list of ready-to-build package names in progress.pkgs. Build them with build.py. When properly configured, it will:

• clone the distgit repository of each component to _fedpkg_cache_dir
• if a component is bconded, apply patches
• bump spec, git commit, push and send a built to the side-tag

Run as python build.py <pkgname> or with parallel and multiple pkgnames: parallel python build.py -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//') > log

Obsoleting not working packages

Some packages will never successfully build with the new Python. If people have them installed on their machines, they would remain there forever, although they'd be unusable, as they would require python(abi) of Python X.Y-1. To avoid not working packages lingering on the people's machines, we can employ a special package - fedora-obsolete-packages. It will remove the defined packages from the user machine on a system update.

Fedora supports upgrades up to two versions at once. For the odd Fedora version, when we change the main Python version, it should happen in both the Beta Freeze and Final Freeze phases. And then again, for the even Fedora version, which still supports upgrades from one Fedora version with the obsolete main Python, it should happen in a Final Freeze phase. See table for illustration:

We've got to find packages existing in the supported upgrade path versions which still require python(abi) = 3.Y-1. Use a script doing exactly that and paste its output to fedora-obsolete-packages.spec, the Python packages section.

For the first PR for a given main Python version, prepare one commit removing the old contents, and add a new commit with a list of new obsolete packages. For inspiration see Fedora 41 first PR.

For the even Fedora version, generate the list of obsoleted packages, but work with the resulting diff, so that packages are only included to the list, not removed. See Fedora 42 obsolete update.

Because we do it in the Freeze phase, we need a Freeze Exception for this exercise. Create an obsolete packages bugzilla and set it to block the desired target: BetaFreezeException or FinalFreezeException. Reuse that bug for both releases containing the same Python version. Block the main Python tracker too.

To remember about the important dates, subscribe to fedora-release-schedule-ical which is a cleaned-up official Fedora calendar.