# New Python bootstrap in Fedora

:::warning
:atom_symbol: This process is a nuclear power plant. We don’t expect casual contributors to follow it.
:::

:::info
:construction: This is a WIP document. It'll be better in time. Ping @befeleme if you want to understand the unclear parts of it.
There's also an older document which may contain interesting details not covered here: https://fedoraproject.org/wiki/SIGs/Python/UpgradingPython.
:::

:::success
:bulb: Progress status: https://status.fedoralovespython.org/
:::

## What is covered by this guide?

- creating a new Fedora Change for the new Python X.Y version
- creating a new Python X.Y component in Fedora distgit
- creating a new Copr repository to build packages with the new Python X.Y as a main one
- testing the critical path components
- performing the mass rebuild in Koji
- obsoleting Fedora packages that require Python X.Y-1 version

## Initial bootstrap

Don't panic.

**Process gist:**
Start with Fedora Change.
Prepare the Fedora copy of Python and create the first Python builds.
Use Koji to run Python scratch builds (the builds last long and Koji has got all the architectures).
Once having a working build, start with Copr and move the Fedora dist-git component in parallel.
There may be different issues happening only in Copr or Koji, that's alright.

### Fedora Change

0. Find the last Fedora Change for Python (e.g. [Python3.14](https://fedoraproject.org/wiki/Changes/Python3.14))
1. Create a new Fedora Change for the newest Python (create a new page from [the empty template](https://fedoraproject.org/wiki/Changes/EmptyTemplate), copy the applicable contents to each section, change dates & links, add new sections if template changed in the meantime)
2. Open releng ticket at https://pagure.io/releng/issues ([example](https://pagure.io/releng/issue/11728))
3. Link the releng ticket to Fedora Change and set `ChangeReadyForWrangler` only after the previous step is done

### Update Fedora-Python git repository with new Python

We maintain our git copy of Python, including the downstream-only patches which have to be rebased on top of the upstream code.

**Repeat for every new Python release.**
See [Fedora Python interpreter maintenance guide](https://hackmd.io/9f64YNIZTCy0ZzKb5wKtqQ) for the details of Python interpreter maintenance.

1. Go to your local copy of https://github.com/python/cpython (clone if you don't have it)
2. `git pull origin`
3. Make sure you've got the desired tag: `git log vXXX` (e.g. `v3.13.0a1`)
4. (only the first time) Add the fedora-python repository as a remote (https://github.com/fedora-python/cpython); it's called `fedora` in my local copy -- referred as such later on
5. Fetch from the `fedora` repository: `git fetch fedora`
6. (only the first time) Checkout to the latest released version branch from fedora: `git checkout fedora-X.Y` (e.g. `fedora-3.12`)
7. Create and checkout a new branch for new Python: `git checkout -b fedora-X.Y` (e.g. `fedora-3.13`)
8. Rebase on top of the latest origin tag: `git rebase --interactive vXXX` (e.g. `v3.13.0a1`) -- you'll see a lot of commits relevant to the after-branch phase of the previous release, remove them and leave only the Fedora's patches
9. Solve merge conflicts...
10. Push new branch: `git push fedora fedora-X.Y` (e.g. `fedora-3.13`)
11. Tag and push the new fedora tag:
- `git tag fedora-3.13.0a1-1`
- `git push fedora fedora-3.13.0a1-1`

In case something went wrong, don't worry.
Force pushing is allowed and in fact desired.

### Add the new Python component to the Fedora dist-git

We ship the new Python as a non-main Python in all Fedoras immediately.
Even the one that'll be EOL soon.
It's handy. Folks can use it to test their projects, e.g. using `tox`.

**Process gist:**
We import the whole latest state of the Python n-1 to the distgit repository.
Then we disable the possibility for direct pushes to the repository.
The initial review for the new Python component takes place in Pull Requests.
Only after merge the component is actually the new Python version.

1. Request the new repo (adding alternate versions has got review exception, so no Bugzilla review ticket): `fedpkg request-repo pythonX.Y --exception --no-initial-commit`
2. Add details to the request: skip README and initial commit! ([example](https://pagure.io/releng/fedora-scm-requests/issue/57251)) - it may or may not work, this step will take a while - proceed with the next step in the meantime
3. Use the previous Python distgit repository as a starting point, but don't push into it
4. Start updating Python:
- rename specfile, strip changelog, leave only one changelog entry
- change hardcoded old Python version to new
- change it also in `tests/*`!
- download new sources (but don't upload them with `fedpkg new-sources` on the old component)
- enable the bootstrap bcond
- do any other necessary changes the specfile requires
- locally: run build until prep section (it's quicker), if succeeds: run a mock build with tweaks: `--with bootstrap`
- fix missing/overflowing files
- build the full package in Koji rather than locally - it takes time

5. When the repository is created (in step 3), we need to import the Python history into it.
- stash all your started changes
- git pull (to make sure you've got the latest changes)
- add the new repository as a remote of the current one
- `git push <remote_name> rawhide`

6. Clone the new repository, request all active Fedora branches and sync them with rawhide:
- `fedpkg request-branch fxx` for each branch,
- `git merge rawhide`,
- `git push --no-verify` (`--no-verify` because git will complain about the sources)

7. Change repository settings:
- close the repo for direct pushes,
- disable creating new branches by pushing,
- add admins (see the older Python list),
- change default bugzilla assignee to python-maint,
- create the main->rawhide alias
- add repo to Zuul CI (needs to be done in [fedora-project-config](https://pagure.io/fedora-project-config))

8. `fedpkg new-sources` the new sources + bring all your stashed changes to the new repository
9. Open Pull Requests for all the active Fedoras with the initial changes to the package to make it the new Python
10. Wait for CIs & ask for a review

Once shipped, you should open follow-up PRs to disable the bootstrap bcond.

### New Python Copr

In parallel, the new Python Copr is created.
It mocks an environment where the new Python is a main Python, so the packages actually build using it.
We use Fedora Rawhide as the destination system, even though the actual switch to the new main Python will happen in Fedora Rawhide+1.

0. Clone [whatdoibuild](https://github.com/hroncok/whatdoibuild), locate the branch which was used the last time, typically python3.N-1, and look at the top comment in `config.toml` - it contains the bootstrap packages with their bconds to switch and build manually
1. Make sure you're part of the `@python` group in Copr (`@python-packagers-sig` in FAS).
2. Create a new Copr repository (use the description of the previous copr repo, adjust (e.g. [Python 3.13](https://copr.fedorainfracloud.org/coprs/g/python/python3.13/))
3. Start with `python-rpm-macros`, option `custom build` with script (replace the `N` in the script with the actual major version number)

fedpkg clone -a python-rpm-macros
cd python-rpm-macros
sed -E -i 's/%__default_python3_version\s+.*/%__default_python3_version 3.N/' macros.python-srpm 
fedpkg srpm

- add `fedpkg` as dependency
- set the result directory to `python-rpm-macros`

This will ensure that generators are built with the new Python (3.N) as the main version - the basics for the rest of the builds

4. Build `Python n-1`, `Python n` and verify:
- `Python n-1` MUST contain versioned artifacts: e.g. `python3.12-devel`, it MUST NOT contain `python3-devel`
- `Python n` MUST contain the main artifacts: `python3-devel` etc.
5. Build `gdb` without Python
6. Once Pythons are built (wait for both of them), go for the rest of the main dependencies [from `config.toml`](https://github.com/hroncok/whatdoibuild/blob/10304cf739ceae32e03ebb9ed3abe894af645aa0/config.toml#L1)
7. Prepare for the repeated builds: Change configuration of the repos in `config.toml`: you want to get metadata from Koji Rawhide and build in Copr, this will change again after the switch of the main Python in Koji (see [older commit](https://github.com/hroncok/whatdoibuild/commit/901c6a23596937bd2fe1d896c420792f169b29b6) for inspiration)

#### Rinse and repeat: build all the Python packages

Once the main dependencies are built successfully, it's time to start running builds in Copr.

0. If you were doing this in the previous year, clean your repo: `rm -rf _dnf_cache_dir _fedpkg_cache_dir`, remove `progress.pkgs`.

1. `whatdoibuild` contains two scripts important for now:

- `jobs.py`:
- run with `python -u jobs.py > progress.pkgs`
- run immediately after the previous part has been built
- it creates a dnf sack and tries to resolve all packages, it'll dump the packages that are ready to rebuild to `progress.pkgs` file

- `bconds.py`
- run `python bconds.py`
- run repeatedly every few days
- it runs scratchbuilds in Koji for each package found in the `config.toml` with bconds, it'll switch them on/off, build the variants and get the metadata about the requirements. It takes a long time. Often we can build a package without tests/docs way before the full version can be built.

2. working with the output of `jobs.py` - `progress.pkgs` file
- it contains a sorted list of components that the automation figured out as ready for rebuild (= all of their dependencies are already available)
- we want to build them in bulk in Copr, it looks differenty for the bconded components and the full ones
- once a component appears in the list, it means it's ready for rebuild.
- when it disappears after another `jobs.py` run - it was rebuilt successfully, that's the happy path scenario
- when it stays in `progress.pkgs`, it means it failed to build. We need to investigate. Once in a while we submit builds for the whole `progress.pkgs` contents, it sometimes solves transient issues

- (only first time) Set `COPR=@python/python3.N` and send it to Copr with:

# "full" components don't contain ":" in the name
$ parallel copr add-package-distgit $COPR --webhook-rebuild on --commit rawhide --name -- $(cat progress.pkgs | grep -v :)
$ parallel copr build-package --nowait $COPR --background --name  -- $(cat progress.pkgs | grep -v :)
# bconded components are easily distinguishable because of ":"
$ for pkg in $(cat progress.pkgs | grep :); do; copr build --nowait $COPR --background _fedpkg_cache_dir/$pkg/*.src.rpm; done

- commit `progress.pkgs` to git each time after submitting packages to Copr to be able to see the updated diff after each `jobs.py` run

- (regular updates): run `jobs.py` and explore if there's anything new to build with `git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//'`; if yes, then:

# "full" components don't contain ":" in the name
$ parallel copr add-package-distgit $COPR --webhook-rebuild on --commit rawhide --name -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//'| grep -v :)
$ parallel copr build-package --nowait $COPR --background --name  -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep -v :)
# bconded components are easily distinguishable because of ":"
$ for pkg in $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep :); do; copr build --nowait $COPR --background _fedpkg_cache_dir/$pkg/*.src.rpm; done

- repeat until there are no changes to the diff - it's time to start unblocking

## Next steps - recurring during the whole bootstrap year

### Unblocking the possibility to build new packages

`jobs.py` outputs:
* The 50 most commonly needed components
* The 20 most commonly last-blocking components
* The 20 most commonly last-blocking small combinations of components
* Detected dependency loops

1. Components

last-blocking means that when this component builds, it will directly unblock the given amount of packages.
To determine what the package needs (and has not been built yet) use `jobs.py` with the package name as the argument:

python jobs.py python-cffi

It can take more arguments, as you go down the dependency tree.
Find the packages that were not built successfully, even though their dependencies are ready - these need to be repaired.

2. Dependency loops

They need to be broken, typically by bconds.
Examine package in dist-git.
If there's an applicable bcond, add it to `config.toml`.
If there isn't, investigate the ways to break the loop (typically by conditionalizing the test run or docs build), commit and submit a PR to the package with an explanation why you do it.
Once merged, add it to `config.toml`.
Every time `config.toml` is updated, run `python bconds.py` to run the new builds and update the cached data.

Note that packages listed in `config.toml` are excluded from dependency loop detection (the detection assumes you broke the loop). But packages evolve over time and the bconds listed in configuration might no longer actually help. The list needs to be maintained.

When done, new packages from `jobs.py` may emerge.

### Reporting the failures

1. Bugzillas

Start as soon as possible to get community involved in reporting and fixing upstream/downstream.
It's impossible for us to fix everything, so that's crucial.
Some reports will be moot, wrong component, issue not related to the new Python - it's normal and expected.

There's a script for that, [monitor_check.py](https://github.com/befeleme/mini-mass-rebuild/tree/python3.13)

- update the constants in the script: the new Copr name, bugzilla tracker, rawhide tracker and so on
- update the dictionary of failure reasons found in the logs - for the found matches, the description will be less generic and more to the point

In the beginning there will likely be hundreds of failures with repeating patterns. Take one and add it to the reasons dict in the script.

Then run the script:
`python monitor_check.py --open-bug-reports --with-reason`

This will only open the bugs for packages with matches in the reasons dict.
When you build the starting few reasons and get rid of the repeating patterns, it's time to report the rest.
Drop the `--with-reason` option to open bugz for all the failures.

2. Upstream

Important components that block a lot of packages will use our heavy lifting.
Report upstream, ask the team experts on CPython (@vstinner) for help, submit PRs that are in your capacity.

### Knowledge base on issues we encounter

#### cmake

Heaven knows why we have to add new Python to cmake every year:
[example](https://src.fedoraproject.org/rpms/cmake/pull-request/43).
Many projects bundle cmake scripts, so expect to patch ~10 of others just to make them see Python 3.N as a valid interpreter.

#### The latest commits in distgit cause package to FTBFS

**The background:**
In Fedora distgit there are commits to a component that cause FTBFS. The last build in Koji failed (or the changes were never attempted to build).
Our script gets the data from Koji, i.e. from the latest **successful** build.
Then, once it figures out the dependencies, we build the state that's present in Fedora distgit - which is broken.
Koschei will not notify us about such situations, because it rebuilds from the latest SRPMs rather than from distgit.

**The solution:**
None known. When inspecting weird build failures, always take a look at the list of the latest commits in distgit of a component. Look at the build information associated with each commit (it's displayed as a number next to the commit hash in Pagure): there will be none or it will be written in red-coloured font. Explore the logs, if available, to check if the failure is similar. Open a generic FTBFS bugzilla.

#### Dirty hacks

**The background:**
The package FTBFS in Copr. We decide to do some modifications to the spec (e.g. disable tests, remove some dependencies) and fire a build in Copr with such local change. The change is too hacky to add to the real component or we deliberately keep it only local for a moment. Then the actual mass rebuild happens and everything falls apart, because we forgot :face_palm:.

**The solution:**
We keep a document containing such changes: https://hackmd.io/@befeleme/shorime-za-to-v-pekle and regularly review the issues there.

## Critpaths testing

When mass-rebuild is approaching and your Copr is in a quite good shape, it's beneficial to test the critical paths components and the python-classroom one.
We don't want to start a mass rebuild and render an uninstallable Fedora by accident.
A typical issue arises with anaconda, which doesn't run any tests during the build and hence is often uninstallable.

Critical path components are stored in [fedora-comps](https://www.pagure.io/fedora-comps) repository,
the list is maintained in a relevant file, e.g. [comps-f43.xml.in](https://www.pagure.io/fedora-comps/blob/main/f/comps-f43.xml.in).
Clone the repository to work with it locally.
Grab all the paths from the file:
`grep -F '>critical-path' comps-fNN.xml.in`

Initialize empty mock with your copr config, ensure the Python version is the new one and try to install a critical path one after another, e.g.:

$ mock -r fedora-rawhide-python314 init && mock -r fedora-rawhide-python314 install 'python3 >= 3.14' @critical-path-server

Review the transaction errors to unblock the components on time.

## Mass rebuild in Koji

TODO:
- config https://github.com/hroncok/whatdoibuild/commit/78c670e0299397c04b12c6e0ec0ccd71f8f5c396
- https://github.com/hroncok/whatdoibuild/commit/d9821a877b51695e079707edc00beb3d091dbb88
- coordinate with possible other mass rebuilds in the distribution
- openqa testing

### Side-tag requests

- side-tag creation request
See e.g. https://pagure.io/releng/issue/12130 for inspiration
Make sure to ask for signing the packages, it'll speed up the side-tag merge.
- side tag merge request
See e.g. https://pagure.io/releng/issue/12155

### E-mail announcements

When you're about to start the mass-rebuild, make sure you communicate with the Fedora community.
Send the comms to `[e-mail]`, `[e-mail]`, `[e-mail]`

It'll likely be a few e-mails in total:
- heads up - that you plan to start (see [2024 thread](https://lists.fedoraproject.org/archives/list/[e-mail]/thread/JR3TQYWJWAGUD3K7LORMHZUIN7YY4QJ6/))
- the actual start of the rebuild -- ask folks not to build anything in the regular Rawhide now (see [2024 message](https://lists.fedoraproject.org/archives/list/[e-mail]/message/L5SH43A6UQCPVI3SA5FCZNB2G33PAOOE/))
- periodical statuses, especially if your rebuild lasts longer than just a few days (see [2024 message](https://lists.fedoraproject.org/archives/list/[e-mail]/message/WWUNCWG4HVWTXNSIGUMU3MY6MHRKIOWX/)). Use [fedora-package-maintainers script](https://pagure.io/fedora-misc-package-utilities) to generate the lists of maintainers to append to the e-mails
- the merge of side tag - include instructions how to build packages now and how to solve issues (see [2024 message](https://lists.fedoraproject.org/archives/list/[e-mail]/message/QFSHZEATFRPM5OL6UJ3ZUGB4UYC6YYWF/))

### Preparing mass-rebuild

When the side tag is ready to build in it, we need to bootstrap Python and the basic components again there, very similar way we did it in [Copr](#Create-new-Python-Copr).

- distgit changes for the real rebuild
- python-rpm-macros (https://src.fedoraproject.org/rpms/python-rpm-macros/c/a389cc467425766a0122be2c143a9a4fdb76191e?branch=rawhide)
- bootstrap Python again (https://src.fedoraproject.org/rpms/python3.13/pull-request/70)
- bootstrap the first packages
- clone them to `_fedpkg_cache_dir`
- add the applicable `%global _with{out}_<bcond_name>` to the first line of the specfile
- `git diff > ../../patches_dir/<component_name>.patch`
- build package in side-tag `fedpkg build --target=<side-tag>`
- git commit the content of `patches_dir`

- Changes to config.toml https://github.com/hroncok/whatdoibuild/commit/78c670e0299397c04b12c6e0ec0ccd71f8f5c396,
- try it in a dry mode first
- clean progress.pkgs
- when ready, comment out the exception raising (https://github.com/hroncok/whatdoibuild/commit/e24e03418c58a211bde50ea090f6f7556a72751a)

For the initial bootstrap sequence we need sequential builds.
When one is done, wait for the build to appear in the repository and request (to make things quicker) it:
`koji wait-repo f43-python --build=python-rpm-macros-3.14-1.fc43 --request`

### Mass-rebuild

Use jobs.py as with copr to generate the list of ready-to-build package names in progress.pkgs.
Build them with `build.py`.
When properly configured, it will:
- clone the distgit repository of each component to `_fedpkg_cache_dir`
- if a component is bconded, apply patches
- bump spec, git commit, push and send a built to the side-tag

Run as
`python build.py <pkgname>`
or with parallel and multiple pkgnames:
`parallel python build.py -- $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//') > log`

If someone builds a package in Rawhide while the rebuild continues,
just rerun the build.
`repo_findold.py` from mini-mass-rebuild to identify pakcages built outside of the side tag - build them again with `build.py`

### Conditions to merge side tag

- Fedora comps must install (mock + koji repo added)
- kernel is built (someone from the kernel team must do it)
- all rawhide packages built in the meantime must be bumped and rebuilt in the side tag again (repofindold script)
- not too many packages left to build -- ideally ~500
- openQA tests are run, pass and we have green light from Fedora QA (Adam W. is likely to run some tests on the composes)

### Post mass-rebuild actions

The side tag is merged by releng.
- Create FTI bugzillas (mhroncok is handling this)
- changes to config.toml - set a last compose with the old Python as a reference value for progress statistics, see [2024 commit](https://github.com/hroncok/whatdoibuild/commit/d9821a877b51695e079707edc00beb3d091dbb88)
- add remaining packages to copr
- when building in copr for the next python, don't delete old succeeded builds to test if there are abi breakages
- run scripts to clean up bugzillas: [close duplicates](https://github.com/befeleme/mini-mass-rebuild/blob/python3.14/dupe_ftbfs_fti_bugzillas.ipynb), [block missing dependencies bz](https://github.com/befeleme/mini-mass-rebuild/blob/python3.14/block_missing_deps_bugzillas.ipynb), [block fti bz](https://github.com/befeleme/mini-mass-rebuild/blob/python3.14/block_fti_bugzillas.ipynb) - one-time activity
- obsolete not working packages (see below for instructions)
- when the dust settles, around the end of August, open BZ to ask upstreams to publish wheels + ask upstreams to publish wheels: [example from 3.13](https://bugzilla.redhat.com/show_bug.cgi?id=2312845)

## Obsoleting not working packages

Some packages will never successfully build with the new Python.
If people have them installed on their machines, they would remain there forever, although they'd be unusable, as they would require python(abi) of Python X.Y-1.
To avoid not working packages lingering on the people's machines,
we can employ a special package - [fedora-obsolete-packages](https://src.fedoraproject.org/rpms/fedora-obsolete-packages/). It will remove the defined packages from the user machine on a system update.

Fedora supports upgrades up to two versions at once.
For the **odd** Fedora version, when we change the main Python version, it should happen in both the **Beta Freeze** and **Final Freeze** phases.
And then again, for the **even** Fedora version, which still supports upgrades from one Fedora version with the obsolete main Python, it should happen in a **Final Freeze** phase.
See table for illustration:

| Fedora version | Main Python | Supported upgrade from |
| -------- | -------- | -------- |
| 41 | 3.13 | 39 (3.12), 40 (3.12) |
| 42 | 3.13 | 40 (3.12), 41 |
| 43 | 3.14 | 41 (3.13), 42 (3.13) |
| 44 | 3.14 | 42 (3.13), 43 |
| 45 | 3.15 | 43 (3.14), 44 (3.14) |

We've got to find packages existing in the supported upgrade path versions which still require python(abi) = 3.Y-1. Use a [script](https://github.com/befeleme/mini-mass-rebuild/blob/python3.14/obsolete_packages.py) doing exactly that and paste its output to `fedora-obsolete-packages.spec`, the Python packages section.

For the first PR for a given main Python version, prepare one commit removing the old contents, and add a new commit with a list of new obsolete packages.
For inspiration see [Fedora 41 first PR](https://src.fedoraproject.org/rpms/fedora-obsolete-packages/pull-request/94).

For the even Fedora version, generate the list of obsoleted packages, but work with the resulting diff, so that packages are only included to the list, not removed. See [Fedora 42 obsolete update](https://src.fedoraproject.org/rpms/fedora-obsolete-packages/pull-request/109).

Because we do it in the Freeze phase, we need a Freeze Exception for this exercise. Create an [obsolete packages bugzilla](https://bugzilla.redhat.com/show_bug.cgi?id=2302853) and set it to [block](https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process) the desired target: `BetaFreezeException` or `FinalFreezeException`. Reuse that bug for both releases containing the same Python version. Block the main Python tracker too.

To remember about the important dates, subscribe to [fedora-release-schedule-ical](https://github.com/befeleme/fedora-release-schedule-ical) which is a cleaned-up official [Fedora calendar](https://fedorapeople.org/groups/schedule/).

## Dealing with packages which don't build with new Python

There will always be a lingering few packages that can't be successfully built in time for the release of the new Fedora.
If they have bugs in NEW state, they eventually get retired. If ASSIGNED, it's more complicated. Possible outcomes:
- let the broken packages sink into the new release (not ideal but doesn't really break anything)
- retire the packages with FESCo fast-track ticket (e.g. https://pagure.io/fesco/issue/3276)
- obsolete the packages in fedora-obsolete-packages: if they get built, they will get a new release number and all is good, if they don't - they won't kill anybody's update