:::warning
:atom_symbol: This is an extreme WIP document. It'll be better in time. Come back to it in some weeks or ping @befeleme for clarifications
:::

## Process step by step

Don't panic.

Start with Fedora Change. Prepare the Fedora copy of Python and prepare the first Python builds. Use Koji to run scratch builds (the builds last long and Koji has got all the architectures). Once having a working build, start with Copr and move the Fedora dist-git component in parallel. There may be different issues happening only in Copr or Koji, that's alright.

### Fedora Change

0. Find the last Fedora Change for Python (e.g. https://fedoraproject.org/wiki/Changes/Python3.13)
1. Create new Fedora Change for the newest Python (create a new page from template, copy the applicable contents to each section, change dates & links, add new sections if template changed in the meantime)
1) Open releng ticket at https://pagure.io/releng/issues (e.g. https://pagure.io/releng/issue/11728)
2) Link the releng ticket to Fedora Change and set ChangeReadyForWrangler only after the previous step is done

### Download the newest Python to Fedora-Python git repository

We maintain our git copy of Python, including the downstream-only patches which have to be rebased on top of the upstream code.

**Repeat for every new Python release.**
See https://hackmd.io/9f64YNIZTCy0ZzKb5wKtqQ for the details of Python interpreter maintenance.

1) Go to your local copy of https://github.com/python/cpython (clone if you don't have it)
2) `git pull origin`
3) Make sure you've got the desired tag: `git log vXXX` (e.g. `v3.13.0a1`)
4) (only the first time) Add the fedora-python repository as a remote (https://github.com/fedora-python/cpython); it's called `fedora` in my local copy -- referred as such later on
5) Fetch from the `fedora` repository: `git fetch fedora`
6) (only the first time) Checkout to the latest released version branch from fedora: `git checkout fedora-3.1x` (e.g. `fedora-3.12`)
7)
a) (only the first time) Create new branch for new Python: `git checkout -b fedora-3.1x` (e.g. `fedora-3.13`)
b) (when repeating) Checkout to the branch for the new Python `git checkout fedora-3.1x`
8) Rebase on top of the latest origin tag: `git rebase --interactive vXXX` (e.g. `v3.13.0a1`) -- leave only Fedora's patches
10) Solve merge conflicts...
11) Push new branch: `git push fedora fedora-3.1x` (e.g. `fedora-3.13`)
12) Tag and push the new fedora tag:
- `git tag fedora-3.13.0a1-1`
- `git push fedora fedora-3.13.0a1-1`

In case something went wrong, don't worry.
Force pushing is allowed and in fact desired.

### Create new Python component in the Fedora dist-git

We ship the new Python as a non-main Python in all Fedoras immediately.
Even the one that'll be EOL soon.
It's handy. Folks can use it to test their projects, e.g. using `tox`.

The process goes like this: we import the whole latest state of the Python n-1 to the distgit repository.
Then we disable the possibility for direct pushes to the repository.
The initial review for the new Python component takes place in Pull Requests.
Only after merge the component actually is the new Python version.

1) Request the new repo (adding alternate versions has got review exception, so no Bugzilla review ticket): `fedpkg request-repo python3.1x --exception --no-initial-commit`
2) Add details to the request: skip README and initial commit! (e.g. https://pagure.io/releng/fedora-scm-requests/issue/57251) - it may or may not work, this step will take a while - proceed with the next step in the meantime
3) Use the previous Python distgit repository as a starting point, but don't push into it
4) Start updating Python:
- rename specfile, strip changelog, leave only one changelog entry
- change hardcoded old Python version to new
- change it also in `tests/*`!
- download new sources (but don't upload them with fedpkg new-sources on the old component)
- enable bootstrap bcond
- do any other necessary changes the specfile requires
- locally: run build until prep section (it's quicker), if succeeds: run a mock build with tweaks: `--without tests --without optimizations`
- fix missing/overflowing files
- build the full package in Koji rather than locally - it takes time

5) When the repository is created (in step 3), import the whole history of the Python n-1 distgit into it and push it (without your update) to rawhide and active fXX branches
6) Change repository settings: close the repo for direct pushes, disable creating new branches by pushing, add repo to Zuul CI, add admins (see the older Python list), change default bugzilla assignee to python-maint, create the main->rawhide alias
7) fedpkg new-sources the new sources
8) Open Pull Requests for all the active Fedoras with the initial changes to the package to make it the new Python
9) Wait for CIs & ask for a review

Once shipped, you should open followup PRs to disable the bootstrap bcond.

### Create new Python Copr

In parallel, the new Python Copr is created.
It mocks an environment where the new Python is a main Python, so the packages actually build using it.
We use Fedora Rawhide as the destination system, even though the actual switch to the new main Python will happen in Fedora Rawhide+1.

0. Clone https://github.com/hroncok/whatdoibuild and look at the top comment in `config.toml` - it contains the bootstrap packages with their bconds to switch and build manually (locate the branch which was used the last time, typically python3.N-1)
1. Make sure you're part of the `@python` group in Copr (python-packagers-sig in FAS).
2. Create a new Copr repository (use the description of the previous copr repo, adjust (e.g. https://copr.fedorainfracloud.org/coprs/g/python/python3.13/)
3. Start with `python-rpm-macros`, custom build with script

fedpkg clone -a python-rpm-macros
cd python-rpm-macros
sed -E -i 's/%__default_python3_version\s+.*/%__default_python3_version 3.N/' macros.python-srpm 
fedpkg srpm

this will ensure that generators are built with the new Python (3.N) as the main version - the basics for the rest of the builds

4. Build `Python n-1`, `Python n` and verify:
- `Python n-1` MUST contain versioned artifacts: e.g. `python3.12-devel`, it MUST NOT contain `python3-devel`
- `Python n` MUST contain the main artifacts: `python3-devel` etc.
5. Build `gdb` without Python
6. Once Pythons are built, go for the rest of the main dependencies [from `config.toml`](https://github.com/hroncok/whatdoibuild/blob/10304cf739ceae32e03ebb9ed3abe894af645aa0/config.toml#L1)
7. Once the main dependencies are built successfully, start running builds in Copr:
- change configuration of the repos in `config.toml`: now you want to get metadata from Koji Rawhide and build in Copr, this will change in the future (see: https://github.com/hroncok/whatdoibuild/commit/901c6a23596937bd2fe1d896c420792f169b29b6 for inspiration)
- run with `python -u jobs.py > progress.pkgs` - this creates a dnf sack and tries to resolve all packages, it'll dump the packages that are ready to rebuild to `progress.pkgs` file.
- run `python bconds.py` - this will run scratch builds in Koji for each package found in the configuration as containing defined bconds, it'll switch them on/off, build the variants and get the metadata about requirements. It takes a long time. Often we can build a package without tests/docs way before the full version can be built
- commit `progress.pkgs` to git each time after running packages in Copr to be able to see the diff
- `git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//'`
- `git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep -v :` -> only those which are without bconds, send with parallel to copr (XXX: explain that it has to be built; explain jobs.py/ progress.pkgs meaning)
- `git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep :` -> only those with bconds, run directly srpms:

   for pkg in $(git diff progress.pkgs | grep -E '^\+[^+]' | sed 's/^+//' | grep :); do; copr build --nowait @python/python3.13 --background _fedpkg_cache_dir/$pkg/*.src.rpm; done

- commit and repeat until there are no changes to the diff - it's time to start unblocking

### Next steps - recurring during the whole bootstrap year

#### Unblocking the possibility to build new packages
`jobs.py` outputs:
* The 50 most commonly needed components
* The 20 most commonly last-blocking components
* The 20 most commonly last-blocking small combinations of components
* Detected dependency loops

1. Components

last-blocking means that when this component builds, it will directly unblock the given amount of packages.
To determine what the package needs (and has not been built yet) use `jobs.py` with the package name as the argument:

python jobs.py python-cffi

It can take more arguments, as you go down the dependency tree.
Find the packages that were not built successfully, even though their dependencies are ready - these need to be repaired.

2. Dependency loops

They need to be broken, typically by bconds.
Examine package in dist-git.
If there's an applicable bcond, add it to `config.toml`.
If there isn't, investigate the ways to break the loop (typically by conditionalizing the test run or docs build), commit and submit a PR to the package with an explanation why you do it.
Once merged, add it to `config.toml`.
Every time `config.toml` is updated, run `python bconds.py` to run the new builds and update the cached data.

Note that packages listed in `config.toml` are excluded from dependency loop detection (the detection assumes you broken the loop). But packages evolve over time and the bconds listed in configuration might no longer actually help. The list needs to be maintained.

When done, new packages from `jobs.py` may emerge.

#### Reporting the failures

1. Bugzillas

Start as soon as possible to get community involved in reporting and fixing upstream/downstream.
It's impossible for us to fix everything, so that's crucial.
Some reports will be moot, wrong component, issue not related to the new Python - it's normal and expected.

There's a script for that, [monitor_check.py](https://github.com/befeleme/mini-mass-rebuild/tree/python3.13)

- update the constants in the script: the new Copr name, bugzilla tracker, rawhide tracker and so on
- update the dictionary of failure reasons found in the logs - for the found matches, the description will be less generic and more to the point

In the beginning there will be hundreds of failures with repeating patterns. Take one and add it to the reasons dict in the script.

Then run the script:
`python monitor_check.py --open-bug-reports --with-reason`

This will only open the bugs for packages with matches in the reasons dict.
When you build the starting few reasons and get rid of the repeating patterns, it's time to report the rest.
Drop the `--with-reason` option to open bugz for all the failures.

2. Upstream

Important components that block a lot of packages will use our heavy lifting.
Report upstream, ask the team experts on CPython (vstinner) for help, submit PRs that are in your capacity.

### TODO: Handling discrepancies between distgit state and koji builds/last commit failed to build situations

1. cry
2. cry in fetal position
3. everything is going to be fine