Fedora Release Engineering Repository Reorganization

**Related Pagure ticket:** https://pagure.io/releng/issue/12934 ------------------------------------------------- 1. **`build_composeinfo`** ✅ **Proposed name:** `build-composeinfo.py` **Proposed location:** `builds/compose` **Purpose:** This script generates a `.composeinfo` file for Fedora compose directories. It is a critical part of Fedora's compose infrastructure, generating metadata that other tools use to understand the structure and contents of Fedora distribution trees. **Dependencies (things that rely on this):** - `releng/scripts/build-functions.sh:183` - Called during the compose build process after `pungify` completes. - `releng/scripts/stage-release.sh:75-78` - Called multiple times during release staging for different destination directories. - `pungi-fedora/nightly.sh:119,132,146` - Called three times during nightly compose syncing to generate compose metadata for different target directories. **References to the script (where the name has to be updated):** None other than the dependencies listed above. Not mentioned in any SOP. **Impact if broken:** This would be a critical infrastructure failure. It would block: - Daily Fedora development builds (rawhide/branched). - Official release preparation and staging. - Secondary architecture builds. - Atomic/OSTree compose generation. The script is a single point of failure in multiple critical paths. A break would likely halt Fedora’s entire build and release pipeline until fixed, affecting thousands of users waiting for updates and new releases. The fact that it is called at the end of long-running compose processes makes failures particularly costly, as hours of work could be lost. If this script breaks it will be a "compose yourself" moment for the entire release team. ------------------------------------------------- 2. **`build-current.py`** ❌ **Purpose:** Synchronizes current package builds from primary Koji to secondary architecture Koji instances (ARM-focused, F18 era). <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded for Fedora 18. - Very old ignore list, some of those packages no longer exist. - Uses old Koji API patterns. - Is not a dependency of anything. ------------------------------------------------- 3. **`build-functions.sh`** ❌ **Purpose:** Orchestrates complete Fedora compose creation using `mash`, including repository generation, hardlinking, atomic tree builds, and sync operations. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Expects to be sourced by `buildbranched` and `buildrawhide` scripts but those companion scripts do not exist anymore. - Is not a dependency of anything. ------------------------------------------------- 4. **`build-livecds`** ❌ **Purpose:** Automates building Live CD/DVD images for various Fedora desktop environment spins (Cinnamon, KDE, LXDE, etc.). <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded for Fedora 21. - References URLs that are deprecated. - Obsolete architecture support. - Some of the listed spins no longer exist. ------------------------------------------------- 5. **`build-previous.py`** ❌ **Purpose:** Synchronizes previous (second-newest) package builds from primary Koji to secondary architecture Koji instances for gradual catch-up. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded for Fedora 16. - Very old ignore list, some of those packages no longer exist. - References URLs that are deprecated. ------------------------------------------------- 6. **`build-test-ostree`** ❌ **Purpose:** Creates test OSTree compositions for Fedora Atomic Host across multiple architectures using `pungi-make-ostree`. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded to `f27-build` target. - What it does is now handled by `Pungi`. ------------------------------------------------- 7. **`buildepelbeta`** ❌ **Purpose:** Generates EPEL 7 beta releases using `mash`, including repository creation, dependency checking, and mirror synchronization. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded for EPEL 7 which reached End of Maintenance Support in June 2024. - No EPEL Beta program exists anymore. - References URLs that are deprecated. ------------------------------------------------- 8. **`koji_cleanup_signed.py`** ✅ **Proposed name:** `koji-cleanup-signed-rpms.py` **Proposed location:** `release-process/end-of-life` **Purpose:** This script cleans up signed RPMs after Fedora releases reach End of Life in order to reclaim disk space. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_release_eol.adoc` **Impact if broken:** Disk space would accumulate on `/mnt/koji/packages`. This is a maintenance tool, not critical infrastructure. ------------------------------------------------- > **NOTE ABOUT KOJI INSTANCES** > Historical secondary architecture Koji instances (arm.koji.fedoraproject.org, ppc.koji.fedoraproject.org, s390.koji.fedoraproject.org) have been retired. All of the builds are now handled by the primary koji.fedoraproject.org instance. However, a new secondary Koji instance exists for RISC-V architecture (riscv-koji.fedoraproject.org/koji/) to support emerging hardware development. This infrastructure change renders most of the secondary Koji scripts obsolete, as they either hardcode URLs to non-existent instances or reference deprecated architecture-specific workflows. Scripts using flexible Koji profile configurations may still have utility for RISC-V development. 9. **`koji-build-srpm.py`** ✅ **Proposed location:** `utilities` **Purpose:** This script is a manual utility tool for cross-architecture building between different Koji instances. It downloads source RPMs from the primary Fedora Koji instance and submits them for building on secondary architecture-specific Koji instances (such as ARM, PowerPC, or s390x). The instances no longer exist but a new, RISC-V one does. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** We would not be able to easily submit builds for different architectures. **Additional notes:** As noted above, the old secondary Koji instances are gone now but there is a new, RISC-V instance. Since this script uses the flexible Koji profile system rather than hardcoded URLs it could potentially work with the new instance, if proper profiles are configured. I would say we should not remove it for now. ------------------------------------------------- 10. **`koji-compare.py`** ❌ **Purpose:** Compares package versions between primary and secondary Koji instances. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded URL pattern doesn’t match RISC-V Koji structure. - Assumes `https://%s.koji.fedoraproject.org` format. - Would fail immediately with RISC-V (riscv-koji.fedoraproject.org). - Copyright 2010-2013, Python 2 patterns. ------------------------------------------------- 11. **`koji-import.py`** ❌ **Purpose:** Imports builds from primary Koji to secondary Koji for cross-architecture workflows. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Copyright 2013, over 10 years old. - Python 2 dependencies (urllib2, old patterns). - Designed for retired secondary Koji infrastructure. - No evidence of current usage or RISC-V integration. ------------------------------------------------- 12. **`koji-reimport.py`** ❌ **Purpose:** Resets and re-imports corrupt noarch packages on secondary Koji instances. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Copyright 2014, template script with empty placeholders. - Hardcoded empty package/tag lists requiring manual editing. - Designed for deprecated secondary Koji workflows. - Incomplete implementation state. ------------------------------------------------- 13. **`koji-stalk.py`** ❌ **Purpose:** Monitors fedmsg for completed builds and triggers automated secondary Koji operations. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded to ancient Fedora releases (F20-F23 from 2013-2015). - Uses deprecated `fedmsg` instead of modern `fedora-messaging`. - References non-existent `koji-shadow` infrastructure. - All target releases are End of Life. ------------------------------------------------- 14. **`koji-task-report.py`** ✅ **Proposed location:** `utilities` **Purpose:** Generates statistics about Koji task waiting times and builder utilization. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** This is a pure diagnostic tool. At worst it would be an inconvenience. ------------------------------------------------- 15. **`mashcompose`** ❌ **Purpose:** Creates Fedora 19 branched releases using the `mash` tool with `mock`-based build environments. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - `mash` is no longer used. - Hardcoded for Fedora 19. - Completely replaced by Pungi. ------------------------------------------------- 16. **`mass_rebuild_file_bugs.py`** ✅ **Proposed name:** `mass-rebuild-file-bugs.py` **Proposed location:** `release-process/bug-filing/ftbfs` **Purpose:** Discovers failed builds from Fedora mass rebuilds and files Bugzilla tickets for FTBFS (Fails To Build From Source) packages, including attaching build logs and tracking them under a mass rebuild tracking bug. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_file_ftbfs.adoc` **Impact if broken:** We would not be able to file FTBFS bugs at the time of mass rebuild. ------------------------------------------------- 17. **`mass-rebuild-close-bugs.py`** ✅ **Proposed location:** `release-process/bug-filing/ftbfs` **Purpose:** Closes FTBFS (Fails To Build From Source) Bugzilla tickets that have been fixed after a mass rebuild by detecting successful builds that occurred after the rebuild epoch. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** We would not be able to close FTBFS bugs at the time of mass branching. ------------------------------------------------- 18. **`mass-rebuild-second-run.py`** ❌ **Purpose:** This is a fossil from F35 mass rebuild activities. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded to Fedora 35 (`rebuildid = 'f35'`). - F35 reached End of Life in December 2022, over 2 years ago. - Static list of 800+ packages hardcoded in the script. - No references in SOPs or documentation. - No other scripts import or call it. ------------------------------------------------- 19. **`mass-rebuild-special.py`** ❌ **Purpose:** Performs a special GCC ABI mass rebuild targeting specific packages across two Fedora releases (F26 and F27). Handles cross-branch git operations, spec file bumping, and builds for ABI compatibility testing during GCC upgrades. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Hardcoded to F26/F27 (`buildtag = 'f27'`, `secondbuildtag = 'f26'`). - F26 released July 2017, reached EOL June 2018, over 6 years ago. - F27 released November 2017, reached EOL November 2018, over 6 years ago. - Nothing depends on it. - Not referenced anywhere. ------------------------------------------------- 20. **`mass-rebuild-test.py`** ✅ **Proposed name:** `test-mass-rebuild.py` (standard Python convention) **Proposed location:** `release-process/mass-rebuilds` **Purpose:** This is a unit testing file for the retry mechanisms in `mass-rebuild.py`. **Dependencies (things that rely on this):** Nothing directly depends on this but it is a good idea to have unit tests for our scripts. We should do more of this. **References to the script (where the name has to be updated):** None. **Impact if broken:** It is currently broken. All 3 tests error out with NameError: name 'retry' is not defined because the test file references the retry() function from `mass-rebuild.py` but never imports it. `mass-rebuild.py` has to be refactored to be importable as a module first. ------------------------------------------------- 21. **`mass-rebuild.py`** ✅ **Proposed location:** `release-process/mass-rebuilds` **Purpose:** This is the main script for Fedora mass rebuilds - a critical infrastructure automation tool that rebuilds all Fedora packages during development cycles to ensure compatibility with toolchain updates, new library versions, or other system-wide changes. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** Failure would constitute a critical infrastructure outage, halting mandatory mass rebuild operations and potentially delaying Fedora releases while requiring extensive manual intervention across thousands of packages. ------------------------------------------------- 22. **`massrebuildsinfo.py`** ✅ **Proposed name:** `mass-rebuilds-info.py` **Proposed location:** `release-process/mass-rebuilds` **Purpose:** This is a critical configuration file containing structured data for Fedora mass rebuild operations. **Dependencies (things that rely on this):** - `mass-rebuild.py` - Core mass rebuild automation script. - `mass_rebuild_file_bugs.py` - Files FTBFS bugs after mass rebuilds. - `mass-rebuild-close-bugs.py` - Closes fixed FTBFS bugs. - `mass-rebuild-test.py`. - Unit tests for mass rebuild functionality. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_file_ftbfs.adoc` **Impact if broken:** The four scripts listed above would not work. Failure would disrupt Fedora’s mass rebuild infrastructure by rendering dependent automation scripts unable to access essential configuration data. ------------------------------------------------- 23. **`push-two-week-atomic.py`** ❌ **Purpose:** Automated two-week Fedora Atomic Host release management. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - Fedora Atomic Host deprecated and replaced by Fedora CoreOS with completely different update architecture. - Not mentioned in the documentation. **Dependencies (now obsolete and can be removed):** - `releng/TESTING:14` - Has specific flake8 testing. - `releng/Makefile:54-55` - Dedicated flake8 target for this script only ------------------------------------------------- 24. **`sign-iot.sh`** ❌ **Purpose:** Simple script that signs CHECKSUM files for Fedora IoT composes using Sigul. <ins>***This script is very likely to be obsolete.***</ins> **Evidence:** - IoT composes actively skipped: `compose-tracker.toml` explicitly excludes "IoT" from processing. - No references to "signing IoT" anywhere in the documentation. ------------------------------------------------- 25. **`sigulsign_unsigned.py`** ✅ **Proposed name:** `koji-rpm-signer.py` - `sigulsign_unsigned.py` is confusing because it does not just sign unsigned RPMs (it can sign all RPMs with `--write-all`). - The current name does not indicate it works with Koji. **Proposed location:** `release-process/signing` **Purpose:** This script serves as the core RPM signing utility for Fedora packages, providing a comprehensive interface to Sigul for signing RPMs stored in Koji. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/releng_misc_guide/pages/sop_signing_builds.adoc` - `infra-docs-fpo/modules/release_guide/pages/sop_mass_branching.adoc` - `infra-docs-fpo/modules/release_guide/pages/sop_create_release_signing_key.adoc` - `releng/scripts/sigulsign_unsigned.py` **Impact if broken:** RPM signing would fail completely, blocking all Fedora releases. Unsigned packages cannot be shipped to users, mass branching and release processes would halt, and there would be no way to cryptographically verify package integrity. ------------------------------------------------- 26. **`stage-release-checksum-sign.sh`** ✅ **Proposed location:** `release-process/signing` **Purpose:** This script signs CHECKSUM files for Fedora releases during the staging process. It finds all CHECKSUM files within a compose directory structure and uses Sigul to cryptographically sign them if they are not already signed. The script is designed to run as part of the release staging workflow and must be executed before running the main `stage-release.sh` script. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** Release CHECKSUM files would remain unsigned, preventing users from verifying the integrity of downloaded release media and blocking the final release publication process. ------------------------------------------------- 27. **`stage-release.sh`** ✅ **Proposed location:** `release-process/staging` **Purpose:** This script stages Fedora releases by copying compose artifacts from Koji to the public release directories. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** Fedora releases would not be staged for public access, preventing users and mirrors from downloading release media and effectively blocking the entire release publication process. ------------------------------------------------- 28. **`update_ostree_refs.sh`** ✅ **Proposed name:** `update-ostree-refs.sh` **Proposed location:** `release-process/atomic` **Purpose:** This script manages OSTree references for Fedora’s immutable (Atomic) desktop variants (Silverblue, Kinoite, Onyx, Sericea) during release finalization. It reorganizes OSTree refs by creating updates refs from the main release refs, then making the main refs aliases to the updates refs. This ensures that users who installed from the GA release will automatically follow the updates stream rather than being stuck on the original release compose forever. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_final_release.adoc` **Impact if broken:** Users of Fedora immutable desktops who installed from GA media would not receive updates, as their OSTree refs would point to the static GA compose instead of the updates stream, effectively freezing their systems at GA. ------------------------------------------------- 29. **`branching/create_empty_repos.sh`** ✅ **Proposed name:** `create-empty-repositories.sh` **Proposed location:** `release-process/mass-branching/fedora` **Purpose:** This script creates the initial empty repository structure for a new Fedora release during mass branching. It sets up the complete directory hierarchy for updates and updates-testing repositories across all supported architectures (`x86_64`, `aarch64`, `armhfp`, `i386`, `ppc64le`, `s390x`) and variants (`Everything`, `Modular`), including subdirectories for packages, debug symbols, and delta RPMs. The script then initializes each directory with empty repository metadata using `createrepo_c`. **Dependencies (things that rely on this):** Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_mass_branching.adoc` - `infra-docs-fpo/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc` **Impact if broken:** New Fedora releases would lack the proper repository structure for updates, preventing Bodhi from publishing updates and leaving users unable to receive package updates for the new release. With that in mind, it is somewhat worrying that the very first comment in the script says: `#This whole script needs improvement, it is just a quick fix.` ------------------------------------------------- 30. **`branching/get_all_active_packages_branching.sh`** ✅ **Proposed name:** `get-active-packages.sh` **Proposed location:** `release-process/mass-branching/fedora` **Purpose:** Builds a curated list of active Fedora packages for mass branching by scanning package repositories, filtering out retired or incomplete packages, and generating a text file for downstream branching tools. **Dependencies (things that rely on this)**: Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_mass_branching.adoc` - `/infra-docs-fpo/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc` **Impact if broken:** Mass branching would fail or be incomplete. Missing packages would not get new release branches, while retired packages might incorrectly receive branches, requiring manual cleanup and delaying the entire release timeline. ------------------------------------------------- 31. **`branching/koji-http-toggle.sh`** ✅ **Proposed location:** `release-process/mass-branching/fedora` **Purpose**: Provides a utility to quickly block or unblock external HTTP submissions to Koji during mass branching. **Dependencies (things that rely on this)**: Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **Impact if broken:** During mass branching, inability to block external submissions could lead to unwanted package builds interfering with the branching process, but the mass branching could continue with manual firewall rules as documented in the SOP. ------------------------------------------------- 32. **`branching/make-koji-release-tags`** ✅ **Proposed name:** `create-koji-release-tags.sh` **Proposed location:** `release-process/mass-branching/fedora` **Purpose**: Creates the complete Koji tag hierarchy for a new Fedora release during mass branching by cloning the previous release’s base tag, establishing tag structures with proper inheritance relationships, permissions, and build targets, and updating `rawhide` to point to the new release. **Dependencies (things that rely on this)**: Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_mass_branching.adoc` - `infra-docs-fpo/modules/release_guide/pages/sop_mass_branching_checklist.adoc` - `/infra-docs-fpo/modules/release_guide/pages/sop_epel_minor_mass_branching.adoc` **Impact if broken:** Mass branching cannot proceed without proper Koji tags. The entire build infrastructure for the new release would be non-functional, preventing package builds, updates, containers, and flatpaks from being created or distributed. ------------------------------------------------- 33. **`branching/torrent-generator`** ✅ **Proposed name:** `torrent-generator.py` **Proposed location:** `release-process/torrents` **Purpose:** Generates web pages and RSS feeds for Fedora’s BitTorrent tracker by reading `.ini` files containing torrent metadata and producing formatted HTML tables and RSS feeds listing available torrents grouped by release with descriptions, sizes, and dates. **Dependencies (things that rely on this)**: Nothing directly depends on it. This script is a standalone administrative utility that has to be run manually. **References to the script (where the name has to be updated):** - `infra-docs-fpo/modules/release_guide/pages/sop_torrent_releases.adoc` **Impact if broken:** Torrent web pages would not update with new releases, making it difficult for users to discover and download new Fedora torrents. While existing torrents would still work, new releases would not be properly indexed. **Age concerns:** While functionally required, this is unmaintained legacy Python 2 code from the 2007 with hardcoded developer paths and outdated branding that needs conversion to Python 3 and configuration updates before Python 2 EOL makes it non-functional. ------------------------------------------------- Ansible: ------------------------------------------------- **Curated list of repositories that were taken into consideration:** - https://pagure.io/fedora-infra/ansible - https://pagure.io/releng/compose-tracker - https://pagure.io/dusty/failed-composes - https://pagure.io/fedora-comps - https://pagure.io/fedora-infrastructure - https://pagure.io/fedora-kickstarts - https://pagure.io/fedora-kiwi-descriptions - https://pagure.io/fedora-lorax-templates - https://src.fedoraproject.org/rpms/fedora-release - https://src.fedoraproject.org/rpms/fedora-repos - https://pagure.io/releng/fedora-scm-requests - https://pagure.io/infra-docs-fpo/ - https://pagure.io/pagure-dist-git - https://pagure.io/pungi-fedora - https://pagure.io/fedora-infra/toddlers - https://pagure.io/workstation-ostree-config