# Python Backend Bootstrapping

Jason R. Coombs
2024-07-08

## Overview

System integrators wishing to build environments from source have a bootstrapping problem: if a build backend (e.g. Setuptools or flit) has a (possibly transitive) dependency (e.g. wheel) that is built by that build backend, there is no way break the cycle.

This limitation is captured briefly in [PEP 517 build requirements](https://peps.python.org/pep-0517/#build-requirements), though that section understates the scope of the limitation, which applies not only to build dependencies but to runtime dependencies (regardless of how indicated in `build-system.requires`).

Currently, this limitation has forced backends not to have dependencies (flit-core), to declare dependencies that don't use the same backend (hatchling), to vendor dependencies (setuptools), or not to support pure-source integration (coherent.build).

This design aims to lift this limitation, create a systemic approach that anyone can apply to bootstrapping, and allow build backends to have and declare dependencies naturally and without special handling by the backend.

## Definitions

*System Integrators* are those teams or projects building Python environments as part of a larger system such as Debian or Spack.

*Backend* is [PEP 517](https://peps.python.org/pep-0517/) build backend.

*Frontend* is a PEP 517 build tool for building a package from source into an installable artifact.

*Backend Resource* is a backend project or one of its dependencies.

## Background

Prior to PEP 517 and 518, Setuptools was the de facto backend (and even front end) for installing packages, and even then, Setuptools could not declare dependencies due to the circular dependencies problem ([pypa/setuptools#229](https://github.com/pypa/setuptools/issues/229) and more).

In an aim to satisfy the implicit requirement that backends have no build or runtime dependencies (sits at the bottom of the DAG), Setuptools vendored the dependencies it required.

Today, thanks to the advances of the build backend specifications, there are a number of build backends each exploring different innovations and approaches to this problem and others. This diversity of innovation, while welcome and beneficial, also means that there is no one backend that can address this issue.

## The Challenges

### Vendoring is Unsustainable

While vendoring dependencies does provide a workaround to break the cycle, it creates a number of subtle and not-so-subtle problems:

- vendored dependencies must be provided as pure-Python, platform-agnostic sources or somehow bundle all possible platform-specific behaviors.
- vendoring necessarily pins the bundled versions, requiring constant maintainence.
- integrators will typically unvendor the dependencies and have to reconcile the versions.
- vendored packages are often masked by or mask naturally installed packages.
- systems designed to manage compatible versions have limited visibility to the installed versions.
- essential metadata (entry points) can be missing.
- adding, removing, or updating a dependency creates a huge diff, masking the essential behaviors that drove that change.
- adds substantial size to the package.
- leads to reluctance to adopt useful behaviors.
- some techniques require complicated patches to rewrite imports and other behaviors to support the vendored model.
- vendored packages are only present under certain conditions; order of import affects which versions are available.
- has to be handled by each backend (no systemic approach).
- dependencies cannot be declared, so cannot appear in metadata.
- users get an inconsistent experience when working with a build backend compared to other projects.

### Invisible to Typical Developers

Users of conventional installers on the Python ecosystem (pip, uv) do not encounter this problem as

(a) the community has aggressively pursued publishing of pre-built artifacts (wheels) for most if not all packages, and
(b) conventional installers can bootstrap a build backend using these pre-built artifacts.

It's only environments that wish to build everything from source (including `pip install --no-binary :all:`) where the circular dependency arises.

The approach of requiring pre-built artifacts has meant that other parties (downstream indexes, such as private corporate indexes) must also supply pre-built artifacts and cannot operate source-only.

### Hamstrung Backends

The inability of a backend to declare dependencies without breaking these environments has meant that backend projects are subtly restricted from using standard packaging techniques or from leveraging sophistication available in the Python ecosystem.

Moreover, these limitations are subtle and often difficult to describe and replicate, leading to false starts and confusing reports.

### Affects Self-Builders

Some build backends would like to use that backend to build themselves. There are several ways a project can self-build:

- have no dependencies and provide a `backend-path` to use one's own unbuilt source to build ([flit](https://github.com/pypa/flit/blob/e38b172ca415e0b41c82801131d4e09a51e4bd85/flit_core/pyproject.toml#L2-L4)), or
- simply declare the dependency on the project by name and require a previous build to build newer versions ([coherent.build](https://github.com/coherent-oss/coherent.build/blob/fa07781abbaeea98459a20a7f136590c93c8fd3c/system.toml#L2)), or
- provide a custom backend for bootstrapping ([hatchling](https://github.com/pypa/hatch/blob/3adae6c0dfd5c20dfe9bf6bae19b44a696c22a43/backend/pyproject.toml#L2-L4)).

Only the "no dependencies" approach works in general.

### Affects Non-Backend Projects

Because the issue affects dependencies of backends, those dependencies are also constrained by the backend that elected the dependency.

For example, `hatchling` is currently dependent on a few packages, all of which are currently built by other backends without dependencies (`setuptools` and `flit-core`). If, however, one of those dependencies were to switch to `hatchling`, it would break source-only builds.

It's not enough to special-case the backends, but to special-case the backends and every dependency of every backend.

## Objective

It should be easy and straightforward for a build backend and its dependencies to declare their dependencies naturally. Build systems that can build from existing artifacts already support this model. Systems that rely on everything from source need facilate the bootstrapping. This document outlines a methodology that such system integrators can use.

## Design

Since the existing frontends support buliding these backends by leveraging pre-built artifacts, it demonstrates an approach that's viable and works well for users.

System Integrators could mimic this approach in their build system. For each build backend that exists, the build system would keep a bundle of pre-built artifacts (wheels) necessary to build that backend and its dependencies. These artifacts would be kept with the sources for the system and any time one of the backend resources needs to be built, the build system will make those resources available for the build.

Note that these build resources could still be built from source to be installed into the system - they would only rely on the pre-built artifacts long enough to bootstrap the build resources themselves.

Thereafter, all build resources can declare their dependencies naturally.

System Integrators could coordinate the development and maintenance of these pre-built backends, perhaps maintaining them as importable source artifacts or as opaque wheels. These artifacts could be shared across projects, lessening the maintenance.

This approach essentially shifts the burden of vendoring from build backends to system integrators and seems to be the only approach that will allow build backends to declare dependencies.

## Alternatives Considered

### Source Aggregation

Another possible approach could be for a bootstrapping methodology that doesn't rely on existing built artifacts, but instead relies on collecting source artifacts. Similar to the design above, system integrators would resolve all build resource sources, expand them into a staging environment, and manually "build" them enough to make them viable for import.

This approach unfortunately does not generalize well, as without the build backend to perform the build, there are any number of sophisticated behaviors that would likely be missed or need to be replicated to make a project viable. This approach would require a substantial amount of maintenance to monitor existing and emergent backend resources and ensure they have a bootstrap recipe.

Any such system would need to incorporate a wide variety of behaviors and conventions across build systems.

### Direct all Backend Projects to One Backend

Since `flit-core` requires no dependencies and is thus unaffected by the issue, it's been floated to simply require all build backends and their dependencies to use `flit-core` (or some other compatibile dependency).

This approach would substantially limit the features available to build backends and is philosophically incompatible with strategies to reduce static config.

Furthermore, the interaction between backends and their dependencies makes any constraint on a backend apply to its dependents, so is untenable in general. The backend needs to manage the build and runtime dependencies of all of its dependencies.

### Require Backends to Honor a DAG

Similar to directing all backends to one backend, allow backends to have dependencies, but only on a hierarchy or tier of sophistication. Any backend project can only depend on projects built with a backend on a lower tier.

Flit-core would be tier 0 (no dependencies).

Hatchling is on tier 1, requiring all of its dependencies to be on tier 0.

This approach would require a complicated intercoordination of projects and their dependencies. Any project participating in this system would need to agree to follow a protocol for validating compliance before adopting any dependency, and any dependency adopted by a backend would need to participate in this protocol.

### Compile to Flit-core (Sdist)

[coherent-oss/system#12](https://github.com/coherent-oss/system/issues/12) conceives of a possibility where a build backend could do all of its sophisticated work up front and essentially "compile" the result to an sdist depending only on flit-core (or other simple backend).

This approach would have the advantage of creating artifacts that are readily installable without build cycles. It would still impose that all backend resources be built by flit-core, but wouldn't impose as many limitations on the sophistication of the primary backend.

This approach would not work for Setuptools, at least not in the current regime, as the Setuptools is required at build_wheel time to compile extension modules. Perhaps an alternative backend could be built that provides C/C++ extension module support atop flit-core.

### Document but Keep the Status Quo

If it's deemed untenable to allow backends to declare dependencies or for those dependencies to form a DAG to no dependencies, at the very least, these limitations should be documented as a resource for backend developers to reference when designing their backend.

Currently, the problem is poorly understood, with the requirement being [briefly mentioned in PEP 517](https://peps.python.org/pep-0517/#build-requirements), but does not cover that runtime dependencies are also implicated or that a project cannot have any dependencies that rely on that backend.

### Each Backend Maintains Vendored Artifacts

Similar to the current vendoring, each backend would maintain a set of importable artifacts that an integrator would be responsible to make available when bootstrapping the package or its dependencies.

This set of artifacts could (should) be associated with a given release, and may be stored in the same repository (`pypa/setuptools/_vendor`) or a separate one (`pypa/setuptools-deps`). The artifacts would not be packaged with the sdist in PyPI.

This way, the backend itself would declare dependencies naturally, but a system integrator would checkout and add to the PYTHONPATH the vendored dependencies when building any backend resource.

## Resources

[pypa/packaging-problems#342](https://github.com/pypa/packaging-problems/issues/342) captured the early motivations and discussions around the problem from the perspective of the Python ecosystem.

Doug Hellman's [fromager](https://github.com/python-wheel-build/fromager) aims to provide a possible implementation for build bootstrapping, compatible with this design.