# Nix Python Packaging

- 1-2 sentence description of the project
- When was the project created?
- Why was the project created?
- What is the project's scope as relates to Python distribution/packaging?
- Build Python interpreters?
- Distribute Python interpreters?
- Create Python packages?
- Distribute Python packages?
- Install Python packages?
- What is the project specifically trying to excel at, generally?
- What is the project specifically trying to excel at, as relates to Python distribution/packaging?
- What are some key weak points of the project, as relates to Python distribution/packaging?

## 1-2 sentence description of the project

Nix is a package manager with a focus on reproducible builds from the point of metadata/inputs.

It also features a declarative language in which these builds are written in, the Nix language.

Nixpkgs is a set of packages shipped by default when using Nix.

NixOS is a declaratively configured Linux distrubtion built on top of Nixpkgs.

## When was the project created?

Nix has been around for more than 20 years and Python support has been around since the very early days.

## Why was the project created?

An experiment to make package management stateless by
applying funtional programming paradigm ideas.

Strong focus on reproducible builds, i.e. repeatable process
of going from source to a working software.

*Following is the motivation from the PhD thesis*

To resolve issues with existing systems:

- Dependency specifications are not validated, leading to incomplete deployment.
- Dependency specifications are inexact (e.g., merely nominal).
- It is not possible to deploy multiple versions or variants of a component side-by-side.
- Components can interfere with each other.
- It is not possible to roll back to previous configurations.
- Upgrade actions are not atomic.
- Applications must be monolithic, i.e., they must statically contain all their dependencies.
- Deployment actions can only be performed by administrators, not by unprivileged users.
- There is no link between binaries and the sources and build processes that built them.
<!-- - The system supports either source deployment or binary deployment, but not both; or it supports both but in a non-unified way. -->
<!-- - It is difficult to adapt components. -->
<!-- - Component composition is manual. -->
- The component framework is narrowly restricted to components written in a specific programming language or framework.
- The system depends on non-portable techniques.
<!-- - The objective of the research described in this thesis is to develop a deployment system that does not have these problems. -->

## What is the project's scope as relates to Python distribution/packaging?

### Build Python interpreters?
- CPython multiple versions
- PyPy multiple versions
- Multiple OS: Linux, MacOS, no native Windows support
- Multiple CPU architectures

### Distribute Python interpreters?
- Recipes to build from source along with pre-built binaries

### Create Python packages?
- Helper functions for packaging Python packages with Nix and composing environment
- Outside Nixpkgs important tools such as `poetry2nix`

### Distribute Python packages?
- All handled by the Nix package manager

### Install Python packages?
- Any Python package shipped by Nixpkgs can be installed.
- Where to draw the line what to package?

### What is the project specifically trying to excel at, generally?

Reproducible builds with sandboxing at the package granurality.

- Locking dependency trees all the way from your leaf package down to your `glibc` or even Linux kernel (on NixOS), and other tools and libraries used along the way.

- Build time dependencies

- Creating declarative/imperative developer environments

### What is the project specifically trying to excel at, as relates to Python distribution/packaging?

Handles system dependencies - over 90 0000 packages in Nixpkgs.

Relocatable - can be copied on any other machine with the same platform.

Composition of Python environments
- separate path per package

<!--

Offering any Python package seamlessly as part the 90 000 packages in Nixpkgs.

All that while providing first class customisation options.
-->

### What are some key weak points of the project, as relates to Python distribution/packaging?

- Nix requires a DAG and thus cannot handle cyclic dependencies, while Python allows it.

- Dealing with native dependencies that are opened using e.g. `ctypes`. We need to patch all of these occurences.
- On the flip side, once captured, native dependencies are locked and will guarantee exact reproducibility (from-source builds and cached binaries) on any supported platform.

- In Nixpkgs only a single version of a Python package is provided. For example, there is no version constraint solver. However, `poetry2nix` can use `poetry` to lock Python dependencies on top of Nixpkgs providing the non-Python components.

- Impure builds using e.g. `pip` directly on top of system-provided packages. Especially with NixOS there are no base system packages.

### Limitations of Python packaging for the project

- Lack of standardized lock file format for applications

- Mapping native dependencies into Nix dependencies is manual work, there's no pkgconfig support in Python.

- Non-Python package managers invoked as part of the build, such as npm/yarn/cargo

- Lack of test dependencies / runner specifications.

- Automating conversion of Python packaging metadata into Nix packages using a single command (for example using poetry2nix).

added by dream2nix maintainers:
- dependencies are not full declarative: some packages require executing python code to resolve their dependencies, which in turn can require undeclared system dependencies.