Meeting notes - 19-12-2023 - Multiple environments in pixi

# Meeting Title **Date:** 19-12-2023 **Location:** [meeting](https://calendar.google.com/calendar/event?action=TEMPLATE&tmeid=MmJnZXZvamNwZzE1aTZiMGswNTFhZjMzcGcgcnViZW5AcHJlZml4LmRldg&tmsrc=ruben%40prefix.dev) ## Attendees | Name | Company | GitHub Handle | |------------------|----------------|-----------------------| | Ruben Arts | prefix.dev | @ruben-arts | | Tim de Jager | prefix.dev | @tdejager | | Wolf Vollprecht | prefix.dev | @wolfv | | Travis Hathaway | Anaconda | @travishathaway | | Bela Stoyan | QuantCo | @0xbe7a | | Pavel Zwerschke | QuantCo | @pavelzw | ## Agenda 1. Introduction 2. Create a set of usecases together. 3. Make a decision on the `toml` format. 4. Make a proposal for the cli 5. Discuss shared work ## Discussion Points - **Introduction** - [Discussion detail or summary] - **Create a set of usecases together** - Multiple package versions in one project - Smaller environment for single tools, or tasks. - Dev environments. - Strict superset of the production environment for testing. - Ability to combine environments. - Multiple machines in one project. - Variant configuration for `pixi build` - **Make a decision on the `toml` format** - [Discussion detail or summary] - **Make a proposal for the cli** - [Discussion detail or summary] - **Discuss shared work** - [Discussion detail or summary] ## Action Items - [ ] **Move the below specified proposal into pixi to keep track of the design** - [ ] **Preliminary notion of an Environment in pixi** - [ ] **Start a development branch for this feature** - [ ] **Define sub features required for the multiple environment feature** ## Additional Notes - [Any other important notes or information] ## Next Meeting? # Proposal Design space:  ## Objective The aim is to introduce a enviornment set mechanism in the `pixi` package manager. This mechanism will enable clear, conflict-free management of dependencies tailored to specific environments, while also maintaining the integrity of fixed lockfiles. ### Motivating Example: Task specifc or Test Dependencies and Multiple Python Versions Consider a scenario where a project needs to be tested across multiple Python versions, each requiring a different set of dependencies. In this case, defining separate feature sets for each Python version (like `py39`, `py310`, etc.) allows for easy switching between environments without conflicts. Similarly, for development purposes, a `test` feature set can include dependencies necessary for testing and linting, which are not required in the production environment. Alternatively you could have a really bare bones environment specifically for one task like linting or documentation. ## Design Considerations 1. **User friendliness**: Pixi is a user focussed tool this goes beyond developers. The feature should have good error reporting and helpful documentation from the start. This is opinionated so the user sided PR's should be checked by multiple developers. 2. **No Automatic Combinatorial**: To ensure the dependency resolution process remains manageable, the solution should avoid a combinatorial explosion of dependency sets. By making the environments user defined and not automatically infered by testing a matrix of the features. 3. **Single Feature Activation**: The design should allow only one feature set to be active at any given time, simplifying the resolution process and preventing conflicts. 4. **Fixed Lockfiles**: It's crucial to preserve fixed lockfiles for consistency and predictability. Solutions must ensure reliability not just for authors but also for end-users, particularly at the time of lockfile creation. 5. **Keep it simple**: Not understanding the multiple environments feature shouldn't limit a user to use pixi. The feature should be "invisible" to the non multi env use-cases. ## Proposed Solution ### Feature & Environment Set Definitions Introduce environment sets into the `pixi.toml` this describes environments based on `feature`'s. Introduce features into the `pixi.toml` that can describe parts of environments. As a environment goes beyond just `dependencies` the features should be described including the following fields: - `dependencies`: The conda package dependencies - `pypi-dependencies`: The pypi package dependencies - `system-requirements`: The system requirements of the environment - `activation`: The activation information for the environment - `platforms`: The platforms the environment can be run on. - `channels`: The channels used to create the environment. - `target`: All the above features but also seperated by targets. - `tasks`: Feature specific tasks ```toml= [dependencies] # short for [feature.default.dependencies] python = "*" numpy = "==2.3" [feature.py39.dependencies] python = "~=3.9.0" [feature.py310.dependencies] python = "~=3.10.0" [feature.test.dependencies] pytest = "*" [feature.cuda] dependencies = {cuda = "x.y.z", cudnn = "12.0"} platforms = ["linux-64", "osx-arm64"] activation = {scripts = ["cuda_activation.sh"]} channels = ["conda-forge", "nvidia"] # Would concat instead of overwrite [feature.lint.dependencies] pre-commit = "*" [feature.test.tasks] test = "pytest" [environments] # implicit: default = ["default"] default = ["py39"] py39 = ["py39"] py310 = ["py310"] test = ["test"] test39 = ["test", "py39"] cuda = ["cuda"] lint = {features = ["lint"], default-features = "false"} prod = ["py39"] # NOT: test = ["py39", "test"] test_prod = {environments = ["prod"], features = ["test"]} # future thoughts # stacks_on => activate_after=? test = {extends = "prod", stacks_on = "bla", features = ["test"]} ``` ### Lockfile Structure Within the `pixi.lock` file, a package may now include an additional `environment` field, specifying the environment to which it belongs. This structure ensures clarity. This will duplicate, a possible improvement is to allow for deduplication by flattening the packages that are used in multiple environments. ### User Environment Activation Users can manually activate the desired environment via command line or configuration. This approach guarantees a conflict-free environment by allowing only one feature set to be active at a time. For the user the cli would like like: ```shell pixi run -e test pytest pixi run python # default env pixi shell -e cuda pixi run --environment test pixi shell --environment cuda pixi run test # In the scenario of multiple environments having the test feature. # Which env? # 1. test # 2. test39 ``` Tasks can also except different environments as defaults ```toml [tasks] test = {cmd = "pytest test/", env = "test"} lint = {cmd = "pre-commit run -a", env = "lint"} ``` ### Extending environments For testing how environments work in production, it would be nice to strictly pin the test environment as close to the prod environment as possible. ```toml [dependencies] python = "*" [feature.py39] python = "3.9.*" polars = "*" [feature.test.dependencies] pytest = "*" prod = ["py39"] test-prod = {environments = ["prod"], features = ["test"]} ``` `environments = ["prod"]` means that `test-prod` contains exactly the same dependencies as `prod`. If we only did ```toml test-prod = ["py39", "test"] ``` it could occur that the dependencies are solved in a different way and that our production code doesn't behave the same as in our testing setup.