# Mixing spack and python packages in the same env

The important point about mixing spack envs and python venvs is how to combine them in a way that all the folders containing python packages appear in the right order in the`sys.path` of the final venv. The `sys.path` variable contains the actual sorted list of paths cPython will look into when loading a module, and it is filled during initialization according to:
- run-time arguments (e.g. script mode vs module mode, skip site initialization (-S) or not, ...)
- relevant environment variables (e.g. PYTHONPATH)
- information stored in the interpreter file tree (root prefix, pyenv.cfg, ...)

Check the [_Understanding Python `sys.path` initialization_](#Understanding-Python-syspath-initialization) section below for details.

## Building a hybrid python stack

In our use case, we want to build a python environment with a specific set of packages. Some of these packages are installed with `spack` (in different folders), and some others are installed with `uv`/`pip` in the standard `site-packages` location. Spack uses `PYTHONPATH` to add the location of the spack py-packages while installing the packages, and also sets the variable when activating the environment for no good reason since it's not needed. Setting `PYTHONPATH` when activating the environment affects how the `sys.path` will be computed at python initialization for **ALL** python interpreters executed after loading the uenv, not only the one inside the uenv, which is a really nasty side-effect.

Ideally the list of steps to build a well-behaved python venv should be:

- use spack to select and build the python interpreter version
- use spack to select and build extension packages with heavy C/C++/Fortran dependencies like `numpy` or `cupy`
- create a venv (with `uv` or `venv`) with the option `--system-site-packages` which uses as base installation the interpreter built by spack
- install the rest of python packages in the `venv` as usual
- unset `PYTHONPATH` when activating the spack env (check the `unset` option here: https://spack.readthedocs.io/en/latest/environments.html#modifying-environment-variables)

In principle, this could work and fill the `sys.path` of the python interpreter inside a custom venv with all the folders with python packages in the right order (venv > spack_env) and without affecting any other python interpreter in the system. Check [_Remarks from spack developers_](#Remarks-from-Spack-developers) for more details.

However, there is a problem because the spack view where all the packages are installed is actually a venv, so it can't be used as base python for another venv because of the _3-level problem_.

## The 3-level problem

Creating a venv (e.g. `venv-a`) with the `--system-site-packages` option provides a way to layer a venv on top of a _system-like_ python `base` installation. However, it is not possible to use the same approach to layer a venv (e.g. `venv-b`) on top of another venv: both venvs will have the same `base` prefix of the actual base python instead.

### General solution with `.pth` files

The only general solution I can think of to layer an arbitrary number of venvs on top of each other is to add `.pth` files in the upper layer, pointing to the `site-packages` folder of the lower layers with the right names so, when sorted alphabetically, they reproduce the desired stack of layers. Additionally, the `.pth` files inside the layered venv's `site-packages` folders won't be recursively scanned at loading time, so they should also copied to the upper layers and renamed with some prefix matching its original layer, so they are also loaded in the correct order.

### Simpler solution for just 3 levels

If we are only interested in layering exactly 3 levels like the standard spack view + venv case:
- base: python interpreter installed by spack (probably without any 3rd party packages)
- middle layer: venv created by a spack view which contains all the python packages installed by spack
- top layer: custom venv on top of the spack view

we could just define the`PYTHONUSERBASE` environment variable pointing to the root of the spack view venv, and create the custom venv normally (although `--system-site-packages` still needs to be set!) using the spack-installed python as base prefix. The user site-packages folder is added to the `sys.path` after the venv site-packages folder, but before the system site-packages folder, which means is exactly at the middle layer.

Since this solution relies on defining a environment variable, it still has a small-side effect on other python interpreters in the system, which will see the uenv view as user site-packages folder. However, this is not so important because:
- packages installed on the venvs have higher priority
- venvs created without the `--system-site-packages` option won't use `PYTHONUSERBASE` at all
- `PYTHONUSERBASE` can be fully disabled by launching any interpreter with the [`-s` option](https://docs.python.org/3/using/cmdline.html#cmdoption-s).

    python -s # don't add the user-site to the sys.path

## Understanding Python `sys.path` initialization

### References

- [sysconfig](https://docs.python.org/3/library/sysconfig.html)
- [site](https://docs.python.org/3/library/site.html)
- [venv](https://docs.python.org/3/library/venv.html)

The actual implementation of [site.py](https://github.com/python/cpython/blob/main/Lib/site.py) is also very useful to see all the details.

### Testing from the command line

The following command line template is quite useful to see the effects on `sys.path` of changing different interpreter settings:

PYTHONPATH="foo:bar:..." python [-S] -c 'import sys; print(f"{sys.prefix=}\n\n{sys.base_prefix=}\n\n{sys.path=}")'

### Pseudo-code

This pseudo-code snippet shows a simplified version of how the `sys.path` variable is assembled, according to (my understanding of) the documentation references:

__site_packages__ = not ({"-S", "-I"} & set(sys.argv))
__user_site_packages__ = not (
    ({"-s", "-I"} & set(sys.argv))
    or os.getenv("PYTHONNOUSERSITE", default="")
)
__is_venv__ = os.path.exists(cfg := f"{sys.prefix}/pyenv.cfg")
__system_site_packages__ = __is_venv__ and (
    ConfigParser().read(cfg).get("include-system-site-packages") == "true"
)

# start_dir can be disabled by '-P', '-I' or '$PYTHONSAFEPATH'
start_dir = dirname(__script_arg__) if __script_mode__ else os.curdir

# python_path_env can be disabled by '-E' or '-I'
python_path_env = os.getenv("PYTHONPATH", default="").split(":")

def get_site_dirs(*base_dirs):
    all_dirs = [
        f"{head}/{tail}"
        for head in base_dirs
        if head
        for tail in ("", "lib/pythonX.Y/site-packages")
        ]
    # remove duplicates keeping the same order
    unique_dirs = list(dict.fromkeys(all_dirs))
    # look for dirs in .pth files 
    dirs_in_pth_files = [
        path
        for d in unique_dirs
        for f in sorted(glob("*.pth", root_dir=d))
        for path in f.split()
    ]
    return [*unique_dirs, *dirs_in_pth_files]

site_dirs, users_site_dirs, system_site_dirs = [], [], []

if __site_packages__:
    site_dirs = get_site_dirs(sys.prefix, sys.exec_prefix)

    if __user_site_packages__:
        user_base = os.getenv("PYTHONUSERBASE", default="~/.local")
        user_site_dirs = get_site_dirs(user_base)
        
    if __system_site_packages__:
        system_site_dirs = get_site_dirs(
            sys.base_prefix, sys.base_exec_prefix
        )

sys.path = [
    start_dir,
    *python_path_env,
    *__sys_dirs__,
    *site_dirs,
    *user_site_dirs,
    *system_site_dirs,
] 

## Remarks from Spack developers

From a conversation on Spack Slack with Harmen:

- _installation time_ is different from _views_: upon installation PYTONPATH is all you have
- environment views are always virtual environments, as long as all relevant python packages do `extends("python")`
- Spack computes all runtime variables PATH, MANPATH etc, the only difference with a view is that after computation those prefixes are projected to the view and deduplicated, so `PYTHONPATH=/pkg-1/lib/python3.11/site-packages/:/pkg-2/lib/python3.11/site-packages/:...` is projected and deduped to `PYTHONPATH=<view>/lib/python3.11/site-packages`
- short version: installation requires PYTHONPATH, which spack sets for you. At runtime with environment views, PYTHONPATH is redundant but is still set when you load a view for no particular reason other than that all follows the same code path :stuck_out_tongue: