# EESSI + Spack

## 20260224

attendees:
- EESSI: Loris, Kenneth, Alan
- Spack: Todd, Massimiliano (Max)
- JSC: Jayesh
- Kitware: Ryan Krattiger

### Notes

We can expose EESSI's software to Spack using by defining builds as **external packages** in a `packages.yaml` configuration file.

- Blog post showcasing this approach: [*Using Spack on Top of EESSI: Best of Both Worlds*](https://www.eessi.io/docs/blog/2026/02/05/Spack-on-top-of-EESSI-best-of-both-worlds/)
- Demo code with examples available in the [Spood repo 🕸️](https://github.com/lorisercole/spood)

#### Problematic compatibility-layer packages

We use these commands to detect packages from EESSI *compatibility layer*:

spack external find --all -p ${EESSI_EPREFIX} --exclude gcc
spack external find --all -p ${EESSI_EPREFIX}/usr --exclude gcc

- However, I question whether this is always necessary.
- Some of them showed to be problematic: `curl` and `zlib` do not work properly when building a new `cmake` from source (see [error report](https://github.com/lorisercole/spood/blob/feature/mpi/notes/compat_pkgs_problems.md)). Consequently, I'll exclude them from the detection, and remove them from external package dependencies.
- `curl` is available in the software layer
- `zlib` is not, so it will need to be rebuilt (not a big drawback IMO)

Notes:

- Some issues during the configure step, finds version but for some reason not picked up properly.
- Does Spack find the headers? binaries is the most common case, there are some packages that look for libraries. Would need to actually check the implementation of find for the individual package.
- This could lead to contention as the dependency `zlib`/`curl` must come from somewhere, but will they come from Spack or EESSI?

### Experimenting with OpenMPI

- I've tried to include the `OpenMPI/4.1.6-GCC-13.2.0` installation provided by EESSI.
- It has many dependencies: `hwloc`, `libevent`, `ucx`, `libfabric`, `pmix`, `ucc`
- and many variants.
Spack detection tool helps in this case (often I need to compare the EasyBuild build recipes agains Spack's ones to be sure).


- I've also added FFTW+MPI, FlexiBLAS, ScaLAPACK.

Here is the resulting YAML: [externals_mpi.yaml](https://github.com/lorisercole/spood/blob/feature/mpi/examples/ext_install/externals_mpi.yaml)

- The `libpciaccess` link-dependency of `hwloc` is the only strictly mandatory: without it, the MPI linker fails. This derives from the OpenMPI installation: see more details [here](https://github.com/lorisercole/spood/blob/feature/mpi/notes/NOTES.md#openmpi-tests).

- **LAMMPS builds successfully with OpenMPI !** 🥳

- QuantumESPRESSO showed a [CMake configuration problem](https://github.com/lorisercole/spood/blob/feature/mpi/notes/NOTES.md#quantumespresso-build-problem) due to the build recipe:
- CMake cannot find OpenMP_Fortran libraries when one manually specifies `-DCMAKE_Fortran_COMPILER=mpif90` and `-DCMAKE_C_COMPILER=mpicc`
- This is only a problem when using our external OpenMPI (not built by Spack -- note: I could not try other external OpenMPI installations)
- I'm not sure why these flags are used. **Without them it works smoothly!** I will submit a PR to `spack-packages`...

Notes:

- How does the EasyBuild and Spack recipes compare?
- Spack has a _lot_ of flags, many more than EasyBuild
- EasyBuild leaves a lot of things to defaults which means they may change based on the system
- They use different defaults sometimes
- For OpenMPI, `libpciaccess` was required for linking, which comes via `hwloc`
- Should be solved by setting `LIBRARY_PATH` but not clear why this was not included
- Quantum Espresso
- CMake problem when explicitly defining the compiler
- Saying the MPI compilers are the compilers for CMake might be "wrong" in general

### To Do
- Add YAML specs of important/complex packages like `boost`, ...
The number of variants make it a tedious process.
- Write a mapping EasyBuild --> Spack

### AOB
- Some things can be automated but there is a lot of manual work currently
- EESSI packages require mapping to Spack variants
- Basics are quite feasible: compilers, MPI, basic libraries
- Can be used as a starting point for Spack
- CP2K is a good test case
- The more we add the faster it is
- Python packages (and extensions in general) are very granular in Spack, in EasyBuild there are many cases of bundles
- You can an external find for Python, but not for Python packages
- `pip freeze -v` gives a lists
- if you run `python` you can find the packages are
- some Spack packages have different names than what pip says
- In EESSI, they come in bundles and there's no way to take a subset. Spack will not catch that multiple packages are provided and may conflict
- The effort is only worth it if the package is expensive to build
- bundles are not that large in general
- issue will be that the graph will be correct but the fact that you load python modules from a certain location(s), which may have multiple packages, you might accidentally pull a package from the "wrong" location
- modelling Python has been difficult, taking a venv approach is what is now being done (as `pip` also does)

- Loris is leaving for Microsoft Quantum so someone will else will take over
- We will keep up with things in EESSI and then try to figure out next steps
- Loris may still attend the hackathon

---

## 20260113

attendees:
- EESSI: Loris, Kenneth
- Spack: Todd, Massimiliano (Max)
- JSC: Jayesh
- Kitware: Ryan Krattiger
- working on CI/CD for Spack, build cache, started HPSF Binaries working group
- Kickoff meeting for HPSF Binaries working group Thursday 15 January 2026 at 17:00 CET: https://zoom-lfx.platform.linuxfoundation.org/meeting/97701173216?password=a96bdd87-ca48-4d65-87a9-664d54b23cce
- Ryan also works on visualiation stuff, OpenGL

### Notes

#### Loris' recap from last time

- I showed how to expose EESSI's packages to Spack by creating a custom *upstream* database.
Each node (spec) correspond to a package, located in EESSI's file system (software layer) and is connected to its (build/link/runtime) dependency nodes. Example `cmake` entry in Spack DB for EESSI:

    {'name': 'cmake',
     'version': '3.31.8',
     'arch': {'platform': 'linux', 'platform_os': 'ubuntu24.04',
              'target': {'name': 'haswell', 'vendor': 'GenuineIntel',
                         'features': ['abm', 'aes', 'avx', 'avx2', ...], 'generation': 0,
                         'parents': ['ivybridge', 'x86_64_v3'], 'cpupart': ''}},
     'namespace': 'builtin',
     'parameters': {'build_system': 'generic',
      'build_type': 'Release',
      'doc': False,
      'ncurses': True,
      'ownlibs': True,
      'qtgui': False,
      'cflags': [],
      'cppflags': [],
      'cxxflags': [],
      'fflags': [],
      'ldflags': [],
      'ldlibs': []},
     'package_hash': 'mjpoqjqeinfi67c4coeo6dpj7as6nepv73yjju2uaep2kdmblq7a====',
     'dependencies': [
         {'name': 'bzip2',
          'hash': 'n2dm6r5dlkkc4bx2owtij5s5hf4ai7e7',
          'parameters': {'deptypes': ('link',), 'virtuals': ()}},
         {'name': 'curl',
          'hash': 'kcrzqtj2samxn2hyvdrrlkdnwqvnn55v',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'gcc',
          'hash': 'f4hdzzhaarpa7qoflhswl4rvuhrj5d6d',
          'parameters': {'deptypes': ('build',), 'virtuals': ('c', 'cxx')}},
         {'name': 'gcc-runtime',
          'hash': 'qzvkemkvff7xpkoecxtxmcnzdukdcp2v',
          'parameters': {'deptypes': ('link',), 'virtuals': ()}},
         {'name': 'glibc',
          'hash': 'hrhksjeecpgpidarbd6pk3q3pyvbpkch',
          'parameters': {'deptypes': ('link',), 'virtuals': ('libc',)}},
         {'name': 'libarchive',
          'hash': '35lndizpqsho2u6wev4nmcz76rsxac5g',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'ncurses',
          'hash': 'ooxt42xzzme3j52vmmr74ovy5k7ufskz',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'openssl',
          'hash': '2uuajfgiyekdhbekqzgfw7tckhg6s2g2',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'zlib',
          'hash': '4dmhrplpqqpvqu7nbzh2wg5en7n3t5kp',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}}],
       'annotations': {'original_specfile_version': 5}}

- EESSI OS (compat layer) packages can also be detected by Spack and reused (e.g. `zlib`, `tar`, `xz`, `openssl`, `sed`, etc.)
- We manage to build new packages (e.g. `quantum-espresso`) reusing dependencies from EESSI.
- Problem: compiler can't find `stdlib.h` header files when building Kokkos or GROMACS with Spack
- bug reported to Spack on this: https://github.com/spack/spack/issues/51582
- workaround done locally by Loris via some hardcoding

#### New developments

- Problem fixed: Spack compiler wrapper does not support non-standard sysroot, see [#51582 Compiler with non-standard sysroot is not handled correctly by the compiler wrapper](https://github.com/spack/spack/issues/51582)

- Max's recent PR [#51118 externals: Model as concrete specs and add dependency definitions](https://github.com/spack/spack/pull/51118) makes **external packages** to be treated as all the other packages, and gives one the ability to specify (external) dependencies (see docs: https://spack.readthedocs.io/en/latest/packages_yaml.html#specifying-dependencies-among-external-packages).

We can leverage this new **native** functionality to **make Spack aware of EESSI's builds**:
- no need to generate an upstream database with custom code
- avoid to reinvent the wheel, leverage the powerful spec/dependency parsing capabilities provided by the `ExternalSpecsParser` class. We "just" need to provide a proper YAML file.

**It works well !**

- `ExternalSpecsParser` takes care of handling some things well, produces useful errors
- idea would be to auto-generate the YAML file (per EESSI version & CPU target)
- other YAML files can be included
- specs need to be unique
- Spack doesn't have functionality like this yet, would be nice if this could be reused
- for EESSI, we can make a lot of assumptions about structure
- `spack external find` would be useful
- packages can include some logic that help with this
- something more generic would have to be based on RPATH sections (or `ldd`, `libtree`) + interpreter used by ELF binaries
- fast ELF reader/writer (implemented in Python) available in Spack already, used to avoid having to use `patchelf` for re-locations
- can't only use `patchelf` for re-locations, also have to patch paths in other places

- Spack will use `skylake` (non-AVX-512) when building stuff on Loris' laptop, on top of `haswell` installations provided by EESSI
- could that cause trouble?

- **Importance of explicitly declaring dependencies to Spack**
- Not always clear. In EESSI, dependencies are RPATH-ed...
- Can it avoid conflicts in some corner cases? e.g. I'm building `lammps` that depends on `openblas`, and one of LAMMPS's dependencies provided by EESSI also depends on `openblas` (this is supposedly declared in the `packages.yaml`). We want to make sure that the same `openblas` is used.
- For example, I tried to build `quantum-espresso~mpi` in these 3 cases, and the resulting builds were the same.
1. dependencies of its dependencies (cmake, curl), specified, including compat-layer ones
2. dependencies of its dependencies (cmake, curl), specified, excluding compat-layer ones
3. dependencies of its dependencies not specified
- What would be a good example to test where this might not be the case?
- Max: won't matter much in case of pure build dependencies (like `cmake`), since you're just calling a binary
- Max: more important for run/link dependencies
- like OpenBLAS/OpenMPI
- could lead to conflicts in dependencies of those, could end up with two versions of same library used by a binary, resulting in load race
- OK in some cases, Spack allows multiple versions of `libstdc++` as long as most recent version is guaranteed to be used at runtime
- question comes up because making sure all dependencies as Spack knows them are actually declared for EESSI installations
- relevant: CPS - https://cps-org.github.io/cps/
- improved version of pkgconfig file (`.pc`)

- Wrote a code to **convert external packages to an upstream database**:
- takes a `packages.yaml`-style file that contains external packages and generates an (upstream) database that Spack can use directly. Advantages/disadvantages ??
- use a database json instead of a yaml configuration file?
- avoids the need to parse `packages.yaml` every time - although [PR #51653 reuse.py: avoid _create_external_parser](https://github.com/spack/spack/pull/51653) already improved the performance
- `spack find` will show all the available packages, don't need the `--show-configured-externals` flag
- (optional) it adds `gcc-runtime` and `glibc` dependencies to packages built with a compiler
- Max hasn't really tried this with externals
- `gcc-runtime` is good to have for compatibility, but is not supposed be in external packages, needs to be looked into
- could matter when re-locating stuff built with Spack and not use GCC runtime provided by EESSI ?
- (optional) it can detect and add OS (compat-layer) packages as well
- compilers need to be declared in `packages.yaml`. If no compilers are there, Spack will try to detect and add one the first time.
- Todd: `packages.yaml` is the recommended approach (vs Spack database)
- meant to be used for external consumption (for both human & machines)
- can be seen as a layer on top of the Spack DB
- performance-wise:
- JSON (Spack DB) better than YAML (`packages.yaml`)
- Loris initially notices a performance impact, but less severe now than to improvements to YAML parser
- also supporting `packages.json` by Spack could be considered
- Next:
- try more difficult cases (MPI)
- script to generate `packages.yaml` from metadata provided by EESSI

#### More notes

- blog post on initial PoC: https://www.eessi.io/docs/blog/2026/01/09/Spack-on-top-of-EESSI-PoC/
- with old approach using external Spack database
- Todd: Python externals (like `setuptools`, `numpy`, etc.)
- plan was to deprecate auto-wiring of Python runtime to its dependents
- but that would actually still be useful...
- people who use Spack on top of EESSI may want to package stuff they built & relocate those installations
- HPSF Binary working group
- interest from EasyBuild & EESSI to become an HPSF project
- GPU aspect would be the hard part, cfr. GTL aspect of Cray systems
- follow-up blog post by Loris on work done with external packages

#### Next meeting

- Tue 24 Feb'26 17:30-18:30 CET

---

## 20251112

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (HPC-UGent)
- Jayesh Badwaik (JSC)
- Jan André Reuter (JSC)
- Massimiliano Culpo
- Todd Gamblin (LLNL)
- Loris Ercole (CECAM)

### Notes

- update by Loris
- building & installing QuantumESPRESSO on top of EESSI
- `ldd pw.x` shows that everything is provided by EESSI
- all dependencies provided by EESSI via external database
- process of creating Spack database not fully automated yet, based on manually constructed Python dict
- quite a bit of fiddling to convince Spack to actually use the software installations listed in Spack DB
- what type of problems exactly?
- making sure required variants are there
- link dep on glibc, build dep on GCC
- GCC + glibc are specified as external packages via `packages.yml`
- how relevant are the dependencies specified in Python dict?
- seems hard to get all the details right there
- a bit unclear why specifying those dependencies is really required?
- is it relevant w.r.t. convincing Spack to reuse these installations?
- may be relevant w.r.t. meeting requirements imposed by Spack packages
- => it's generally helpful to provide this information
- auto-detection of dependencies implemented based on `ldd` would be interesting to Spack as well (less so if it would be based on environment modules)
- (Jayesh) basing this on `ldd` output may lead to incorrect results
- (Alan) may apply to stuff coming from EESSI compatibility layer
- also played with Massimiliano's recently merged PR w.r.t. dependencies (?)
- Massimiliano: glibc as external should be redundant, since it's derived from compiler
- Loris: had to change Spack to point it to the correct path of the dynamic linker (which is in a weird place in EESSI)
- Massimiliano: also gcc-runtime should not be an external
- would this be a right path forward vs creating our own Spack DB?
- open problem:
- compiler can't find `stdlib.h` header files (when building Kokkos or GROMACS with Spack)
- unclear why that didn't happen with QuantumESPRESSO (which is mostly Fortran)
- Spack compiler wrappers unset some related environment variables (like `$CPATH`)
- Q&A
- (Jayesh) we could gather extra (meta)data during installation to make creating Spack DB easier
- ties back to idea that Kenneth & Alan had to collect more metadata during installation to enrich information that can be used to create overview of software installations available in EESSI
- (Loris) figure out hashes for installations in Spack DB/externals?
- A:`spack find --show-configured-external`
- (Loris) will Spack use `cmake` from EESSI when there's extra dependencies listed (like `openssl`)?
- (Todd,Max) yes
- MPI in EESSI:
- currently only OpenMPI is used in EESSI
- via host_injections mechanism, custom MPI library can be provided
- in a very specific path, which is controlled by sysadmin
- how would that work when Spack is used?
- Spack compiler wrapper would have to be enhanced to make sure that right MPI library gets picked up
- compiler wrapper picks up on `$LDFLAGS`, can be used to inject paths into RPATH section
- maybe this can be specified in a Spack toolchain (would be unconditional)
- Spack toolchain is basically an alias for compilers (new in Spack 1.0)
- would be nice to only do this when MPI is in play
- next steps
- figure out reason why `stdlib.h` is not found when building GROMACS/Kokkos
- due to resetting of environment variables done in Spack compiler wrapper?
- why isn't GCC compiler in EESSI able to find it's own header files except through `$CPATH` (or similar)?
- (Massimiliano) spack debug build, see `*.in, `*`.out` files (?)
- test QuantumESPRESSO more extensively, like MPI
- script to fully automate populating of Spack DB with given list of software installations/modules from EESSI?
- initial blog post on using Spack on top of EESSI
- talk proposal for Pkg Mgmt devroom at FOSDEM'26?
- deadline Mon 1 Dec 2025!
- https://blog.ecosyste.ms/2025/11/06/fosdem-2026-package-managers-devroom-cfp.html
- co-written by Loris/Todd/Kenneth/Max?
- next meeting
- Tue 16 Dec 17:30 CET

## 20251014

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (CECAM)
- Loris Ercole (CECAM)
- Jayesh Badwaik (JSC)
- Todd Gamblin (LLNL)

### Notes

- work done by Loris with Spack on top of EESSI
- started from PoC script that Todd provided
- build up Spack database with installations available in EESSI
- idea is to start from a dict/YAML file
- convert that into Spack DB
- GCC compiler as starting point
- Todd: no need to have `gcc-runtime` entry in `specs` list
- it's imposed by the compiler
- packages can have a different name (`gmake` in Spack vs `make` in EB)
- `_mark_concrete` vs `_finalize_concretization`
- `spack find` shows the info injected in Spack DB
- compiler is only picked up when dummy entry is defined as external packages in `packages.yml`
- both `gcc` and `gcc-runtime` need to be listed
- for `gmake`, `gcc` build dependency should also be there (+ C version on that edge)
- module registered for external packages is loaded in build environment, and also when `spack env activate` is used
- some environment variables set by modules like `$LIBRARY_PATH` may cause problems in Spack build env
- like CMake treating something like implicit when it's not
- some mapping between names would need to be done
- heuristic to map EB name to Spack name
- lowercase, prefix with `py-` for stuff like `numpy`
- mapping is worth it when building a package build takes a long time
- auto-generate list of specs for software installations provided by EESSI
- do we need to list all run/build deps for every package?
- maybe not as long as modules which are being loaded also load the dependencies
- variants
- like `+ncurses` in `cmake`
- really only matters if some stuff depends on package with that specific variant
- check with `spack pkg grep depends_on | grep cmake`
- Spack has logic to detect variants for an existing installation
- makes sense to rely on that
- start without variants initially
- important ones are
- `cuda`, `cudaarch`
- Boost has a lot of variants
- easy to detect: `static`/`shared`
- [Spack PR #51118](https://github.com/spack/spack/pull/51118) to treat externals as concrete specs with deps
- could eventually lead to only requiring a `packages.yml` to make Spack aware of installations provided by EESSI
- EasyBuild has all the information during the installation
- Spack DB could be updated as part of the installation procedure
- different installations are done in parallel
- Spack DB would need to be updated at ingest time
- DB & specs should have version metadata
- Jayesh working on discovering & parsing of modules created with EasyBuild for different sites
- different module naming schemes
- software version can be determined from the contents of the module file
- determine dependencies from easyconfig
- runtime + build
- should also take filter-deps into account
- can filter list of deps through module file to get more accurate information
- correct information is available during EasyBuild session, so easy to get the right information through an EasyBuild hook
- follow-up meeting: Wed 12 Nov'25 17:30 CET

---

## 20250923

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (CECAM)
- Loris Ercole (CECAM)
- Jayesh Badwaik (JSC)
- Todd Gamblin (LLNL)

### Notes

- not via external packages, but create Spack DB + let Spack use it as an upstream
- cfr. https://gitlab.com/eessi/support/-/issues/170
- Max is working on improved support for externals, see https://github.com/spack/spack/issues/49697
- https://spack.readthedocs.io/en/latest/chain.html
- separate Spack DB per CPU target, maybe also by toolchain?
- initial minimal example
- cfr. Jayesh' issue @ https://github.com/ebpkg/ebpkg/issues/4
- some issues sometimes due to RPATH
- GCC/OpenMPI/CMake coming from EasyBuild or EESSI
- get Spack to pick up on those
- will Spack load external modules at runtime as well?
- build dependencies should *not* be loaded
- even more minimal: let Spack use compiler from EESSI
- build bzip2/zlib with Spack using compiler coming from EasyBuild/EESSI
- `gcc` will inject `gcc-runtime` and `glibc` nodes in DAG
- `gcc` node should have `c`/`cxx` virtuals
- `zlib` only needs compiler + `gmake`
- container to mount EESSI
- see https://www.eessi.io/docs/getting_access/eessi_container/
- Apptainer image that includes CernVM-FS, to get easy access to EESSI
- EESSI in GitHub Actions
- https://github.com/EESSI/eessi-demo/blob/main/.github/workflows/software_repo_native.yml
- gcc-runtime nodes for where libstdc++ lives
- helps solver to unify link deps + tell Spack to RPATH it
- same version as GCC (?, to check)
- mapping of EasyBuild to Spack names
- need to get variants rights
- `zlib` should specify it has `zlib` as a variant
- EESSI compatibilty layer
- which loader will be used for stuff built with Spack using compiler from EESSI
- should be loader from EESSI compat layer since EESSI's GCC was configured with sysroot
- validation of Spack DB?
- node name, correctness of entries, ...
- JSON schema available (see `database_index.py` in `src/spack/lib/spack/schema`)
- interest from CINECA in using Spack on top of EESSI
- CINECA will make EESSI available on their systems through EuroHPC Federation Platform
- would CINECA be interested in working on this?
- maintenance burden
- when stuff is added to EESSI
- when new packages are added to Spack
- next sync meeting
- Tue 14 Oct 2025 17:30 CEST