# EESSI + Spack

## 20260113

attendees:
- EESSI: Loris, Kenneth
- Spack: Todd, Massimiliano (Max)
- JSC: Jayesh
- Kitware: Ryan Krattiger
- working on CI/CD for Spack, build cache, started HPSF Binaries working group
- Kickoff meeting for HPSF Binaries working group Thursday 15 January 2026 at 17:00 CET: https://zoom-lfx.platform.linuxfoundation.org/meeting/97701173216?password=a96bdd87-ca48-4d65-87a9-664d54b23cce
- Ryan also works on visualiation stuff, OpenGL

### Notes

#### Loris' recap from last time

- I showed how to expose EESSI's packages to Spack by creating a custom *upstream* database.
Each node (spec) correspond to a package, located in EESSI's file system (software layer) and is connected to its (build/link/runtime) dependency nodes. Example `cmake` entry in Spack DB for EESSI:

    {'name': 'cmake',
     'version': '3.31.8',
     'arch': {'platform': 'linux', 'platform_os': 'ubuntu24.04',
              'target': {'name': 'haswell', 'vendor': 'GenuineIntel',
                         'features': ['abm', 'aes', 'avx', 'avx2', ...], 'generation': 0,
                         'parents': ['ivybridge', 'x86_64_v3'], 'cpupart': ''}},
     'namespace': 'builtin',
     'parameters': {'build_system': 'generic',
      'build_type': 'Release',
      'doc': False,
      'ncurses': True,
      'ownlibs': True,
      'qtgui': False,
      'cflags': [],
      'cppflags': [],
      'cxxflags': [],
      'fflags': [],
      'ldflags': [],
      'ldlibs': []},
     'package_hash': 'mjpoqjqeinfi67c4coeo6dpj7as6nepv73yjju2uaep2kdmblq7a====',
     'dependencies': [
         {'name': 'bzip2',
          'hash': 'n2dm6r5dlkkc4bx2owtij5s5hf4ai7e7',
          'parameters': {'deptypes': ('link',), 'virtuals': ()}},
         {'name': 'curl',
          'hash': 'kcrzqtj2samxn2hyvdrrlkdnwqvnn55v',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'gcc',
          'hash': 'f4hdzzhaarpa7qoflhswl4rvuhrj5d6d',
          'parameters': {'deptypes': ('build',), 'virtuals': ('c', 'cxx')}},
         {'name': 'gcc-runtime',
          'hash': 'qzvkemkvff7xpkoecxtxmcnzdukdcp2v',
          'parameters': {'deptypes': ('link',), 'virtuals': ()}},
         {'name': 'glibc',
          'hash': 'hrhksjeecpgpidarbd6pk3q3pyvbpkch',
          'parameters': {'deptypes': ('link',), 'virtuals': ('libc',)}},
         {'name': 'libarchive',
          'hash': '35lndizpqsho2u6wev4nmcz76rsxac5g',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'ncurses',
          'hash': 'ooxt42xzzme3j52vmmr74ovy5k7ufskz',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'openssl',
          'hash': '2uuajfgiyekdhbekqzgfw7tckhg6s2g2',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}},
         {'name': 'zlib',
          'hash': '4dmhrplpqqpvqu7nbzh2wg5en7n3t5kp',
          'parameters': {'deptypes': ('build', 'link'), 'virtuals': ()}}],
       'annotations': {'original_specfile_version': 5}}

- EESSI OS (compat layer) packages can also be detected by Spack and reused (e.g. `zlib`, `tar`, `xz`, `openssl`, `sed`, etc.)
- We manage to build new packages (e.g. `quantum-espresso`) reusing dependencies from EESSI.
- Problem: compiler can't find `stdlib.h` header files when building Kokkos or GROMACS with Spack
- bug reported to Spack on this: https://github.com/spack/spack/issues/51582
- workaround done locally by Loris via some hardcoding

#### New developments

- Problem fixed: Spack compiler wrapper does not support non-standard sysroot, see [#51582 Compiler with non-standard sysroot is not handled correctly by the compiler wrapper](https://github.com/spack/spack/issues/51582)

- Max's recent PR [#51118 externals: Model as concrete specs and add dependency definitions](https://github.com/spack/spack/pull/51118) makes **external packages** to be treated as all the other packages, and gives one the ability to specify (external) dependencies (see docs: https://spack.readthedocs.io/en/latest/packages_yaml.html#specifying-dependencies-among-external-packages).

We can leverage this new **native** functionality to **make Spack aware of EESSI's builds**:
- no need to generate an upstream database with custom code
- avoid to reinvent the wheel, leverage the powerful spec/dependency parsing capabilities provided by the `ExternalSpecsParser` class. We "just" need to provide a proper YAML file.

**It works well !**

- `ExternalSpecsParser` takes care of handling some things well, produces useful errors
- idea would be to auto-generate the YAML file (per EESSI version & CPU target)
- other YAML files can be included
- specs need to be unique
- Spack doesn't have functionality like this yet, would be nice if this could be reused
- for EESSI, we can make a lot of assumptions about structure
- `spack external find` would be useful
- packages can include some logic that help with this
- something more generic would have to be based on RPATH sections (or `ldd`, `libtree`) + interpreter used by ELF binaries
- fast ELF reader/writer (implemented in Python) available in Spack already, used to avoid having to use `patchelf` for re-locations
- can't only use `patchelf` for re-locations, also have to patch paths in other places

- Spack will use `skylake` (non-AVX-512) when building stuff on Loris' laptop, on top of `haswell` installations provided by EESSI
- could that cause trouble?

- **Importance of explicitly declaring dependencies to Spack**
- Not always clear. In EESSI, dependencies are RPATH-ed...
- Can it avoid conflicts in some corner cases? e.g. I'm building `lammps` that depends on `openblas`, and one of LAMMPS's dependencies provided by EESSI also depends on `openblas` (this is supposedly declared in the `packages.yaml`). We want to make sure that the same `openblas` is used.
- For example, I tried to build `quantum-espresso~mpi` in these 3 cases, and the resulting builds were the same.
1. dependencies of its dependencies (cmake, curl), specified, including compat-layer ones
2. dependencies of its dependencies (cmake, curl), specified, excluding compat-layer ones
3. dependencies of its dependencies not specified
- What would be a good example to test where this might not be the case?
- Max: won't matter much in case of pure build dependencies (like `cmake`), since you're just calling a binary
- Max: more important for run/link dependencies
- like OpenBLAS/OpenMPI
- could lead to conflicts in dependencies of those, could end up with two versions of same library used by a binary, resulting in load race
- OK in some cases, Spack allows multiple versions of `libstdc++` as long as most recent version is guaranteed to be used at runtime
- question comes up because making sure all dependencies as Spack knows them are actually declared for EESSI installations
- relevant: CPS - https://cps-org.github.io/cps/
- improved version of pkgconfig file (`.pc`)

- Wrote a code to **convert external packages to an upstream database**:
- takes a `packages.yaml`-style file that contains external packages and generates an (upstream) database that Spack can use directly. Advantages/disadvantages ??
- use a database json instead of a yaml configuration file?
- avoids the need to parse `packages.yaml` every time - although [PR #51653 reuse.py: avoid _create_external_parser](https://github.com/spack/spack/pull/51653) already improved the performance
- `spack find` will show all the available packages, don't need the `--show-configured-externals` flag
- (optional) it adds `gcc-runtime` and `glibc` dependencies to packages built with a compiler
- Max hasn't really tried this with externals
- `gcc-runtime` is good to have for compatibility, but is not supposed be in external packages, needs to be looked into
- could matter when re-locating stuff built with Spack and not use GCC runtime provided by EESSI ?
- (optional) it can detect and add OS (compat-layer) packages as well
- compilers need to be declared in `packages.yaml`. If no compilers are there, Spack will try to detect and add one the first time.
- Todd: `packages.yaml` is the recommended approach (vs Spack database)
- meant to be used for external consumption (for both human & machines)
- can be seen as a layer on top of the Spack DB
- performance-wise:
- JSON (Spack DB) better than YAML (`packages.yaml`)
- Loris initially notices a performance impact, but less severe now than to improvements to YAML parser
- also supporting `packages.json` by Spack could be considered
- Next:
- try more difficult cases (MPI)
- script to generate `packages.yaml` from metadata provided by EESSI

#### More notes

- blog post on initial PoC: https://www.eessi.io/docs/blog/2026/01/09/Spack-on-top-of-EESSI-PoC/
- with old approach using external Spack database
- Todd: Python externals (like `setuptools`, `numpy`, etc.)
- plan was to deprecate auto-wiring of Python runtime to its dependents
- but that would actually still be useful...
- people who use Spack on top of EESSI may want to package stuff they built & relocate those installations
- HPSF Binary working group
- interest from EasyBuild & EESSI to become an HPSF project
- GPU aspect would be the hard part, cfr. GTL aspect of Cray systems
- follow-up blog post by Loris on work done with external packages

#### Next meeting

- Tue 24 Feb'26 17:30-18:30 CET

---

## 20251112

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (HPC-UGent)
- Jayesh Badwaik (JSC)
- Jan André Reuter (JSC)
- Massimiliano Culpo
- Todd Gamblin (LLNL)
- Loris Ercole (CECAM)

### Notes

- update by Loris
- building & installing QuantumESPRESSO on top of EESSI
- `ldd pw.x` shows that everything is provided by EESSI
- all dependencies provided by EESSI via external database
- process of creating Spack database not fully automated yet, based on manually constructed Python dict
- quite a bit of fiddling to convince Spack to actually use the software installations listed in Spack DB
- what type of problems exactly?
- making sure required variants are there
- link dep on glibc, build dep on GCC
- GCC + glibc are specified as external packages via `packages.yml`
- how relevant are the dependencies specified in Python dict?
- seems hard to get all the details right there
- a bit unclear why specifying those dependencies is really required?
- is it relevant w.r.t. convincing Spack to reuse these installations?
- may be relevant w.r.t. meeting requirements imposed by Spack packages
- => it's generally helpful to provide this information
- auto-detection of dependencies implemented based on `ldd` would be interesting to Spack as well (less so if it would be based on environment modules)
- (Jayesh) basing this on `ldd` output may lead to incorrect results
- (Alan) may apply to stuff coming from EESSI compatibility layer
- also played with Massimiliano's recently merged PR w.r.t. dependencies (?)
- Massimiliano: glibc as external should be redundant, since it's derived from compiler
- Loris: had to change Spack to point it to the correct path of the dynamic linker (which is in a weird place in EESSI)
- Massimiliano: also gcc-runtime should not be an external
- would this be a right path forward vs creating our own Spack DB?
- open problem:
- compiler can't find `stdlib.h` header files (when building Kokkos or GROMACS with Spack)
- unclear why that didn't happen with QuantumESPRESSO (which is mostly Fortran)
- Spack compiler wrappers unset some related environment variables (like `$CPATH`)
- Q&A
- (Jayesh) we could gather extra (meta)data during installation to make creating Spack DB easier
- ties back to idea that Kenneth & Alan had to collect more metadata during installation to enrich information that can be used to create overview of software installations available in EESSI
- (Loris) figure out hashes for installations in Spack DB/externals?
- A:`spack find --show-configured-external`
- (Loris) will Spack use `cmake` from EESSI when there's extra dependencies listed (like `openssl`)?
- (Todd,Max) yes
- MPI in EESSI:
- currently only OpenMPI is used in EESSI
- via host_injections mechanism, custom MPI library can be provided
- in a very specific path, which is controlled by sysadmin
- how would that work when Spack is used?
- Spack compiler wrapper would have to be enhanced to make sure that right MPI library gets picked up
- compiler wrapper picks up on `$LDFLAGS`, can be used to inject paths into RPATH section
- maybe this can be specified in a Spack toolchain (would be unconditional)
- Spack toolchain is basically an alias for compilers (new in Spack 1.0)
- would be nice to only do this when MPI is in play
- next steps
- figure out reason why `stdlib.h` is not found when building GROMACS/Kokkos
- due to resetting of environment variables done in Spack compiler wrapper?
- why isn't GCC compiler in EESSI able to find it's own header files except through `$CPATH` (or similar)?
- (Massimiliano) spack debug build, see `*.in, `*`.out` files (?)
- test QuantumESPRESSO more extensively, like MPI
- script to fully automate populating of Spack DB with given list of software installations/modules from EESSI?
- initial blog post on using Spack on top of EESSI
- talk proposal for Pkg Mgmt devroom at FOSDEM'26?
- deadline Mon 1 Dec 2025!
- https://blog.ecosyste.ms/2025/11/06/fosdem-2026-package-managers-devroom-cfp.html
- co-written by Loris/Todd/Kenneth/Max?
- next meeting
- Tue 16 Dec 17:30 CET

## 20251014

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (CECAM)
- Loris Ercole (CECAM)
- Jayesh Badwaik (JSC)
- Todd Gamblin (LLNL)

### Notes

- work done by Loris with Spack on top of EESSI
- started from PoC script that Todd provided
- build up Spack database with installations available in EESSI
- idea is to start from a dict/YAML file
- convert that into Spack DB
- GCC compiler as starting point
- Todd: no need to have `gcc-runtime` entry in `specs` list
- it's imposed by the compiler
- packages can have a different name (`gmake` in Spack vs `make` in EB)
- `_mark_concrete` vs `_finalize_concretization`
- `spack find` shows the info injected in Spack DB
- compiler is only picked up when dummy entry is defined as external packages in `packages.yml`
- both `gcc` and `gcc-runtime` need to be listed
- for `gmake`, `gcc` build dependency should also be there (+ C version on that edge)
- module registered for external packages is loaded in build environment, and also when `spack env activate` is used
- some environment variables set by modules like `$LIBRARY_PATH` may cause problems in Spack build env
- like CMake treating something like implicit when it's not
- some mapping between names would need to be done
- heuristic to map EB name to Spack name
- lowercase, prefix with `py-` for stuff like `numpy`
- mapping is worth it when building a package build takes a long time
- auto-generate list of specs for software installations provided by EESSI
- do we need to list all run/build deps for every package?
- maybe not as long as modules which are being loaded also load the dependencies
- variants
- like `+ncurses` in `cmake`
- really only matters if some stuff depends on package with that specific variant
- check with `spack pkg grep depends_on | grep cmake`
- Spack has logic to detect variants for an existing installation
- makes sense to rely on that
- start without variants initially
- important ones are
- `cuda`, `cudaarch`
- Boost has a lot of variants
- easy to detect: `static`/`shared`
- [Spack PR #51118](https://github.com/spack/spack/pull/51118) to treat externals as concrete specs with deps
- could eventually lead to only requiring a `packages.yml` to make Spack aware of installations provided by EESSI
- EasyBuild has all the information during the installation
- Spack DB could be updated as part of the installation procedure
- different installations are done in parallel
- Spack DB would need to be updated at ingest time
- DB & specs should have version metadata
- Jayesh working on discovering & parsing of modules created with EasyBuild for different sites
- different module naming schemes
- software version can be determined from the contents of the module file
- determine dependencies from easyconfig
- runtime + build
- should also take filter-deps into account
- can filter list of deps through module file to get more accurate information
- correct information is available during EasyBuild session, so easy to get the right information through an EasyBuild hook
- follow-up meeting: Wed 12 Nov'25 17:30 CET

---

## 20250923

attendees:
- Kenneth Hoste (HPC-UGent)
- Alan O'Cais (CECAM)
- Loris Ercole (CECAM)
- Jayesh Badwaik (JSC)
- Todd Gamblin (LLNL)

### Notes

- not via external packages, but create Spack DB + let Spack use it as an upstream
- cfr. https://gitlab.com/eessi/support/-/issues/170
- Max is working on improved support for externals, see https://github.com/spack/spack/issues/49697
- https://spack.readthedocs.io/en/latest/chain.html
- separate Spack DB per CPU target, maybe also by toolchain?
- initial minimal example
- cfr. Jayesh' issue @ https://github.com/ebpkg/ebpkg/issues/4
- some issues sometimes due to RPATH
- GCC/OpenMPI/CMake coming from EasyBuild or EESSI
- get Spack to pick up on those
- will Spack load external modules at runtime as well?
- build dependencies should *not* be loaded
- even more minimal: let Spack use compiler from EESSI
- build bzip2/zlib with Spack using compiler coming from EasyBuild/EESSI
- `gcc` will inject `gcc-runtime` and `glibc` nodes in DAG
- `gcc` node should have `c`/`cxx` virtuals
- `zlib` only needs compiler + `gmake`
- container to mount EESSI
- see https://www.eessi.io/docs/getting_access/eessi_container/
- Apptainer image that includes CernVM-FS, to get easy access to EESSI
- EESSI in GitHub Actions
- https://github.com/EESSI/eessi-demo/blob/main/.github/workflows/software_repo_native.yml
- gcc-runtime nodes for where libstdc++ lives
- helps solver to unify link deps + tell Spack to RPATH it
- same version as GCC (?, to check)
- mapping of EasyBuild to Spack names
- need to get variants rights
- `zlib` should specify it has `zlib` as a variant
- EESSI compatibilty layer
- which loader will be used for stuff built with Spack using compiler from EESSI
- should be loader from EESSI compat layer since EESSI's GCC was configured with sysroot
- validation of Spack DB?
- node name, correctness of entries, ...
- JSON schema available (see `database_index.py` in `src/spack/lib/spack/schema`)
- interest from CINECA in using Spack on top of EESSI
- CINECA will make EESSI available on their systems through EuroHPC Federation Platform
- would CINECA be interested in working on this?
- maintenance burden
- when stuff is added to EESSI
- when new packages are added to Spack
- next sync meeting
- Tue 14 Oct 2025 17:30 CEST