---
# System prepended metadata

title: Discuss portability of ReFrame tests (20240131)

---

# Discuss portability of ReFrame tests (20240131)

attendees:
- Caspar van Leeuwen (SURF, MultiXscale/EESSI)
- Satish Kamath (SURF, MultiXscale/EESSI)
- Lara Peeters (HPC-UGent, MultiXscale/EESSI)
- Kenneth Hoste (HPC-UGent, MultiXscale/EESSI)
- Sam Moors (Vrije Universiteit Brussel)
- Vasileios Karakasis (NVIDIA, ReFrame)
- Victor Holanda Rusu (CSCS, ReFrame)
- Eirini Koutsaniti (CSCS, ReFrame)
- Theofilos Manitaras (CSCS, ReFrame)

## Notes

- main goal of EESSI test suite (https://github.com/EESSI/test-suite): tests that are *portable* across systems
    - initial focus is on portable *functionality* tests, but we also want to look into portable *performance* tests
- proof-of-concept test with GROMACS, on top of hpctestlib
    - only need to modify ReFrame configuration file
- Vasileios: how are performance references for tests set?
    - we have some ideas there, would like to get input
    - maybe some support for this can be upstreamed in ReFrame
- Vasileios: we should see if there are common challenges across EESSI/CSCS/NVIDIA
- Victor: what type of tests are you focusing on? application tests that can be run by unpriviledged users?
- Vasileios: could use variables (which can be set on CLI via `-S` option) to expose "knobs" for tests so they can be used as performance benchmarks
- Victor: hpctestlib hasn't been touched much recently partially out of concern for breaking/changing tests that are implemented on top of it
- Vasileios: similar issue @ NVIDIA with the containers being used to run tests w.r.t. assumptions that are made on how software being tested was installed
- on portable performance checks
    - NVIDIA has an internal tool to collect table of perf numbers from ReFrame perf log
    - ideas
        - let framework figure out perf ref (guess)
        - compare with previous runs
        - set performance reference manually (because you know what it should be somehow)
    - where should performance reference data be stored
        - in test (hardcoded), in some external database
        - and when should it be updated (on demand, when tests are run, etc.)
        - there should be an easy way to set/update performance reference data for a specific test
        - SQLite should definitely be an option as database to store performance references
- NVIDIA is running tests through pyxis (https://github.com/NVIDIA/pyxis)
- interest in fleshing out hpctestlib from ReFrame
    - what should focus be?
        - collection of tests implemented in ReFrame that can be used by others
        - much less something like hpctestlib (which gives power users a base to build on top of)


## Discussion topics

- CPU autodetect
- hpctestlib vs common tests

## Action items

- review GROMACS test in EESSI test suite + the one in hpctestlib (Vasileios)
    - try running GROMACS test in EESSI test suite based on documentation
    - https://www.eessi.io/docs/test-suite
- create repo for ReFrame test suite (Vasileios)
    - Victor comes up with a catchy name first
    - protected branches
- ~~set up dedicated channel in ReFrame Slack (Vasileios)~~