# Discuss portability of ReFrame tests (20240131)

attendees:
- Caspar van Leeuwen (SURF, MultiXscale/EESSI)
- Satish Kamath (SURF, MultiXscale/EESSI)
- Lara Peeters (HPC-UGent, MultiXscale/EESSI)
- Kenneth Hoste (HPC-UGent, MultiXscale/EESSI)
- Sam Moors (Vrije Universiteit Brussel)
- Vasileios Karakasis (NVIDIA, ReFrame)
- Victor Holanda Rusu (CSCS, ReFrame)
- Eirini Koutsaniti (CSCS, ReFrame)
- Theofilos Manitaras (CSCS, ReFrame)

## Notes

- main goal of EESSI test suite (https://github.com/EESSI/test-suite): tests that are *portable* across systems
- initial focus is on portable *functionality* tests, but we also want to look into portable *performance* tests
- proof-of-concept test with GROMACS, on top of hpctestlib
- only need to modify ReFrame configuration file
- Vasileios: how are performance references for tests set?
- we have some ideas there, would like to get input
- maybe some support for this can be upstreamed in ReFrame
- Vasileios: we should see if there are common challenges across EESSI/CSCS/NVIDIA
- Victor: what type of tests are you focusing on? application tests that can be run by unpriviledged users?
- Vasileios: could use variables (which can be set on CLI via `-S` option) to expose "knobs" for tests so they can be used as performance benchmarks
- Victor: hpctestlib hasn't been touched much recently partially out of concern for breaking/changing tests that are implemented on top of it
- Vasileios: similar issue @ NVIDIA with the containers being used to run tests w.r.t. assumptions that are made on how software being tested was installed
- on portable performance checks
- NVIDIA has an internal tool to collect table of perf numbers from ReFrame perf log
- ideas
- let framework figure out perf ref (guess)
- compare with previous runs
- set performance reference manually (because you know what it should be somehow)
- where should performance reference data be stored
- in test (hardcoded), in some external database
- and when should it be updated (on demand, when tests are run, etc.)
- there should be an easy way to set/update performance reference data for a specific test
- SQLite should definitely be an option as database to store performance references
- NVIDIA is running tests through pyxis (https://github.com/NVIDIA/pyxis)
- interest in fleshing out hpctestlib from ReFrame
- what should focus be?
- collection of tests implemented in ReFrame that can be used by others
- much less something like hpctestlib (which gives power users a base to build on top of)

## Discussion topics

- CPU autodetect
- hpctestlib vs common tests

## Action items

- review GROMACS test in EESSI test suite + the one in hpctestlib (Vasileios)
- try running GROMACS test in EESSI test suite based on documentation
- https://www.eessi.io/docs/test-suite
- create repo for ReFrame test suite (Vasileios)
- Victor comes up with a catchy name first
- protected branches
- ~~set up dedicated channel in ReFrame Slack (Vasileios)~~