owned this note
owned this note
Published
Linked with GitHub
# EasyBuild 5.0
Useful links:
* EUM'23 talk: https://easybuild.io/eum23/#easybuild5
* EUM'24 talk: https://users.ugent.be/~kehoste/eum24/003_eum24_easybuild5.pdf
* GitHub Project: https://github.com/orgs/easybuilders/projects/18
* Sync meeting notes: https://github.com/easybuilders/easybuild/wiki/EasyBuild-5.0-working-group
## Meetings
Every 1st Wed of the month at 14:00 CE(S)T, via Zoom
### Next monthly sync meetings
- Wed 1 May 2024, 14:00 CEST is public holiday => Wed 8 May 2024, 14:00 CEST
### Next sprint meetings (max. 30 min)
every Monday, 15:00 CE(S)T
- 13-17 May => ISC'24 week (so no sprint meeting)
- Wed 22 May 15:00 CEST
- Wed 29 May 15:00 CEST
---
## Meeting notes
### 2024-05-08 (14:00 CEST)
attending: Alex, Simon, Bart, Jasper, Mikael
#### Merged PRs
- Checksums:
- switch checksum default to sha256 ([framework PR #4523](https://github.com/easybuilders/easybuild-framework/pull/4523))
- deprecate old checksum options (incl. md5) ([framework PR #4526](https://github.com/easybuilders/easybuild-framework/pull/4526))
- RPATHs:
- always include rpath configuration setting in output of --show-config ([framework PR #4515](https://github.com/easybuilders/easybuild-framework/pull/4515))
- Reproducible tarballs:
- Fix stability of reproducible tarballs from git repos ([framework PR #4517](https://github.com/easybuilders/easybuild-framework/pull/4517))
- use generic chmod command to reset permissions of reproducible archives ([framework PR #4522](https://github.com/easybuilders/easybuild-framework/pull/4522))
- Easyblocks:
- use ERROR global from tools.config in clang easyblock ([easyblocks PR #3318](https://github.com/easybuilders/easybuild-easyblocks/pull/3318))
- use ERROR global from tools.config in TensorFlow easyblock ([easyblocks PR #3320](https://github.com/easybuilders/easybuild-easyblocks/pull/3320))
#### Open (active) PRs
- Log shell cmds to temp file which allow users to revisit them ([PR #4486](https://github.com/easybuilders/easybuild-framework/pull/4486))
- Alex: will review and merge it asap
- Prefer `$EBPYTHONPREFIXES` over `$PYTHONPATH` ([framework PR #4496](https://github.com/easybuilders/easybuild-framework/pull/4496))
- Mikael: EBPYTHONPREFIX is not understood by anything but EB, this makes it not a workable solution for software in SYSTEM (or Conda)
- Alex: we need something that can be understood by upstream Python, is there something softer than PYTHONPATH?
- Mikael: new logic with EBPYTHONPREFIX can be limited to PythonPackage easyblock (and its derivates), so moving from framework to easyblock
- Bart: many users load virtual environments before loading software modules. Our Python modules set PYTHONPATH to a sitecustomization.py that sets sys.path
- Mikael: complex to implement as opt-in as needs to consider all custom PYTHONPATH definitions in easyconfigs
- add support for alternate easyconfig parameters/templates/constants ([framework #4511](https://github.com/easybuilders/easybuild-framework/pull/4511))
- Jasper: unit test fails in CI, but not when run standalone locally. Somewhere templates stopped to be resolved.
- add 'archive' extra to optionally install easybuild-easyconfigs-archive ([PR #20435](https://github.com/easybuilders/easybuild-easyconfigs/pull/20435))
- Aled: ready to merge
- set CMAKE_INSTALL_LIBDIR to lib by default
- set CMake installation LIBDIR to lib by default in CMakeMake easyblock ([PR #3227](https://github.com/easybuilders/easybuild-easyblocks/pull/3227))
- remove CMAKE_INSTALL_LIBDIR settings from configopts and add test to enfore use of install_libdir instead ([PR #20487](https://github.com/easybuilders/easybuild-easyconfigs/pull/20487))
- Alex: ready to merge
- Standarize behaviour on CWD changes
- add new get_cwd function to tools.filetools to retrieve current working directory ([PR #4525](https://github.com/easybuilders/easybuild-framework/pull/4525))
- return to original working directory after executing a command in run_shell_cmd ([PR #4390](https://github.com/easybuilders/easybuild-framework/pull/4390))
- Alex: conbination of faulty storage and build on install dir makes `os.getcwd` to fail (storage in zombie state). Proposal to standarize behaviour of `run_shell_cmd` to switch to starting working directory at the end of execution
- Mikael: this changes behaviour and should not be necessary as any changes to working directory inside the shell executing the command should not affect the CWD of the main Python process. Will be troublesome to execute multiple commands that rely on directory changes by previous commands.
- Alex: I still hit the error with `os.getcwd` outside of the subprocess shell
- Replace run_cmd with run_shell_cmd in custom easyblock for QuantumESPRESSO ([PR #3269](https://github.com/easybuilders/easybuild-easyblocks/pull/3269))
- Simon: there is some on-going discussion about deciding which one of the two easyblocks for QuantumESPRESSO becomes the default one (aka `EB_QuantumESPRESSO`)
- update ConfigureMake easyblock to error out on unknown configure args ([PR #3025](https://github.com/easybuilders/easybuild-easyblocks/pull/3025))
- Companion PR adding deprecation warning/error: https://github.com/easybuilders/easybuild-easyblocks/pull/3026
- seems ready for merge
#### Discussion
- HPCNow interested in adding additional error types and exit codes to EasyBuild: https://github.com/easybuilders/easybuild-framework/issues/4426
- Jasper tagged fior EB5 the PR from Flamefire fixing compilation on AMD Zen with Intel compilers (https://github.com/easybuilders/easybuild-framework/pull/3797), as it is quite annoying to deal with those errors. Not a blocker but nice to have.
### 2024-04-29 sprint meeting (14:00 CEST)
attending: ...
#### Merged PRs
- Fix stability of reproducible tarballs from git repos ([framework PR #4517](https://github.com/easybuilders/easybuild-framework/pull/4517))
#### Open (active) PRs
- Log shell cmds to temp file which allow users to revisit them #4486
- ready to merge
- Prefer `$EBPYTHONPREFIXES` over `$PYTHONPATH` ([framework PR #4496](https://github.com/easybuilders/easybuild-framework/pull/4496))
- may need to move logic to pick EBPYTHONPREFIXES vs PYTHONPATH to `EasyBlock.make_module_req` (rather than in `ModuleGenerator.update_paths`)
- maybe need to logic in both `make_module_req` (default + make `PythonPackage` behave) + `ModuleGenerator` (to deal with `PYTHONPATH` used in `modextrapaths`)
- EB should produce a warning if it detects that `PYTHONPATH` is used when it shouldn't be
- Mikael will continue working on this
- add support for alternate easyconfig parameters/templates/constants ([framework #4511](https://github.com/easybuilders/easybuild-framework/pull/4511))
- always include rpath configuration setting in output of `--show-config` ([framework PR #4515](https://github.com/easybuilders/easybuild-framework/pull/4515))
- 5.0.x consideration: ruff format `filetools.py` ([PR #4521](https://github.com/easybuilders/easybuild-framework/pull/4521))
- not so much about which tool/linter is used, but rather whether the community thinks it's worth adopting an approach like this
- ruff can be heavily configured, we could configure it to match our current code style
- definitely impacts usefulness of "git blame"
- should we recommend that developers install a pre-commit hook to run `ruff` on their code changes?
- should we adopt a bot that automatically add a commit to fix code style changes?
- add 'archive' extra to optionally install easybuild-easyconfigs-archive ([PR #20435](https://github.com/easybuilders/easybuild-easyconfigs/pull/20435))
- need to introduce protected branch in archive repo => fixed
#### Discussion
- how to mitigate impact of collapsing `5.0.x` branches in `develop`
- will (often) break use of `--from-pr` with EasyBuild 4.x
- see also Simon's [framework PR #4520](https://github.com/easybuilders/easybuild-framework/pull/4520)
- include `easybuild_version = '5.0'` in easyconfig that require EasyBuild v5.0
```python
# requires EasyBuild 5.0 which enables use_pip by default
easybuild_version = '5.0'
```
- also will be useful when we start renaming easyconfig parameters
- error message should be improved
- motivation to rename branches in GitHub?
- `--from-pr` on merged PR currently pulls from `develop`
- only use `5.0.x` style branches going forward?
- how does `--new-pr` know which branch to target by default?
---
### 2024-04-19 sprint meeting (14:00 CEST)
attending:
#### Next EasyBuild release
- v4.9.1 released 2024-04-05 (14 days ago)
- next release will hopefully be 5.0.0 (famous last words)
#### Status of 5.0.x branches
- recently synced with `develop`
- framework: 2024-04-14 ([framework PR #4508](https://github.com/easybuilders/easybuild-framework/pull/4508))
- easyblocks: 2024-04-06 ([easyblocks PR #3287](https://github.com/easybuilders/easybuild-easyblocks/pull/3287))
- easyconfigs: 2024-04-06 ([easyconfigs PR #20314](https://github.com/easybuilders/easybuild-easyconfigs/pull/20314))
#### Merged PRs
**framework**
- implement support for running interactive commands with `run_shell_cmd` ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453))
- fix getting all available output for interactive commands + fix logging end of output when no match was found in `run_shell_cmd` ([PR #4504](https://github.com/easybuilders/easybuild-framework/pull/4504))
- replace spaces/line breaks in question/wait patterns with regex pattern that matches one or more spaces/line breaks in `run_shell_cmd` ([PR #4509](https://github.com/easybuilders/easybuild-framework/pull/4509))
- deprecate `run_cmd` and `run_cmd_qa` & co, move them to `easybuild._deprecated` module ([PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433))
**easyblocks**
- various PRs to migrate from `run_cmd_qa` to `run_shell_cmd`
- remove a bunch of unused easyblocks (WRF-Fire, HEALPix, pplacer, Modeller, Molpro)
- drop support for versions < 9.5 in custom easyblock for Geant4 ([PR #3296](https://github.com/easybuilders/easybuild-easyblocks/pull/3296))
- add warning if .Renviron detected during install of RPackage ([PR #3263](https://github.com/easybuilders/easybuild-easyblocks/pull/3263))
**easyconfigs**
- Remove dead dotNET codes ([PR #20259](https://github.com/easybuilders/easybuild-easyconfigs/pull/20259))
#### Open (active) PRs
**framework**
- add support for disabling strict RPATH sanity check + print a warning when mixing of non-RPATH and RPATH installations was detected ([PR #4475](https://github.com/easybuilders/easybuild-framework/pull/4475))
- add mechanism for easyconfig template deprecation ([PR #4485](https://github.com/easybuilders/easybuild-framework/pull/4485))
- stuck on broken tests
- Log shell cmds to temp file which allow users to revisit them ([PR #4486](https://github.com/easybuilders/easybuild-framework/pull/4486))
- Enable module-depends-on by default ([PR #4500](https://github.com/easybuilders/easybuild-framework/pull/4500))
- Make module-extensions true by default ([PR #4501](https://github.com/easybuilders/easybuild-framework/pull/4501))
- add support for alternate easyconfig parameters/templates/constants ([PR #4511](https://github.com/easybuilders/easybuild-framework/pull/4511))
- Prefer EBPYTHONPREFIX over PYTHONPATH #4496
**easyblocks**
- Replace `run_cmd` with `run_shell_cmd` in custom easyblock for QuantumESPRESSO (quantumespresso.py) ([PR #3269](https://github.com/easybuilders/easybuild-easyblocks/pull/3269))
#### TODO
**framework**
- go through open PRs/issues for EasyBuild 5.0: what do we really still need to do that is *required* for EasyBuild 5.0
**easyblocks**
- complete migration to `run_shell_cmd` (ABAQUS, GAMESS-US, QuantumESPRESSO, WIEN2k)
- go through open PRs/issues for EasyBuild 5.0: what do we really still need to do that is *required* for EasyBuild 5.0
#### Alternate plan for renaming easyconfig parameters & co
- not a "big bang" approach by deprecated `configopts` & co for EasyBuild 5.0
- first add support for new names, then gradually adopt them
- not necessarily in EasyBuild 5.0
- only deprecate when framework/easyblocks/easyconfigs have been migrated to new news (EasyBuild 5.x with x > 0)
- drop support for old names in EasyBuild 6.0 (in X years)
- need to take into account hooks using old/new names...
#### Notes
- weird issue with `libjpeg.so.8` not being found become Meson strips out part of RPATH section fixed
- see https://github.com/easybuilders/easybuild-easyconfigs/issues/16256 + https://github.com/easybuilders/easybuild-easyconfigs/pull/20386
---
### 2024-04-03 sprint meeting (14:00 CEST)
attending: Alex, Bart, Jasper, Simon, Kenneth
#### Next EasyBuild release (v4.9.1)
- being prepared
#### Status of 5.0.x branches
- unchanged since 2024-03-28 meeting, except for merging PRs to sync with `develop` branch
- framework: in sync with `develop` (since 202403011)
- easyblocks: [PR #3261](https://github.com/easybuilders/easybuild-easyblocks/pull/3261)
- easyconfigs: [PR #20236](https://github.com/easybuilders/easybuild-easyconfigs/pull/20236)
#### Merged PRs
##### framework
- hide `readelf` output in trace output in RPATH sanity check [PR #4491](https://github.com/easybuilders/easybuild-framework/pull/4491)
- switch from `ls` to `bash` in tests that are expecting this to be a binary [PR #4492](https://github.com/easybuilders/easybuild-framework/pull/4492)
##### easyblocks
- use `sysconfig.get_config_vars` from Python standard library in `PythonPackage` easyblock (instead of `distutils.sysconfig.get_config_vars`) [PR #3264](https://github.com/easybuilders/easybuild-easyblocks/pull/3264) (Python 3.12)
- also run easyblocks test suite with Python 3.12 [PR #3266](https://github.com/easybuilders/easybuild-easyblocks/pull/3266)
- remove unused easyblock for ncurses (`ncurses.py`) [PR #3265](https://github.com/easybuilders/easybuild-easyblocks/pull/3265)
##### easyconfigs
- cleanup Python < 2.6 test skip [PR #20253](https://github.com/easybuilders/easybuild-easyconfigs/pull/20253)
#### Open (active) PRs
##### framework
- deprecate `run_cmd` and `run_cmd_qa` & co, move them to `easybuild._deprecated` module (WIP) ([PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433))
- blocked by open `run_shell_cmd` [PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453)
- implement support for running interactive commands with `run_shell_cmd` (WIP) ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453))
- add support for disabling strict RPATH sanity check ([PR #4475](https://github.com/easybuilders/easybuild-framework/pull/4475))
- our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
- we can merge support for `--disable-strict-rpath-sanity-check`, and always mention it in error message for failing RPATH sanity check for now
- should also be controllable via easyconfig parameter (for OCaml for example)
- then later come up with a good way of detected mixed stack
- `ldd` check should *always* be run to check for missing libraries, regardless of using RPATH or not
- we could/should only care of `NEEDED` libraries (direct dependencies, via `readelf -d`) if `ldd` reports them as `not found` => no problem in mixed stack situation
- Rename easyconfig parameters ([WIP PR #4483](https://github.com/easybuilders/easybuild-framework/pull/4483))
- Log shell cmds to temp file which allow users to revisit them ([PR #4486](https://github.com/easybuilders/easybuild-framework/pull/4486))
- add mechanism for easyconfig template deprecation ([PR #4485](https://github.com/easybuilders/easybuild-framework/pull/4485))
##### easyblocks
- fix extension filter for Perl packages ([PR #2699](https://github.com/easybuilders/easybuild-easyblocks/pull/2699))
- set CMake installation `LIBDIR` to lib by default in `CMakeMake` easyblock ([PR #3227](https://github.com/easybuilders/easybuild-easyblocks/pull/3227))
- update `ConfigureMake` easyblock to error out on unknown configure args ([PR #3025](https://github.com/easybuilders/easybuild-easyblocks/pull/3025))
- use `run_shell_cmd` in custom easyblock for WRF [PR #3270](https://github.com/easybuilders/easybuild-easyblocks/pull/3270)
##### easyconfigs
- Replace `parallel` by `maxparallel` ([PR #19375](https://github.com/easybuilders/easybuild-easyconfigs/pull/19375))
- Remove whitespace at EOL from ECs ([PR #20082](https://github.com/easybuilders/easybuild-easyconfigs/pull/20082))
---
### 2024-03-28 sprint meeting (09:00 CET)
attending: Mikael, Jasper, Kenneth, Simon
#### Next EasyBuild release (v4.9.1)
- blocked by [framework PR #4468](https://github.com/easybuilders/easybuild-framework/pull/4468) to add support for `--from-commit` and `--include-easyblocks-from-commit`
#### Status of 5.0.x branches
- framework: in sync with `develop` (since 202403011)
- easyblocks: [PR #3261](https://github.com/easybuilders/easybuild-easyblocks/pull/3261)
- easyconfigs: [PR #20236](https://github.com/easybuilders/easybuild-easyconfigs/pull/20236)
#### Merged PRs
##### framework
- change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs ([PR #4248](https://github.com/easybuilders/easybuild-framework/pull/4248))
- Eliminate use of `distutils.version.StrictVersion`, use `LooseVersion` instead ([PR #4476](https://github.com/easybuilders/easybuild-framework/pull/4476))
- also run unit tests with Python 3.12 + add it to classifiers in `setup.py` ([PR #4484](https://github.com/easybuilders/easybuild-framework/pull/4484))
##### easyblocks
*(none)*
##### easyconfigs
*(none)*
#### Open (active) PRs
##### framework
- deprecate `run_cmd` and `run_cmd_qa` & co, move them to `easybuild._deprecated` module (WIP) ([PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433))
- blocked by open `run_shell_cmd` [PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453)
- implement support for running interactive commands with `run_shell_cmd` (WIP) ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453))
- add support for disabling strict RPATH sanity check ([PR #4475](https://github.com/easybuilders/easybuild-framework/pull/4475))
- our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
- we can merge support for `--disable-strict-rpath-sanity-check`, and always mention it in error message for failing RPATH sanity check for now
- should also be controllable via easyconfig parameter (for OCaml for example)
- then later come up with a good way of detected mixed stack
- Rename easyconfig parameters ([WIP PR #4483](https://github.com/easybuilders/easybuild-framework/pull/4483))
- Log shell cmds to temp file which allow users to revisit them ([PR #4486](https://github.com/easybuilders/easybuild-framework/pull/4486))
- add mechanism for easyconfig template deprecation ([PR #4485](https://github.com/easybuilders/easybuild-framework/pull/4485))
##### easyblocks
- fix extension filter for Perl packages ([PR #2699](https://github.com/easybuilders/easybuild-easyblocks/pull/2699))
- set CMake installation `LIBDIR` to lib by default in `CMakeMake` easyblock ([PR #3227](https://github.com/easybuilders/easybuild-easyblocks/pull/3227))
- update `ConfigureMake` easyblock to error out on unknown configure args ([PR #3025](https://github.com/easybuilders/easybuild-easyblocks/pull/3025))
##### easyconfigs
- Replace `parallel` by `maxparallel` ([PR #19375](https://github.com/easybuilders/easybuild-easyconfigs/pull/19375))
- Remove whitespace at EOL from ECs ([PR #20082](https://github.com/easybuilders/easybuild-easyconfigs/pull/20082))
---
### 2024-03-11 sprint meeting (15:00 CET)
attending: Kenneth, Mikael, Simon, Jasper, Alex, Bart, Adam
#### Next EasyBuild release
- working towards EasyBuild v4.9.1 release using `develop` branch, hopefully this week...
#### Status of `5.0.x` branches
- framework: sync with `develop` (20240311) [PR #4482](https://github.com/easybuilders/easybuild-framework/pull/4482)
- easyblocks: sync with `develop` (20240311) [PR #3251](https://github.com/easybuilders/easybuild-easyblocks/pull/3251)
- easyconfigs: sync with `develop` (20240311) [PR #20076](https://github.com/easybuilders/easybuild-easyconfigs/pull/20076)
#### Merged PRs
##### framework
- rename unclear `*run*` methods to `*install_extension*` + rename `install_extensions` to `install_all_extensions` ([PR #4400](https://github.com/easybuilders/easybuild-framework/pull/4400))
- replace `log.warn` with `log.warning` ([PR #4473](https://github.com/easybuilders/easybuild-framework/pull/4473))
- Eliminate `distutils.util.strtobool` ([PR #4477](https://github.com/easybuilders/easybuild-framework/pull/4477))
- fix easyconfig parameter deprecation ([PR #4479](https://github.com/easybuilders/easybuild-framework/pull/4479))
- Clean up parameter deprecation warnings, and update easybuild doc urls ([PR #4480](https://github.com/easybuilders/easybuild-framework/pull/4480))
##### easyblocks
- rename `run` method to `install_extension`, and likewise for `prerun` to `pre_install_extension`, `postrun` to `post_install_extension`, and `run_async` to `install_extension_async` ([PR #3064](https://github.com/easybuilders/easybuild-easyblocks/pull/3064))
##### easyconfigs
- migrate archive of easyconfigs to `easybuild-easyconfigs-archive` ([PR #20006](https://github.com/easybuilders/easybuild-easyconfigs/pull/20006))
- see also ([easybuild-easyconfigs-archive PR #2](https://github.com/easybuilders/easybuild-easyconfigs-archive/pull/2))
- `setup.py` of easybuild-easyconfigs will be enhanced to easily install archive (but it won't be by default)
#### Open PRs
##### framework
- change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs ([PR #4248](https://github.com/easybuilders/easybuild-framework/pull/4248))
- implement support for running interactive commands with `run_shell_cmd` (WIP) ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453))
- add support for disabling strict RPATH sanity check ([PR #4475](https://github.com/easybuilders/easybuild-framework/pull/4475))
- our idea to detect a "mixed stack" (mix of RPATH and non-RPATH installations) isn't really working out...
- Eliminate use of `distutils.version.StrictVersion` ([PR #4476](https://github.com/easybuilders/easybuild-framework/pull/4476))
- remove `distutils` use from `setup.py` ([PR #4478](https://github.com/easybuilders/easybuild-framework/pull/4478))
- Rename easyconfig parameters ([WIP PR #4483](https://github.com/easybuilders/easybuild-framework/pull/4483))
##### easyblocks
- fix extension filter for Perl packages ([PR #2699](https://github.com/easybuilders/easybuild-easyblocks/pull/2699))
- update `ConfigureMake` easyblock to error out on unknown configure args ([PR #3025](https://github.com/easybuilders/easybuild-easyblocks/pull/3025))
- set CMake installation `LIBDIR` to `lib` by default in `CMakeMake` easyblock ([PR #3227](https://github.com/easybuilders/easybuild-easyblocks/pull/3227))
##### easyconfigs
- Replace parallel by maxparallel ([PR #19375](https://github.com/easybuilders/easybuild-easyconfigs/pull/19375))
#### To discuss
- refactor easyconfig parameters and template constants to use a consistent naming style ([issue #4464](https://github.com/easybuilders/easybuild-framework/issues/4464))
- enabling RPATH linking by default
- RPATH sanity check fails for OCaml ([easyblocks issue #3247](https://github.com/easybuilders/easybuild-easyblocks/issues/3247))
- RPATH sanity check fails for LAMMPS when (not all) dependencies are installed with RPATH ([easyconfigs issue #20051](https://github.com/easybuilders/easybuild-easyconfigs/issues/20051))
#### TODO
---
### 2024-03-06 sync meeting (14:00 CET)
attending: Kenneth, Bart, Simon, Alex, Jasper, Mikael
#### Next EasyBuild release
- working to push out EasyBuild v4.9.1 using current `develop` in next couple of days...
#### Status of `5.0.x` branches
- `5.0.x` branches recently synced with `develop` ([framework PR #4469](https://github.com/easybuilders/easybuild-framework/pull/4469), [easyblocks PR #3238](https://github.com/easybuilders/easybuild-easyblocks/pull/3238), [easyconfigs PR #20004](https://github.com/easybuilders/easybuild-easyconfigs/pull/20004))
#### Merged PRs
- framework
- problematic `test_toy_exts_parallel` fixed ([PR #4471](https://github.com/easybuilders/easybuild-framework/pull/4471))
- broken `test_github_merge_pr` fixed ([PR #4470](https://github.com/easybuilders/easybuild-framework/pull/4470), also applied to `5.0.x` branch)
- fix bug (wrong working dir) in `run_shell_cmd` for running commands asynchronously ([PR #4471](https://github.com/easybuilders/easybuild-framework/pull/4471))
- replace `log.warn` with `log.warning` ([PR #4473](https://github.com/easybuilders/easybuild-framework/pull/4473))
- easyblocks
- update `RPackage` to make sure that asynchronous installation command is run in correct working directory ([PR #3242](https://github.com/easybuilders/easybuild-easyblocks/pull/3242))
- more PRs to migrate easyblocks to `run_shell_cmd`
- replace log.warn with log.warning ([PR #3245](https://github.com/easybuilders/easybuild-easyblocks/pull/3245))
- easyconfigs
- none
#### Open PRs
- framework
- change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs ([PR #4248](https://github.com/easybuilders/easybuild-framework/pull/4248)) [Alex]
- rename unclear `*run*` methods to `*install_extension*` + rename `install_extensions` to `install_all_extensions` ([PR #4400](https://github.com/easybuilders/easybuild-framework/pull/4400)) [Kenneth]
- implement support for running interactive commands with `run_shell_cmd` (WIP) ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453)) [Kenneth]
- deprecate `run_cmd` and `run_cmd_qa` & co, move them to `easybuild._deprecated` module (WIP) ([PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433)) [Kenneth]
- easyblocks
- rename `run` method to `install_extension`, and likewise for `prerun` to `pre_install_extension`, `postrun` to `post_install_extension`, and `run_async` to `install_extension_async` ([PR #3064](https://github.com/easybuilders/easybuild-easyblocks/pull/3064)) [Alex]
- set CMake installation `LIBDIR` to `lib` by default in `CMakeMake` easyblock ([PR #3227](https://github.com/easybuilders/easybuild-easyblocks/pull/3227)) [Simon]
- replace `run_cmd` with `run_shell_cmd` in custom easyblock for LAMMPS (`lammps.py`) ([PR #3244](https://github.com/easybuilders/easybuild-easyblocks/pull/3244)) [Kenneth]
- failing test report when not all dependencies were installed with RPATH... :-/
- easyconfigs
- migrate archive of easyconfigs to easybuild-easyconfigs-archive ([PR #20006](https://github.com/easybuilders/easybuild-easyconfigs/pull/20006))
- Migrate archive of easyconfigs from easybuild-easyconfigs ([easybuild-easyconfigs-archive PR #2](https://github.com/easybuilders/easybuild-easyconfigs-archive/pull/2))
#### To discuss
- Are we sticking to enabling RPATH by default? [Kenneth]
- installing with RPATH on top of dependencies not installed with RPATH may fail...
- cfr. Kenneth's recent experience with LAMMPS ([issue #20051](https://github.com/easybuilders/easybuild-easyconfigs/issues/20051))
```
== FAILED: Installation ended unsuccessfully: Sanity check failed: Library libhdf5_hl.so.310 not found for
/software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/bin/lmp
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/bin/lmp
Library libhdf5_hl.so.310 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib/liblammps.so.0
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib/liblammps.so.0
Library libhdf5_hl.so.310 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib64/liblammps.so.0
Library libbz2.so.1.0 not found for /software/LAMMPS/2Aug2023_update2-foss-2023a-kokkos/lib64/liblammps.so.0 (took 23 mins 13 secs)
```
- for a mixed software stack (installations with RPATH + without RPATH), we need a configuration option to loosen up the RPATH sanity check (to not unset `$LD_LIBRARY_PATH`)
- need to use `--disable-strict-rpath-sanity-check`
- can we produce a warning to hint towards need for `--disable-strict-rpath-sanity-check`?
- when we notice that a dependency was installed without RPATH (like netCDF, in case of LAMMPS)
- Kenneth will look into implementation, review by Simon
- stop relying on `distutils` (will no longer be available in Python 3.12) ([issue #3963](https://github.com/easybuilders/easybuild-framework/issues/3963)) [Bart]
- refactor easyconfig parameters and template constants to use a consistent naming style ([PR #4464](https://github.com/easybuilders/easybuild-framework/issues/4464)) [Jasper]
- `eb --modernize-easyconfigs foo.eb bar.eb`
- TODO: revise EasyBuild configuration options
- Alex: review issues tagged for 5.0 ([see here](https://github.com/easybuilders/easybuild-framework/issues?q=is%3Aissue+is%3Aopen+label%3AEasyBuild-5.0))
### 2024-02-29 sprint meeting (17:00 CET)
attending: Mikael, Simon, Jasper, Simon, Kenneth, Alex
#### Next EasyBuild release
- we're still weeks away from releasing EasyBuild 5.0
- should look into EasyBuild v4.9.1 release...
- and then collapse `5.0.x` into `develop` (and keep `4.9.x` as backup branch)
- this will affect people using `--from-pr` and `--include-easyblocks-from-pr`
#### Status of `5.0.x` branches
- sync PRs open:
- https://github.com/easybuilders/easybuild-framework/pull/4469
- ready to merge (?)
- https://github.com/easybuilders/easybuild-easyblocks/pull/3238
- ready to merge
- https://github.com/easybuilders/easybuild-easyconfigs/pull/20004
- ~40 failing tests, some cleanup is needed w.r.t. `use_pip = True`, etc. to make CI happy
#### Merged PRs
- docs:
- RPATH is enabled by default in EasyBuild v5.0 ([PR #241](https://github.com/easybuilders/easybuild-docs/pull/241))
- framework:
- add support for running shell commands asynchronously with `run_shell_cmd` ([PR #4444](https://github.com/easybuilders/easybuild-framework/pull/4444))
- unresolved bug, see intermittent failures of `test_toy_exts_parallel` in https://github.com/easybuilders/easybuild-framework/pull/4453
- working directory is not always correct for shell commands being run somehow?
- enable RPATH linking by default ([PR #4448](https://github.com/easybuilders/easybuild-framework/pull/4448))
- allow only alphanumeric characters in the output filename used in `run_shell_cmd` ([PR #4454](https://github.com/easybuilders/easybuild-framework/pull/4454))
- add cuda compute capability integer format templates ([PR #4463](https://github.com/easybuilders/easybuild-framework/pull/4463))
- Enhance download instructions by mentioning active source path ([PR #4459](https://github.com/easybuilders/easybuild-framework/pull/4459))
- easyblocks:
- update run_async methods in `RPackage`, `Rmpi`, and `Rserve` easyblocks to use submit `run_shell_cmd` call into thread pool ([PR #3163](https://github.com/easybuilders/easybuild-easyblocks/pull/3163))
- replace `run_cmd` with `run_shell_cmd` in custom easyblock for NWChem ([PR #3197](https://github.com/easybuilders/easybuild-easyblocks/pull/3197))
- Revert "remove unneeded custom easyblock for OCaml (ocaml.py)" ([PR #3237](https://github.com/easybuilders/easybuild-easyblocks/pull/3237))
- easyconfigs
- fix source definition for dialog ([PR #19906](https://github.com/easybuilders/easybuild-easyconfigs/pull/19906))
- archive old style `Kent_tools` easyconfig ([PR #19933](https://github.com/easybuilders/easybuild-easyconfigs/pull/19933))
- archive oldest `MATLAB` easyconfigs ([PR #19937](https://github.com/easybuilders/easybuild-easyconfigs/pull/19937))
- *lots* of PRs to add missing `download_instructions`
- detect easyconfigs where missing information about downloading sources ([PR #19881](https://github.com/easybuilders/easybuild-easyconfigs/pull/19881))
#### Open (active) PRs/issues
- archive (https://github.com/easybuilders/easybuild-easyconfigs-archive)
- add `README` [PR #1](https://github.com/easybuilders/easybuild-easyconfigs-archive/pull/1)
- support installation of archive via `pip install easybuild-easyconfigs[archive]` (or `pip install easybuild-easyconfigs-archive`)
- docs
- split out default config changes and remove proposed changes ([PR #242](https://github.com/easybuilders/easybuild-docs/pull/242))
- `download_instructions` enhancements in EB5 ([PR #243](https://github.com/easybuilders/easybuild-docs/pull/243))
- framework
- change `tar` command used in `get_source_tarball_from_git` to get reproducible tarballs ([PR #4248](https://github.com/easybuilders/easybuild-framework/pull/4248))
- rename unclear `*run*` methods to `*install_extension*` + rename `install_extensions` to `install_all_extensions` ([PR #4400](https://github.com/easybuilders/easybuild-framework/pull/4400))
- ready for final review + merge **[Kenneth]**
- implement support for running interactive commands with `run_shell_cmd` (WIP) ([PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453))
- rebased after merge of [PR #4444](https://github.com/easybuilders/easybuild-framework/pull/4444)
- still WIP...
- missing support for `qa_wait_patterns`
- more existing `run_cmd_qa` tests should also be implemented for `run_shell_cmd`
- to be tested with real easyblock
- deprecate `run_cmd` and `run_cmd_qa` & co, move them to `easybuild._deprecated` module (WIP) ([PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433))
- blocked by [PR #4453](https://github.com/easybuilders/easybuild-framework/pull/4453)
- refactor easyconfig parameters and template constants to use a consistent naming style ([issue #4464](https://github.com/easybuilders/easybuild-framework/issues/4464))
- can we reach consensus on which things we will rename, and what not?
- `dependencies` vs `deps`
- what about EasyBuild configuration options, do we need to rename something there too?
- does current deprecation mechanism take into account setting of easyconfig parameters in a hook? (see `EasyConfig.__setitem__`?)
- rules
- separate words with underscores (`versionsuffix` -> `version_suffix`)
- aim for consistency (`mod` + `module` -> all `module`)
- if a shorthand is used, no room for confusion (`deps`/`exts`/`doc`/`mod` is OK), less room for typos, must be at least 50% gain
- `mod_`/`module_` -> `env_mod_` (no room for confusion with other types of modules)
- `modulename` -> `import_name` (see dedicated issue)
- `options = {'modulename': 'torch'}` => `import_name = 'torch'` (avoid "module")
- should also look at easyblock-specific easyconfig parameters
- script or `eb` option to "modernize" easyconfigs (rename parameters/templates/constants)
- Always set `$EBPYTHONPREFIXES` instead of `$PYTHONPATH` ([issue #2887](https://github.com/easybuilders/easybuild-easyblocks/issues/2887))
- counter argument: `$PYTHONPATH` is ubiquitous
- problem with setting `$PYTHONPATH` is getting worse due to switch to introduction of `Python-bundle-PyPI`
- someone should do the homework to figure out what needs to be implemented to support this => Mikael
- global EasyBuild configuration option: `--prefer-ebpythonprefixes-over-pythonpath`, `--avoid-pythonpath=PythonPackage,PythonBundle`
- make `PythonBundle` + `PythonPackage` to be aware of it
- what about corner cases like `PLUMED/2.7.2-foss-2021a.lua` which has `prepend_path("PYTHONPATH", pathJoin(root, "lib/plumed/python"))` in env. module
- rename `lib/plumed/python` to `lib/pythonX.Y/site-packages` to avoid having to use `$PYTHONPATH`
- similar: `Graphviz`, `ROOT`, `PYTHIA`
- easyblocks
- [~~15~~ 3 open PRs](https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc+label%3AEasyBuild-5.0+run_shell_cmd+run_cmd) to port easyblocks to `run_shell_cmd`
- [5 open PRs](https://github.com/easybuilders/easybuild-easyblocks/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc+label%3AEasyBuild-5.0+remove) to remove unused custom easyblocks
- easyconfigs
- Replace `parallel` by `maxparallel` ([PR #19375](https://github.com/easybuilders/easybuild-easyconfigs/pull/19375))
### 2024-02-12 sprint meeting (15:00 CET)
attending: Kenneth, Alex, Simon, Bart, Sebastian, Lara
- status of `5.0.x` branches
- framework: synced with `develop` on 8 Feb'24 (CI OK)
- easyblocks: synced with `develop` on 12 Feb'24 (CI OK)
- easyconfigs: synced with `develop` on 12 Feb'24 (CI broken)
- "`Missing dependencies: Gblocks/0.91b`" => Gblocks easyconfig should be pulled back out of archive...
- should start doing syncs of `5.0.x` with `develop` via PRs to avoid breaking CI
- merged PRs (`5.0.x` branches)
- docs
- fix typo in example code for `run_shell_cmd` ([PR #240](https://github.com/easybuilders/easybuild-docs/pull/240))
- framework
- create `lib` -> `lib64` symlink (or vice versa) *before* running `postinstallcmds` ([PR #4435](https://github.com/easybuilders/easybuild-framework/pull/4435))
- make sure that Brunsli libraries are installed into /lib ([PR #19805](https://github.com/easybuilders/easybuild-easyconfigs/pull/19805))
- `-DCMAKE_INSTALL_LIBDIR=lib` should be set by `CMakeMake` easyblock => Alex will open PR
- easyblocks
- over 20 PRs to remove unused easyblocks
- ~50 PRs to migrate from `run_cmd` to `run_shell_cmd` in software-specific easyblocks
- fail on non-zero exit code in command to run Perl test suite ([PR #3170](https://github.com/easybuilders/easybuild-easyblocks/pull/3170))
- enable `use_pip` for extensions that are installed along with Python ([PR #3221](https://github.com/easybuilders/easybuild-easyblocks/pull/3221))
- may need to revisit this and come up with a way to inherit defaults from `PythonPackage` easyblock?
- easyconfigs
- clean up easyconfigs that explicitly set `use_pip`, `sanity_pip_check` and `download_dep_fail` ([PR #19265](https://github.com/easybuilders/easybuild-easyconfigs/pull/19265))
- archive or delete old patches ([PR #19827](https://github.com/easybuilders/easybuild-easyconfigs/pull/19827))
- replace `run_cmd` with `run_shell_cmd` in easyconfigs testsuite ([PR #19818](https://github.com/easybuilders/easybuild-easyconfigs/pull/19818))
- open PRs
- framework
- rename unclear `*run*` methods to `*install_extension*` + rename `install_extensions` to `install_all_extensions` ([PR #4400](https://github.com/easybuilders/easybuild-framework/pull/4400))
- what happens if a `PythonPackage` that has not been migrated yet is used along with an easyblock on top (like `numpy`) that has been migrated
- we can probably detect this and produce a meaningful error that easyblocks must be migrated
- add support for running shell commands asynchronously with `run_shell_cmd` ([PR #4444](https://github.com/easybuilders/easybuild-framework/pull/4444))
- enable RPATH linking by default ([PR #4448](https://github.com/easybuilders/easybuild-framework/pull/4448))
- see also [docs PR #241](https://github.com/easybuilders/easybuild-docs/pull/241)
- waiting for Kenneth to re-review both PRs & merge
- easyblocks
- ~5 PRs to remove unused easyblocks
- ~18 PRs to migrate software-specific easyblocks to `run_shell_cmd`
- OpenBLAS easyblock still uses `parse_log_for_error` (which we want to deprecate, cfr. [framework PR #4433](https://github.com/easybuilders/easybuild-framework/pull/4433))
- OpenSSL => should catch `RunShellCmdError` instead of `EasyBuildError`
- rename `run` method to `install_extension`, and likewise for `prerun` to `pre_install_extension`, `postrun` to `post_install_extension`, and `run_async` to `install_extension_async` ([PR #3064](https://github.com/easybuilders/easybuild-easyblocks/pull/3064))
- update `run_async` methods in `RPackage`, `Rmpi`, and `Rserve` easyblocks to use submit `run_shell_cmd` call into thread pool ([PR #3163](https://github.com/easybuilders/easybuild-easyblocks/pull/3163))
- easyconfigs
- add check to make sure that `download_dep_fail`, `sanity_pip_check`, `use_pip` are not explicitly set to `True` in easyconfigs ([PR #19830](https://github.com/easybuilders/easybuild-easyconfigs/pull/19830))
- WIP
- (Kenneth) support for running interactive commands in `run_shell_cmd` (no PR yet)
- notes
- for [framework PR #4248](https://github.com/easybuilders/easybuild-framework/pull/4248)
- could use `touch` to reset timestamp to `0` rather than relying on `--mtime` option of `tar` => Alex will comment in PR
- easyconfigs archive should be in a separate repo
- not included by default in EasyBuild installation
- requires extra `pip install easybuild-easyconfigs-archive`
- Alex will open PRs for this...
- https://github.com/easybuilders/easybuild-easyconfigs-archive
- TODO
- ...
---
**Older notes copied to https://github.com/easybuilders/easybuild/wiki/EasyBuild-5.0-sync-meetings**
---
#### TODO
* framework: https://github.com/easybuilders/easybuild-framework/milestone/71
* easyblocks: https://github.com/easybuilders/easybuild-easyblocks/milestone/73
* easyconfigs: https://github.com/easybuilders/easybuild-easyconfigs/milestone/72