EasyBuild 5.0

Useful links:

Meetings

Every 1st Wed of the month at 14:00 CE(S)T, via Zoom

Next monthly sync meetings

  • …

Next sprint meetings (max. 30 min)

  • …

Meeting notes

2025-03-12

  • release is imminent: "days, not weeks" -> "hours, not days"
  • see issues found by Alex during regtesting in https://hackmd.io/@3QLwmjO3ToOJFmKFqbREjA/Hk-CXnTjyg
  • problem with XML-LibXML points to problem in framework to be fixed
  • docs: most important is PR that adds structure for EB5 landing page (PR #297)

2025-03-10

  • final release of EasyBuild 5.0.0 on Wed 12 March, or release candidate?
    • is there enough time for quick bugfix releases after EasyBuild v5.0.0?
    • set up "known issues" section on EasyBuild 5.0 landing page
    • broad regression test in coming days
      • close to default configuration
      • Kenneth on RHEL9, Alex on Rocky Linux 8
        • also some more limited testing on Ubuntu 22.04? => Kenneth
    • Alex & Mikael have been using 5.0.x branch in production
    • for some known issues, --from-pr or --include-easyblocks-from-pr will be very helpful
  • blockers:
    • complete overview of changes in docs landing page is "good enough"
  • known issues
    • Fix cargo build when git branch is referenced, affects polars easyconfigs (easyblocks #3654)
    • installation of NCCL with system toolchain failing in RPATH sanity check
  • next sync meeting
    • Wed 12 March 14:00 CET (OK for Simon, Alex, Bart, Kenneth)
      • discuss regression test results
      • progress on docs
      • prep what we'll communicate during EasyBuild conf call
      • call a vote on releasing EasyBuild v5.0.0 vs release candidate in #maintainers

2025-03-03

Merged PRs

framework

  • copy build directory and/or log file(s) if installation failed to path specified via --failed-install-build-dirs-path or --failed-install-logs-path (PR #4601)
  • Add context manager for allowing unresolved templates and make the state members private + remove support for directly setting enable_templating and expect_resolved_template_values (PR #4735)
  • enable keeping of symbolic links by default when copying (set keepsymlinks easyconfig parameter to True) (PR #4773)
  • add support for ModuleLoadEnvironment aliases in modextrapaths easyconfig parameter + deprecate allow_append_abs_path, allow_prepend_abs_path, modextrapaths_append easyconfig parameters (PR #4774)
  • set maximum level of parallellism via parallel configuration option rather than fixed value for max_parallel easyconfig parameter (PR #4777)
  • limit output included in sanity check failure message for missing RPATH section in output of 'readelf -d' + fix regex used to look for '(RPATH)' (PR #4779)

easyblocks

  • #3547
  • #3605

easyconfigs

  • avoid using buildcmd in PySide2 easyconfig (PR #22380)
  • remove deprecated allow_prepend_abs_path from libglvnd and SpaceRanger (PR #22416)
  • replace hardcoded 'CPATH' in modextravars with MODULE_LOAD_ENV_HEADERS constant (PR #22417)

TODO

Timeline

  • next couple of days: merge last PRs + update docs + compose release notes
  • release 5.0.0 Wed 12 March + release talk during conf call slot 17:00 CET
    • release from 5.0.x branches
  • collapse 5.0.x into develop right after 5.0.0 release
    • to avoid breaking --from-pr before EasyBuild 5.0.0 is available
    • to make sure that --new-pr, --from-pr & co work with EasyBuild 5.0.0 when released
  • sync (all?) open PRs with develop after collapsing 5.0.x (with eb --sync-pr)
    • most recent 100 PRs right after collapsing 5.0.x to develop
    • rest over the weekend?
  • swap open PRs targeting 5.0.x to develop branch

2025-02-24

  • attending: Kenneth, Bart, Simon, Alex, Mikael, Jasper, Jörg

Merged PRs

framework

  • Let jobs retweak easyconfigs themselves by passing down --try-* options (PR #4669)
  • don't enable keeping of debug symbols by default (PR #4764)
  • Deprecate use of parallel easyconfig parameter and fix updating the template value (PR #4580)

easyblocks

  • finish adopting module_load_environment in easyblocks (~24 easyblock PRs)
  • remove custom easyblocks for MTL4 (PR #3617) + Primer3 (PR #3621)
  • Update easyblocks to use EasyConfig.parallel property (PR #3557)
  • Fix $PYTHONPATH for hermetic python in TensorFlow builds (PR #3568)
  • Fix building PyTorch when using setup.py as the build command (PR #3574)

TODO

  • (WIP Kenneth) review/merge framework PR #4735: add context manager for allowing unresolved templates and make the state members private + remove support for directly setting enable_templating and expect_resolved_template_values
    • and accompanying easyblocks PR #3547: Use context managers for templating changes in Bundle easyblock
  • how to deal with $CPATH being updated via modextrapaths in easyconfigs?
    • last hurdle to close framework issue #3331
    • close to 100 easyconfigs set $CPATH via modextrapaths
    • auto-translate to C_INCLUDE_PATH & co in framework based on module-search-path-headers configuration setting?
      • downside: can't set $CPATH if really needed
    • modextrapaths should be handled via module_load_environment?
    • replace use of modextrapaths to control CPATH with: 
      ​​​​​​modextrapaths = {SEARCH_PATH_HEADERS: 'include/libelf'}
  • (Kenneth) review/merge of huge PR that revamps LLVM easyblock (easyblocks PR #3373)
  • Add build_target parameter to PythonPackage (easyblocks PR #3575)
    • only enhancement, shouldn't block EasyBuild 5.0 PR?
    • buildcmd is set in some PyTorch easyblocks, but not actually used at all
    • buildcmd used in PySide2 => can use buildopts instead?
  • fix framework issue #4706: Extensions can should not be allowed to impact global environment during builds
    • not trivial to this correctly when extensions are being installed in parallel
    • long-standing bug, should not block EasyBuild 5.0 release
    • easyblocks used for extensions shouldn't change any global state (environment, working directory)
      • but we can't really enforce that?
    • should be easy to fix for sequential installation of extensions…
  • review/merge framework PR #4601: Copy build log and artifacts to a permanent location after failures
    • would be nice to have this included for EasyBuild 5.0
    • wouldn't help with eb --job --from-pr, then you still need to set tmpdir to shared storage location

Timeline for next few weeks

  • release 5.0.0beta2 ASAP with what's in 5.0.x right now
  • Get all PRs to 5.0.x branch merged by Sun 2 March
  • 3-7 March: extensive regression testing on 5.0.x branch + prepare docs
  • Wed 12 March: release of EasyBuild 5.0.0…

Older notes copied to https://github.com/easybuilders/easybuild/wiki/EasyBuild-5.0-sync-meetings

TODO

Select a repo