# EESSI hackathon Jan'22 - installing software on top of EESSI

Live collaborative notes for this task during the 2nd EESSI hackathon; see also general notes at https://hackmd.io/vSOfDj-SQSmYglhxz_R4hg

Notes from Dec'21 hackathon: https://github.com/EESSI/meetings/wiki/EESSI-hackathon-Dec'21

Zoom for this task: https://uib.zoom.us/j/63932793319?pwd=Ry9BQ3VUaHcwTncwSFFrdU1PWnF3QT09

## Custom GCCcore with RPATH wrappers
Prepare environment (e.g. w/ EESSI 2021.06)

source /cvmfs/pilot.eessi-hpc.org/2021.06/init/bash
module load EasyBuild/4.4.1

Notes (Kenneth):

* Make sure that EasyBuild is properly configured (by sourcing `configure_easybuild` script from https://github.com/EESSI/software-layer/)
* note: both `$WORKDIR` and `$EPREFIX` environment variables must be defined!
* Use fast local disk for build directories: `export EASYBUILD_BUILDPATH=/tmp/$USER/easybuild/build`

Build custom GCCcore with rpath wrapper scripts in place:

export EASYBUILD_PREFIX=$HOME
eb GCCcore-9.3.0.eb --include-easyblocks-from-pr 2638 --force --try-amend=add_rpath_wrappers=True

Which picks up the GCCcore-9.3.0.eb from https://github.com/easybuilders/easybuild-easyblocks/pull/2638/files#diff-a2355c6aa1836d1f465436c868adde40651d28cb2bd59cd0e201614fe3b94fe9

To use this custom GCC core module:

module use $HOME/modules/all
module avail GCCcore/9.3.0
module load GCCcore/9.3.0

Now `which gcc` points to a wrapper script (also for `ld` and other commands) that automagically puts the right RPATHs in place when compiling something.

$ gcc hello_lorld.cxx -o hello_world_withrpath
$ readelf -d hello_world_withrpath

### Centrally install custom `GCCcore/9.3.0` that includes RPATH wrapper scripts on CitC cluster

source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
module use /mnt/shared/02_software_on_top/easybuild/modules/all
module load foss/2020a
which gcc
file $(which gcc)

$ gcc hello_world.c -o hello_world
/mnt/shared/02_software_on_top/easybuild/software/GCCcore/9.3.0/bin/rpath_wrappers/gcc: line 40: /tmp/eb-eai87q5k/rpath_wrapper_gcc.log: No such file or directory

Fix is to create expected directory:

$ mkdir -p /tmp/eb-eai87q5k
$ gcc hello_world.c -o hello_world
$ readelf -d hello_world

## Subtasks
- https://github.com/easybuilders/easybuild-framework/issues/3918
- Kenneth: module_generator.py
- Martin: modify prepare_rpath_wrappers() in toolchain.py of EasyBuild framework

## To Figure Out
- Install `ld` wrappers with `binutils` instead?
- Put everything in `GCC (= binutils + GCCcore)`?
- How to deal with rpath wrappers for other compilers? (e.g. intel) => hints towards GCCcore or binutils
- Is it enough to just install ld wrappers? (Martin will look into this)
- in EESSI context:
- we use `binutils` from compat layer (not installed with EasyBuild)
- if we want to wrap `ld`, it would have to be done either in `GCCcore`, or in `GCC`
- this implies more general support is required for installing RPATH wrappers (not just through `GCCcore` easyblock)?

## Building Software on top of EESSI (ie not using EasyBuild)
Notes (Frank Everdij):

Use EESSI 2021.12 stack.
Pick suitable software package candidate:
- LAMMPS is chosen. It has fairly trivial dependencies and it's missing from the current EESSI stack
- Also serves as a test for OpenMPI functionality when not using EasyBuild

### LAMMPS Development and Runtime issues

The naive way of building LAMMPS:

wget https://download.lammps.org/tars/lammps-stable.tar.gz
tar zxvf lammps-stable.tar.gz
cd lammps-29Sep2021
source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
module load Eigen
module load OpenMPI
module load FFTW
module load FFmpeg
mkdir build && cd build && cmake -C ../cmake/presets/most.cmake ../cmake -DCMAKE_Fortran_COMPILER=gfortran -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=~/lammps
make -j8 all >& make.log &
make install

This produces a LAMMPS `lmp` binary in `~/lammps` with missing `libfftw3.so` RPATH. The LAMMPS CMakeLists.txt does not properly address this RPATH issue:

[frankeverdij@mgmt bin]$ ldd lmp | grep fftw
	libfftw3.so.3 => not found
	libfftw3_omp.so.3 => not found

Moreover, this does not take advantage of the recommended development environment already in EESSI.

A fix for this is to use a development stack with RPATH wrappers enabled:

source /cvmfs/pilot.eessi-hpc.org/versions/2021.12/init/bash
module use /mnt/shared/02_software_on_top/easybuild/modules/all
module load foss/2020a
which gcc
[EESSI pilot 2021.12] $ which gcc
/mnt/shared/02_software_on_top/easybuild/software/GCCcore/9.3.0/bin/rpath_wrappers/gcc

This replaces the gcc/g++/ld commands with the wrapper functionality of EasyBuild. The result is a correct(ed) RPATH and the linker now finds the correct library:

[frankeverdij@fair-mastodon-c5d-2xlarge-0001 bin]$ ldd lmp | grep fftw
	libfftw3.so.3 => /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/skylake_avx512/software/FFTW/3.3.8-gompi-2020a/lib/../lib64/libfftw3.so.3 (0x00007fde2e39d000)
	libfftw3_omp.so.3 => /cvmfs/pilot.eessi-hpc.org/versions/2021.12/software/linux/x86_64/intel/skylake_avx512/software/FFTW/3.3.8-gompi-2020a/lib/../lib64/libfftw3_omp.so.3 (0x00007fde3073c000)

The used buildscript and example slurm-job can be found in the hackathon repo:
https://github.com/EESSI/hackathons/tree/02_software_on_top/2022-01/02_software_on_top

### Other issues found
- Don't forget the FOSS module for your development! CMake needs to be loaded separately. Perhaps this can be included in future FOSS modules.
- The temporary wrapper is currently built for a specific GCCCore which is not the default. If you load library modules and assume the default version, these will override the GCCcore module with a newer (10) version and your wrappers aren't there anymore.
- The LAMMPS install script also wants to change the `lmp` executable RPATH setting to remove the build directory. This may not be caught with wrapper scripts.

### To Do

- Add RPATH wrapper functionality into the EESSI to make bare EESSI development builds easier without resorting to a custom GCCcore.
- Alternatively a script can be made with `readelf` `patchelf` `ldd` and `pkgconf` to fix RPATH's in binaries and libraries.

### GetFEM build and runtime

GetFEM is chosen as an example of a `configure;make;make install` build. It produces a python module with C++ library.

The issue when building this package is that it finds the compatibility compilers first, even if one loads the wrapper script.
When loading the wrapper scripts and then run `configure` it finds the wrappers, compiles the application but the linker still uses the compat `ld` binary

[EESSI pilot 2021.12] $ which gcc
/mnt/shared/02_software_on_top/easybuild/software/GCCcore/9.3.0/bin/rpath_wrappers/gcc

and building with:

./configure --prefix=$HOME/getfem --enable-shared --enable-python --enable-metis --with-blas=`pkgconf --variable=libdir  openblas`/libopenblas.so

gives

checking for ld used by gcc... /cvmfs/pilot.eessi-hpc.org/versions/2021.12/compat/linux/x86_64/usr/x86_64-pc-linux-gnu/bin/ld

which results in a `lib/libgetfem.so` library with mixed compat and software RPATH's. The library `lib/python3.8/site-packages/getfem/_getfem.cpython-38-x86_64-linux-gnu.s` seems to have the correct path to libopenblas.so

The example program in https://github.com/EESSI/hackathons/tree/02_software_on_top/2022-01/02_software_on_top/GetFEM_test_cantilever can be called with `python3 - cantilever` and produces point displacement output. This is not a thorough test however and i haven't checked the result, but it serves as an example of a `configure` build script together with `libtool`